johnhenriksson.se

Veeam Immutable Repository with NetApp ONTAP S3

2025-10-31T00:00:00+00:00

In this blog post I will go through how to use the NetApp ONTAP storage to enable Immutable S3 repository in Veeam.

I’ve tested this with ONTAP 9.17.1 and Veeam v12.3.2

The basics steps to enable Immutable repository in Veeam with ONTAP S3 is

Create the bucket in System Manager
Enable Object lock & ensure all permissions on object lock are enabled for the user that access the bucket, I’ve noticed that after you enable object lock more permissions options are available.
Configure the bucket in Veeam, be sure to disable auto provisioning.
The immutability is now managed from Veeam

Create the bucket in ONTAP System Manager

Set the required size and be sure to enable versioning

Enable object lock, set the required mode, compliance in my case and set retention period to “Not Set” as this will be managed by Veeam

Be sure the user accessing the bucket from Veeam has all the required permissions to set object lock configuration

Configure the bucket in Veeam

Create a new S3 compatible object storage repository in Veeam, enter the correct endpoint and credentials. You may have to import the certificate to the Veeam server if it’s not trusted.

Enter the bucket name, browse the bucket and create the desired folder. Be sure to disable automatic bucket creation.

Configure the desired immutable retention, this can be changed at a later stage as well by editing the repo in Veeam.

Continue and create the repo and now the backups are immutable in ONTAP S3, managed by Veeam.

Here’s a full video of the process that I’ve described in this blog post.

Build Ubuntu 24.04 VM on vSphere with Packer

2025-09-21T00:00:00+00:00

A couple of years ago I did a post on how to use Packer to build Ubuntu 20.04 VM on vSphere. You can find it here. It go a lot of reads and now I’ve made an update focused on 24.04 and also changed from yaml to hcl code. I did this because I’m on a mission to move VM’s from an old KVM host to vSphere, instead of using some shady migration tool I will rebuild the VM’s and move the configuration files. I want to free the KVM host to install Proxmox instead, more on that in a later post.

What is Packer?

Let’s hear it from the source:

Packer is a community tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

Introduction to Packer

So basically, you can use Packer to build VM’s from ISO’s based on code. Infrastructure as code, just how we like it. Also, it’s easy to implement in your CI/CD pipeline to go full hipster.

Packer builds the VM from the desired ISO and is also able to perform advanced provision tasks. Also, I’m able to inject my SSH public key to enable key-based authentication on the template.

I’ll later use this VM template as a source when deploying infrastructure resources with Terraform, but more on that in another blog post.

Install Packer

I will not go through the installation process in this blog. Please visit the getting started guide on packer.io if you need help installing. You’ll find it here.

Create the Build

In my home lab, I’m now able to build a fresh template in ~12 minutes. That’s pretty good!

You’ll need these files. The ubuntu.pkr.hcl file is where you configure settings for Packer to build and provision the VM, connect to vSphere, and so on. The ubuntu.auto.pkrvars.hcl contains environment specific settings such as vCenter password, so keep it well guarded. The user-data and meta-data files are used for unattended installation of Ubuntu. Although only the user-data files contain information, both must be present. Packer uses these files and mounts them as CDROM’s to the vSphere guest VM.

As a bonus I’ve added docker-engine.sh, a shell script to be used as a Packer provisioner to install docker engine on my template, as all my VM’s use this to run containers.

❯ tree .
.
├── docker-engine.sh
├── http
│   ├── meta-data
│   └── user-data
├── ubuntu.auto.pkrvars.hcl
├── ubuntu.pkr.hcl
└── variables.pkr.hcl

2 directories, 6 files

Also, you’ll need to upload an ISO to the vSphere datastore and edit the iso_url variable. I’ve uploaded the ISO to my datastore2 datastore, as you see here.

Download Ubuntu Server ISO

#ubuntu.auto.pkrvars.hcl

iso_url = "[datastore2] ISO/ubuntu-24.04.3-live-server-amd64.iso"

Important code snippets in the build file

SSH Timeout

So, I had some trouble with Packer where it failed to connect to SSH. The issue is that SSH is present when Ubuntu performs the unattended installation, then Ubuntu reboots and performs the cloud-init part and configures my user. Packer thinks the VM is ready on the first SSH connectivity.

Be sure to add this snippet to workaround the issue.

#ubuntu.pkr.hcl
ssh_timeout            = "60m"
ssh_handshake_attempts = 100

Generate Password

You need to generate a password salted with SHA512. Unfortnaetly I faced some issues with my Mac where mkpasswd is not present. Well, docker to the rescue! The below command works fine to generate the password with the help of docker.

> docker run --rm -ti alpine:latest mkpasswd -m sha512
Password: 
$6$eZsKuZ3lwrfwvrJ3$FSmB.RPl9doTTe57fHDvbMjyu/wFZOcNfuGZ82id4ZZYbUZ0P9GP6ZWamYK3JZzwnMIjL1WEVczqHf03.IqYS/

Configure Identity

Edit the user-data file with the newly created password.

#user-data
identity:
  hostname: ubuntu-packer
  username: ubnt
  password: $6$RAVeOPg.9rJylcnK$tm6KVpRXxTGb0et.1PQcmLVCFLjIVaLye.Weiizxw3ecGRI2kA0JH7nFxkJCz4oeZphSO4qzYSwncE/s34UHg0

SSH Settings

First, you’ll need your SSH public key if you want to enable key-based authentication in your template.

❯ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com

Secondly, inject the key in the user-data file.

#user-data
ssh:
  install-server: yes
  allow-pw: true
  authorized-keys: 
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com

Build

When you’ve created the required files and filled in all the variables, you’re ready to build the VM. Use the below to start the build. If you’re overwriting the VM with each iteration, use the -force flag.

❯ packer init .
❯ packer build -force .

When you’re done, you’ll find the template in vSphere ready for use!

Required Files

ubuntu.pkr.hcl

packer {
  required_plugins {
    vsphere = {
      source  = "github.com/hashicorp/vsphere"
      version = ">= 1.0.0"
    }
  }
}

source "vsphere-iso" "ubuntu" {
  # Point to your ESXi host here
  vcenter_server      = var.vsphere_server
  #host                = var.vsphere_server

  username            = var.vsphere_user
  password            = var.vsphere_password
  insecure_connection = true

  # Remove or comment these when using standalone ESXi
  datacenter = var.vsphere_datacenter
  cluster    = var.vsphere_cluster

  datastore           = var.vsphere_datastore
  guest_os_type       = "ubuntu64Guest"

  CPUs                = var.vm_cpu_num
  RAM                 = var.vm_mem_size
  RAM_reserve_all     = false

  # Must be a list for your plugin version
  disk_controller_type = ["pvscsi"]

  storage {
    disk_size             = var.vm_disk_size
    disk_thin_provisioned = true
  }

  network_adapters {
    network      = var.vsphere_network   # ESXi portgroup name
    network_card = "vmxnet3"
  }

  vm_name             = var.vm_name
  notes               = "Build via Packer"
  convert_to_template = false

  ssh_username           = "ubnt"
  ssh_password           = "ubnt"
  ssh_timeout            = "60m"
  ssh_handshake_attempts = 100


  iso_paths = [var.iso_url]

  cd_files = [
    "./http/user-data",
    "./http/meta-data",
  ]
  cd_label  = "cidata"
  boot_wait = "10s"

  boot_command = [
    "c", "<wait3s>",
    "linux /casper/vmlinuz autoinstall ds=nocloud;s=/cdrom/", "<enter><wait3s>",
    "initrd /casper/initrd", "<enter><wait3s>",
    "boot", "<enter>",
  ]

  shutdown_command = "echo 'ubnt'|sudo -S shutdown -P now"

}



build {
  sources = ["source.vsphere-iso.ubuntu"]
  provisioner "shell" {
    inline = [
      "echo set debconf to Noninteractive", 
      "echo 'debconf debconf/frontend select Noninteractive' | sudo debconf-set-selections" ]
  }
  provisioner "shell" {
    inline = [
      "sudo apt-get -y install util-linux-extra" ]
  }
  provisioner "shell" {
    script = "./docker-engine.sh"
    pause_before = "10s"
  }
}

docker-engine.sh

#docker-engine.sh

sleep 30

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get -y install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

ubuntu.auto.pkrvars.hcl

# WARNING: This file contains secrets. Don't commit it. Add to .gitignore. 🔒
vsphere_server     = "10.128.128.101"                   # vCenter address if using datacenter/cluster
vsphere_user       = "administrator@vsphere.local"
vsphere_password   = "supersecretpassword"
vsphere_datacenter = "Datacenter"
vsphere_cluster    = "Cluster"             # Must be the vSphere Cluster name or ESXi hostname

vsphere_network   = "VLAN128"
vsphere_datastore = "datastore2"

vm_name      = "ubnt-packer"
vm_cpu_num   = 1
vm_mem_size  = 1024
vm_disk_size = 25600

iso_url = "[datastore2] ISO/ubuntu-24.04.3-live-server-amd64.iso"

variables.pkr.hcl

variable "vsphere_server" {
  type = string
}

variable "vsphere_user" {
  type = string
}

variable "vsphere_password" {
  type      = string
  sensitive = true
}

variable "vsphere_datacenter" {
  type = string
}

variable "vsphere_cluster" {
  type = string
}

variable "vsphere_network" {
  type = string
}

variable "vsphere_datastore" {
  type = string
}

variable "vm_name" {
  type = string
}

variable "vm_cpu_num" {
  type = number
}

variable "vm_mem_size" {
  type = number
}

variable "vm_disk_size" {
  type = number
}

variable "iso_url" {
  type = string
}

user-data

#cloud-config
autoinstall:
  version: 1
  locale: en_US
  keyboard:
    layout: se
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubuntu-packer
    username: ubnt
    password: $6$eZsKuZ3lwrfwvrJ3$FSmB.RPl9doTTe57fHDvbMjyu/wFZOcNfuGZ82id4ZZYbUZ0P9GP6ZWamYK3JZzwnMIjL1WEVczqHf03.IqYS/
  ssh:
    install-server: yes
    allow-pw: true
    authorized-keys: 
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com
  user-data:
    disable_root: false
  late-commands:
    - echo 'ubnt ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu

Block Youtube Shorts on Network

2025-09-18T00:00:00+00:00

YouTube Shorts is an annoying, addictive video service that my kids love. I’ve been looking for a way to disable just the shorts feature as they are allowed to view “classic videos”. In my search I stumbled upon this guide, which I’ve extended on https://dealancer.medium.com/blocking-youyube-shorts-with-nginx-and-nextdns-c2ccb0ff5452

What I learned is that the requests to YouTube have the parameter &ctier=SH so you need to intersect the traffic to YouTube and block these specific request, while allowing the rest.

In my setup I use pihole as DNS server and I use a CA certificate to sign the Nginx certificate on a seperate Ubuntu server. I need a CA as that is the only way to get the trust to iPhone/iPad.

The flow looks something like this:

Pi-hole configuration

First, I needed my DNS server to in a wildcard manner redirect *.googlevideos.com to my nginx server at 10.128.128.102

This is done through dnsmasq and if you go to Settings -> All Settings -> Miscellaneous -> misc.dnsmasq_lines and you add the line as below.

This way *.googlevideo.com will be redirected to 10.128.128.102

address=/googlevideo.com/10.128.128.102

Set up certificates and nginx proxy on Ubuntu

First step done, now it’s time to move on to the ngnix server, hosted on Ubuntu 22.04.

We need SSL certificates as we will do a SSL proxy. I will generate a CA certificate and then sign the certificate needed for nginx, this as iPhones won’t accept to import and trust a standalone self-signed certificate.

First you need to install openssl

sudo apt install openssl

We need a key for our CA

openssl genrsa -aes256 -out ca-key.pem 4096

Next step, create a certificate for our CA, important to specify the days to 365 as Apple won’t accept any CA beyond 1 year

openssl req -new -x509 -sha256 -days 365 -key ca-key.pem -out ca.pem

The ca.pem file is the one you need to install in the trusted root store on your clients

Time to create the cert needed for the nginx service, let’s start with the key file used for the cert

openssl genrsa -out cert-key.pem 4096

Now we create the cert sign request

openssl req -subj "/CN=*.googlevideo.com" -sha256 -new -key cert-key.pem -out cert.csr

Once the request is created we create the file san.cnf to hold the certificate configuration

[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_req
prompt             = no

[ req_distinguished_name ]
CN = *.googlevideo.com

[ req_ext ]
subjectAltName = @alt_names

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = *.googlevideo.com
DNS.2 = *.a1.googlevideo.com
DNS.3 = *.c.googlevideo.com
DNS.4 = *.dai.googlevideo.com

Now we have everything to generate a CA signed certificate

openssl x509 -req -sha256 -days 365 \
  -in cert.csr \
  -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial \
  -out cert.crt \
  -extfile san.cnf -extensions v3_req

You can verify the cert with the below command

openssl x509 -in cert.crt -text -noout | grep -A1 "Subject Alternative Name"

Configure proxy

Now it’s time to configure the nginx web proxy

To install nginx do

apt update
apt install nginx

Add this to /etc/nginx/nginx.conf - I’ve copied this from the excellent guide mentioned at the start

# Use DNS server to resolve IP address of a domain
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

server {
        # Listeon on HTTPS
        listen 443 ssl http2;

        # Listen on any domain
        server_name _;

        # Use self signed certificates
        ssl_certificate /certs/mydomain.crt;
        ssl_certificate_key /certs/mydomain.key;

        location / {
                # Block YouTube shorts based on query parameter
                if ($arg_ctier = "SH") {
                        return 403;
                }

                # Proxy pass to the same domain
                set $backend_host $host;
                proxy_pass https://$backend_host$request_uri;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_ssl_server_name on;
                proxy_ssl_protocols TLSv1.2 TLSv1.3;
                proxy_ssl_ciphers HIGH:!aNULL:!MD5;
        }
}

Restart nginx to apply the configuration

service nginx restart

Conclusion

Now, every device using pihole as DNS will have YT shorts blocked.

Finally, I wanted only my kids devices to use pihole DNS, allowing me to control the traffic, both block unwanted sites through DNS lists and also redirect YouTube to my proxy.

What I’ve done is that I use Edgerouter X as router and in the DHCP configuration I edit optional parameters to the static mapping to leverage my pihole DNS for these clients.

Go to the config tree -> service -> DHCP-server -> shared-network-name -> (your network) -> subnet -> (your subnet) -> static mapping -> desired device

add this parameter to static-mapping-parameters where the IP would be your pihole server

option domain-name-servers 10.128.128.93;

Now your desired clients use pihole as DNS and will redirect the YouTube traffic to the nginx proxy

Locking Down Your NetApp ONTAP: A Comprehensive Security Guide for Safeguarding Your Data

2024-02-27T00:00:00+00:00

Welcome to my blog post on the security hardening guide for NetApp ONTAP! In this post, we will provide you with all the information you need to ensure the security of your NetApp ONTAP deployment. We will cover topics such as image validation, local storage administrator accounts, authentication methods, and more. So let’s dive in!

ONTAP Image Validation

Image validation is an essential security measure that helps verify the authenticity and integrity of ONTAP images installed on your system. It ensures that the images have not been tampered with and are produced by NetApp. Upgrade image validation, introduced in ONTAP 9.3, allows you to validate images installed through image updates or automated updates. Boot-time image validation, available in ONTAP 9.4 and later versions, enables secure booting by validating the bootloader and preventing unauthorized modifications.

Documentation of cluster image validate https://docs.netapp.com/us-en/ontap-cli-9111/cluster-image-validate.html

Local Storage Administrator Accounts

Managing administrator accounts is crucial for maintaining the security of your ONTAP system. With role-based access control (RBAC), you can assign specific roles to users based on their job requirements. ONTAP provides predefined roles for cluster administrators and storage virtual machine (SVM) administrators. You can also create custom roles and specify account restrictions for each role. By limiting administrative access to the necessary level, you can prevent unauthorized actions and minimize the risk of privilege escalation.

Authentication Methods

ONTAP supports various authentication methods to ensure secure access to the system. These methods include password authentication, SSL certificate authentication, SNMP community strings, and more. You can configure the authentication method based on your organization’s security policies and requirements. Additionally, ONTAP 9.11.1 introduces multi-admin verification (MAV), which allows certain operations to be executed only after approvals from designated administrators. MAV prevents unauthorized or undesirable changes by requiring multiple administrators to approve critical actions.

Documentation of Authentication and access control https://docs.netapp.com/us-en/ontap/authentication-access-control/index.html

ONTAP provides several password parameters to enforce password complexity and expiration policies. You can set minimum password length, alphanumeric requirements, special character requirements, and more. Additionally, you can configure password expiration time, account lockout policies, and password change delay. These parameters help ensure that user accounts adhere to your organization’s password security guidelines. By enforcing strong password policies, you can reduce the risk of unauthorized access and data breaches.

Documentation of Requirements for local user passwords https://docs.netapp.com/us-en/ontap/smb-admin/requirements-local-user-passwords-concept.html

System Administration Methods

Command-line access is a common method for administering ONTAP systems. When it comes to remote command-line access, SSH is the recommended and most secure option. NetApp highly recommends using SSH for secure remote access, as it encrypts the connection and provides authentication mechanisms. Telnet and RSH, on the other hand, are disabled by default due to their security vulnerabilities. By using SSH, you can establish a secure connection and protect your system from unauthorized access.

Login banners and MOTD are essential for communicating acceptable use policies and access restrictions to users. ONTAP allows you to modify the login banner and MOTD to display custom messages. The login banner is displayed before the authentication step during SSH and console device login, while the MOTD is displayed after successful login. You can customize these messages to inform users about system policies, security guidelines, and any other relevant information.

Documentation of Managing the banner https://docs.netapp.com/us-en/ontap/system-admin/manage-banner-reference.html

Conclusion

In this blog post, we discussed various security measures and best practices for NetApp ONTAP. By following these guidelines, you can ensure the confidentiality, integrity, and availability of your information system. From image validation to authentication methods and password parameters, each aspect plays a crucial role in securing your ONTAP deployment. Remember to regularly review and update your security settings to stay ahead of evolving threats. Stay secure and keep your data protected with NetApp ONTAP!

Read more in the full technical report: https://www.netapp.com/pdf.html?item=/media/10674-tr4569.pdf

Download and Activate ONTAP One

2024-02-26T00:00:00+00:00

In early 2023 NetApp released the new licensing model ONTAP One which made it possible for everyone with active support to get use of premium-priced features such as SnapLock and Anti Ransomware. In this post, I will cover how you download and enable the new license on your system

The license change is non-disruptive and you can enable the new unlocked features in a safe manner when you wish.

Step 1, pre-reqs

Before you start you need;

Serial number of the nodes to upgrade
Make sure you have active support and purchased some premium bundle to be able to upgrade to ONTAP One. If you have only the core bundle you will get the ONTAP Base which includes only the base funcitonallity and not the extended features like ARP & SnapLock.
- If you do not have purchased a premium bundle you can purchase the ONTAP One compability package - reach out to your NetApp sales rep.
- Review the table below to find what new license you’re entitled to.

ONTAP Software Bundle on System	New Software suite being matched	Additional Keys entitled
Core	ONTAP Base	NVMeoF
Base	ONTAP Base	NVMeoF, FlexClone, S3, SnapRestore
Core + Data Protection, Flash, Premium	ONTAP One	NVMeoF, ARP, MTKM, SnapLock, SnapMirror Cloud, S3 SnapMirror
Core + S&C	ONTAP Base	NVMeoF

NVMeoF – Enables the NVMeoF protocol for front end client IO, both NVMe/FC and NVMe/TCP
FlexClone – Enables rapid creation of space efficient cloning of data based on snapshots
S3 – Enables the S3 protocol for front end client IO
SnapRestore – Enables rapid recovery of data from snapshots
ARP – Autonomous Ransomware Protection, enables the automatic protection of NAS file shares when abnormal filesystem activity is detected
MTKM – Multi Tenant Key Manager, enables the ability to have multiple key managers for different tenants on the system
SnapLock – Enables the protection of data from modification, deletion or corruption on the system
SnapMirror Cloud – Enables the replication of system volumes to object targets
S3 SnapMirror – Enables the replication of ONTAP S3 objects to alternate S3 compatible targets

Generate the new license

Login to https://mysupport.netapp.com, click on Systems menu, choose Software Licenses and search by System Serial Number. Locate your CIFS_2 license and click on Check link in the Eligibility column

The Check Eligibility form dialog box will appear

Click on the Generate Licenses for 9.10.x and later button

The conversion process will take some time to complete, so click on Close to close the dialog box

After 2 hours, return to the Software Licenses page and query using your System Serial Number.

The ONTAP One license associated to your system should be available

Click on the Get NetApp License File link and then follow the prompts to email or save your NLF.

Note: If you encounter any issues during the procedure, open a non-technical case for further assistance on https://.netapp.com/site/help

Activate the new license in ONTAP

Select Cluster > Settings.
Under Licenses, select arrow icon.
Select Browse. Choose the NetApp License File you downloaded.

ONTAP 9.10.1 and later licensing overview https://kb.netapp.com/onprem/ontap/os/ONTAP_9.10.1_and_later_licensing_overview

How to get an ONTAP One license when the system has NLFs already https://kb.netapp.com/onprem/ontap/os/How_to_get_an_ONTAP_One_license_when_the_system_has_NLFs_already

Monitor Trident with Prometheus and Grafana

2021-09-13T00:00:00+00:00

Trident provides Prometheus metrics endpoints that you can use to monitor Trident, its performance, and understand your environment. In this blog, we’ll deep dive into how to install and configure Prometheus to monitor Trident.

I’ve set up my Kubernetes cluster with kubeadm, and it consists of four nodes, one master node, and three worker nodes. As Kubernetes has no built-in storage, I’ve deployed the software-defined version of NetApp ONTAP called ONTAP Select. I’ve deployed the CSI-compliant storage orchestrator, Trident. I will not go into detail on setting up K8s or Trident in this post.

This diagram presents the environment we’re working with.

The goal of this post is to set up a Grafana dashboard monitoring my storage solution in Kubernetes.

State of my Kubernetes cluster. As you see, I have all the nodes configured and installed with Kubernetes v1.20.2. Also, I have trident version 21.07.1 deployed.

❯ k get nodes
NAME        STATUS  ROLES         AGE  VERSION
k8s-controller-1  Ready  control-plane,master  204d  v1.20.2
k8s-worker-1    Ready  <none>         204d  v1.20.2
k8s-worker-2    Ready  <none>         204d  v1.20.2
k8s-worker-3    Ready  <none>         204d  v1.20.2

❯ k get tver -n trident
NAME   VERSION
trident  21.07.1

What we need to do now is to deploy Prometheus on the cluster, leverage the ServiceMonitor CRD that watches the trident metrics and feeds back to Prometheus. Lastly, we hook up the Prometheus instance to my Grafana server. Easy peasy, right?! No worries, I’ll help you get it running.

Prometheus Operator

I know some people deploy the whole monitoring stack on Kubernetes, with both Grafana and Prometheus. I guess it’s because it’s easy to deploy. However, with a background in ITOps, it goes against my belief in deploying the monitoring stack on top of the kit you monitor. So, as I already have a standalone Grafana stack in my lab, I will just run Prometheus in k8s and hook that instance up to my Grafana server.

Let’s get going and start the installation of the Prometheus Operator.

The Prometheus Operator is software that helps install and configure Prometheus instances on the Kubernetes cluster. You also get access to Custom Resource Definitions such as the ServiceMonitor that we need to monitor Trident. The operator allows you to easily set up multiple Prometheus instances, one for each application if you like. Thus, everything is streamlined and automated.

Let’s clone the GitHub repo and install the operator on the cluster.

❯ git clone https://github.com/prometheus-operator/prometheus-operator.git
❯ cd prometheus-operator
❯ k apply -f bundle.YAML
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
serviceaccount/prometheus-operator created
service/prometheus-operator created

As you see in the terminal output, we get a lot of custom resource definitions deployed. CRD’s are essential, as, without them, we would not be able to deploy the ServiceMonitor we need, and we would get an error message like this one no matches for kind "ServiceMonitor in version "monitoring.coreos.com/v1".

Trident Metrics

I said earlier that Trident exposes Prometheus metrics. To elaborate, Trident reveals these metrics on a specific port called metrics. First, let’s have a look at the Trident service.

❯ k describe svc -n trident -l app=controller.csi.trident.netapp.io
Name:              trident-csi
Namespace:         trident
Labels:            app=controller.csi.trident.netapp.io
                   k8s_version=v1.20.2
                   trident_version=v21.07.1
Annotations:       <none>
Selector:          app=controller.csi.trident.netapp.io
Type:              ClusterIP
IP:                10.108.176.147
Port:              https  34571/TCP
TargetPort:        8443/TCP
Endpoints:         10.44.0.1:8443
Port:              metrics  9220/TCP
TargetPort:        8001/TCP
Endpoints:         10.44.0.1:8001
Session Affinity:  None
Events:            <none>

As you see, we have a Trident service that exposes several ports, and we can quickly identify the one of interest for us, metrics. It’s of importance that we always reference this port by its name, not the port number. That’s how Prometheus works.

We can leverage curl to verify these metrics, but as the service is type ClusterIP and only internal to k8s, we need to use the port-forward command.

❯ k port-forward service/trident-csi -n trident :metrics
Forwarding from 127.0.0.1:59666 -> 8001
Forwarding from [::1]:59666 -> 8001
Handling connection for 59666

When the port-forward is running, open another terminal window and query the local port with curl.

❯ curl -s localhost:59666 | grep trident_backend_count
# HELP trident_backend_count The total number of backends
# TYPE trident_backend_count gauge
trident_backend_count{backend_state="online",backend_type="ontap-nas"} 1

Set up Prometheus

Ok, so we have installed the Prometheus Operator, and we have access to Trident metrics. Now we need to deploy a Prometheus instance, ServiceMonitor CRD, other dependencies, and connect everything.

So a list of what we need to deploy:

Serice Account to run Prometheus
ClusterRole & ClusterRoleBinding
Prometheus Instance
Prometheus Service (NodePort)
Trident ServiceMonitor

I’ll walk through each part, but I’ve put all the resources in a single file named prom.yaml.

ServiceAccount

First, let’s look at the ServiceAccount, nothing fancy, but it needs to be there.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-standalone

ClusterRole

We need to create a role for the Prometheus service account. You’ll find the required permissions in this file.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-standalone
rules:
- apiGroups: [""]
  resources:
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

ClusterRoleBinding

Use ClusterRoleBinding to tie together the ServiceAccount and the ClusterRole.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-standalone
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-standalone
subjects:
- kind: ServiceAccount
  name: prometheus-standalone
  namespace: default

Prometheus Instance

With this snippet, we leverage the Prometheus Operator to deploy an instance of Prometheus wherever we want it. This way, you can embed the Prometheus instance with the application however you prefer.

Note that we tell the Prometheus Operator what version we want to deploy, and also we define how ServiceMonitors will be picked up. This is done with the ServiceMonitorSelector. For example, I want to pick up the ServiceMonitor labeled as app: trident - nothing else in this deployment.

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-standalone
  labels:
    prometheus: k8s
spec:
  replicas: 1
  version: v2.29.2
  serviceAccountName: prometheus-standalone
  serviceMonitorSelector:
    matchLabels:
      app: trident

Prometheus Service

Once we have Prometheus up and running, I want to expose the Prometheus port to connect it to my standalone Grafana server. As I don’t have any load balancer deployed in my lab, I will use the service type of NodePort.

With this service, Prometheus is exposed on port 9090 on all my nodes.

---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: prometheus
  name: prometheus-prometheus-standalone-0
spec:
  ports:
  - port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus
    app.kubernetes.io/instance: prometheus-standalone
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/version: 2.29.2
    controller-revision-hash: prometheus-prometheus-standalone-5999b488c
    operator.prometheus.io/name: prometheus-standalone
    operator.prometheus.io/shard: "0"
    prometheus: prometheus-standalone
    statefulset.kubernetes.io/pod-name: prometheus-prometheus-standalone-0
  type: NodePort
status:
  loadBalancer: {}

Trident ServiceMonitor

We need the ServiceMonitor that feeds back metrics to Prometheus. Note in this file that we specify the label app: trident so Prometheus picks it up as we’ve previously configured.

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: trident-sm
  labels:
    app: trident
spec:
  jobLabel: trident
  selector:
    matchLabels:
      app: controller.csi.trident.netapp.io
  namespaceSelector:
    matchNames:
    - trident
  endpoints:
  - port: metrics
    interval: 15s

Deploy and Configure

We got everything prepared, all the YAML is written, we’re coffeed up and ready to go!

I’ve concatenated everything but the ServiceMonitor to one file. This makes everything easy and smooth to deploy.

prom.yaml contains everything but the ServiceMonitor.

❯ tree .
.
├── prom.yaml
└── sm-trident-prom.yaml

0 directories, 2 files

Deploy Prometheus on the cluster

First, apply the prom.yaml file. I put everything in the default namespace just because I can..

❯ k apply -f prom.yaml
serviceaccount/prometheus-standalone created
clusterrole.rbac.authorization.k8s.io/prometheus-standalone created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-standalone created
prometheus.monitoring.coreos.com/prometheus-standalone created
service/prometheus-prometheus-standalone-0 created

We quickly see everything we defined now being created on the cluster. To verify, we can query the Prometheus resources.

❯ k get prometheus
NAME                    VERSION   REPLICAS   AGE
prometheus-standalone   v2.29.2   1          2m57s

Deploy the Trident ServiceMonitor

Now, we have a Prometheus instance on the cluster. But for now, it’s empty, as we configured the ServiceMonitorSelector parameter to look for the label app: trident, and we have no ServiceMonitors with that label.

Deploy the sm-trident-prom. yaml file on the cluster.

❯ k apply -f sm-trident-prom.yaml
servicemonitor.monitoring.coreos.com/trident-sm created

❯ k get servicemonitor
NAME         AGE
trident-sm   23s

We’ve successfully created the ServiceMonitor resource on the cluster.

Verify Prometheus and Trident ServiceMonitor

Everything is deployed, and Kubernetes once again showed its power of automation and agility, Isn’t it nice to request something in a file, apply, and it’s just there? Much easier than a traditional installation/configuration.

We must verify that everything works this far. It’ll make our lives easier when we move over to the next step of connecting Grafana to Prometheus.

In the configuration, we defined a NodePort service for Prometheus. We can use this to log in to the Prometheus web interface.

❯ k get svc prometheus-prometheus-standalone-0
NAME                                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
prometheus-prometheus-standalone-0   NodePort   10.111.214.23   <none>        9090:32093/TCP   12m

We need to connect to the 32093 port of one of our nodes. But, first, let’s find out what IP our nodes have.

❯ kubectl get nodes -o yaml | grep -- "- address:"
    - address: 10.128.128.111
    - address: k8s-controller-1
    - address: 10.128.128.121
    - address: k8s-worker-1
    - address: 10.128.128.122
    - address: k8s-worker-2
    - address: 10.128.128.123
    - address: k8s-worker-3

I’ll choose one of my workers, http://10.128.128.122:32093.

Head over to Status -> Targets and you’ll find the ServiceMonitor dicovered by Prometheus.

If we go to the Graph page and type in tri, we get Trident’s metrics. This way, we know everything is working.

Configure Grafana

I’ve already set up my Grafana instance on a standalone VM outside of my k8s cluster. So this is not a guide on installing Grafana. Instead, in this post, I’ll show you how to hook it up to monitor Trident.

Open Grafana go to Configuration -> Data Sources and add a Prometheus data source. Define the URL that we used earlier http://10.128.128.122:32093. That’s about it. Now we have integration between Grafana and Prometheus.

If you intend to run this setup in production, you need to think about security, secure communication between Grafana and Prometheus, and restrict access.

Add a Dashboard

You’ll want a friendly dashboard to overview your Trident environment quickly. This dashboard can be used to set up a TV in a NOC, or in your living room for that matter!

I’ve not created any dashboards myself. I recommend using the ready built examples from Yves Weisser. You find them on his github.

In this example, I use the dashboard Trident_Dashboard_21.04.json.

My environment is quite dull as I have just one backend and one PVC with no snapshots. However, as you add backends, volumes, and snapshots, that’ll all reflect the dashboard, and you get an excellent overview of what’s going on with storage in your Kubernetes environment.

Resources

Yves github repo have helped me alot in setting this up.
Kudos to the Prometheus Operator project on github that’s behind all the magic in this setup.
NetApp documentation on how to monitor Trident.
Want to learn more about Trident? Head over to netapp.io

I hope you’ve enjoyed this blog post, leave a comment down below on how you liked it!

ONTAP Select vCenter Plug-In

2021-04-22T00:00:00+00:00

This was a new discovery for me! Maybe everyone already knows about the ONTAP Select vCenter Plug-in - but for me, it’s a new acquaintance. With the plug-in, you quickly deploy and manage ONTAP clusters on your VMware environment, all from vCenter.

Let us start with what ONTAP Select is and go through the architecture of the ONTAP Select deployment process.

What is ONTAP Select?

ONTAP Select is the software-defined version of NetApp’s storage operating system ONTAP. Essentially it’s the same operating system that runs in the appliances AFF & FAS. With ONTAP Select, you can choose to deploy it on custom hardware, KVM, or VMware.

You can find ONTAP Select on the NetApp support site. Login to your NetApp SSO account on https://mysupport.netapp.com and navigate to the ONTAP Select download page.

ONTAP Select Deploy VM

When we look closer into how ONTAP Select (OTS for short) works, we quickly discover that OTS uses a deployment VM to provision storage node VM’s to the ESXi cluster. OTS Deploy VM provides automation of installation and provisioning on these storage nodes. As an administrator, you simply fill out a web form wizard, and OTS Deploy takes care of the rest.

ONTAP Select vCenter Plug-In

The vCenter plug-in is packaged with the OTS Deploy VM. When you’re done installing OTS Deploy VM, the next step is to provision the vCenter plug-ing from OTS Deploy. You can have multiple vCenter servers in one Deploy VM, but you can only install one instance of the plug-in at each vCenter server.

Deploy vCenter Plug-in

Let’s dive into the fun stuff!

First, you need to deploy the ONTAP Select Deploy OVA - I’ll not cover that in this post, so please ensure you have that done before going ahead with these steps.

When you have OTS Deploy installed, go ahead and log in to the web interface.

Navigate to the pane where you register vCenter servers - Administration -> Management Servers -> Add vCenter

Add the details of the vCenter server connection and add the vCenter server

You’ll see a message of successful provisioning of the vCenter server

Furthermore, in vCenter UI, you clearly see the plug-in deployed

Let’s navigate to the Global Inventory List to view our new ONTAP Select Plug-in

In the right pane, you see the ONTAP Select Clusters

Now that we’re in the vCenter Plug-in, we’re able to deploy new storage clusters directly from this view. Let’s click the New ONTAP Select button to deploy a new OTS instance

This is the same view as the wizard in the native OTS Deploy VM - so let’s add the details of our new OTS workload cluster

You need to specify which host you want to deploy to, as well as VLAN’s and storage pool capacity

Enter the desired OTS admin password and click Create Cluster to begin the automated deployment

We clearly see that the new OTS cluster create operation is initialized

We also see the newly created cluster in the vCenter Plugin-in view

In vCenter, we see that the cluster is initializing

When we navigate to the OTS Deploy VM, we see the same view as in vCenter

Once the cluster create operation is completed, we see that the OTS node turns green in the vCenter view, and everything is done. Click on the button at the right to launch System Manager

That brings us directly to the System Manager of the deployed ONTAP Select cluster. Let’s log in and have a look

Once logged in, we see that OTS is just the same ONTAP that we’re used to managing, just that it runs on top of VMware instead of on an appliance from NetApp

Likewise, in the ONTAP Select Deploy VM, we see that the cluster create operation is completed

Deploy K8s VM's with Terraform on vSphere

2021-01-21T00:00:00+00:00

I’m on my way to deploying Kubernetes the hard way, following Kelsey Hightowers GitHub tutorial. The guide uses GCP for compute resources, but I want to utilize my home lab based on VMware, and I want to do it in a cloud fashion, with Infrastructure as Code. For this, I leverage tools like Packer and Terraform. In my previous post, I showed you how to build a VM template on vSphere with Packer. In this post, I’ll show you how to provision VMs from that template with Terraform.

The mission is to deploy three controller nodes and three worker nodes, a total of 6 VMs. All these VMs should be full clones of my Ubuntu template. I want to create a folder in vSphere called k8s and put all the VMs in it. Also, I want Terraform to give me the IP addresses of the VMs so I can easily connect and continue configuration.

So, what I need from Terraform is;

VM folder named k8s
Controller nodes, k8s-controller-1, k8s-controller-2, k8s-controller-3
Worker nodes, k8s-worker-1, k8s-worker-2, k8s-worker-3
IP address output of each VM

I’ll use my Ubuntu 20.04 template built with Packer as the base for these VMs. If you haven’t read my previous post, please do so now.

The prepared template already has SSH keys, user accounts, settings, and so on set up. I highly recommend you set up your template as it makes things ALOT easier! When I need to rebuild my lab, I can generate a new VM within minutes, and everything is ready to connect once provisioned.

Important Note; You need to disable cloud-init in the template. To achieve this, add a shell provisioner to the build file with a local script file.

"provisioners": [{
  "type": "shell",
  "script": "post-script.sh"
  }
]

Then create the script file as below. Besides disabling cloud-init, I’ve added the fix for the VMware customizations issue KB56409. This issue’s symptoms are network-related; vSphere fails to connect the network adapter without the hotfix.

#!/bin/bash

# Fix VMware Customization Issues KB56409
sudo sed -i '/^\[Unit\]/a After=dbus.service' /lib/systemd/system/open-vm-tools.service
sudo awk 'NR==11 {$0="#D /tmp 1777 root root -"} 1' /usr/lib/tmpfiles.d/tmp.conf | sudo tee /usr/lib/tmpfiles.d/tmp.conf

# Disable Cloud Init
sudo touch /etc/cloud/cloud-init.disabled

Manage secrets with Pass

It’s always a challenge on how to manage credentials and other secrets when doing Infrastructure as Code. I need to protect my username and password when checking in to a code repo such as GitHub. If I check in the secrets in clear-text, anyone with access to the repo can exploit my credentials.

As I’m a single user, I’ve chosen the local password manager Pass. With this tool, each password lives in a gpg encrypted file.

Install Pass on Mac with homebrew brew install pass. If you need more info on how to install Pass, please visit the website passwordstore.org

With Pass, we can view the password file hierarchy:

And we can insert a new password:

We can show passwords

Use Pass in Terraform Configuration

We’ll use environment variables to set Terraform configuration variables. The environment variables need to be in the format TF_VAR_name, and Terraform will automatically pick it up.

In the configuration of Terraform, we declare the variables vsphereuser and vspherepass. To utilize Pass, we need to export the environment variable like this;

export TF_VAR_vsphereuser=$(pass vsphere-user)
export TF_VAR_vspherepass=$(pass vsphere-password)

Terraform

We’re all set with a Ubuntu 20.04 template, and our passwords are safe in the password manager. Let us move over to the fun part, create the Terraform configuration!

A Terraform project is any directory that contains tf files and has been initialized using the init command, which sets up Terraform caches and default local state.

We need a couple of files to get Terraform to do as we want, all these files uses the Hashicorp Configuration Language called HCL.

main.tf

This file contains the main configuration, and it’s here that you configure what resources and outputs you expect from Terraform.

# Create VM Folder
resource "vsphere_folder" "folder" {
  path          = "k8s"
  type          = "vm"
  datacenter_id = data.vsphere_datacenter.dc.id
}

variables.tf

This file declares the variables used in the project.

variable "example" {}

terraform.tfvars

This file automatically loads variables with defined values.

example = "example value"

Terraform CLI Commands

Once you have all the configuration set up and it’s time to execute, you need to be aware of Terraform’s different commands. I’ll walk you through the most commonly used ones.

The workflow of Terraform is to plan, then apply. You can use the plan in a CI/CD pipeline with applicable change management processes suitable for your organization. As the plan output clearly shows what changes you intend to make, it’s a powerful Terraform feature.

The workflow is like this;

terraform plan -out changes
terraform apply changes
terraform destroy

Plan

The terraform plan command utilizes the Terraform configuration files, compares the configuration with the current state, and shows you the changes needed to achieve the declared state. It does not perform any changes. It just presents a plan for making changes.

It’s possible to create a run file with the terraform plan -out command to be used with terraform apply.

Let’s say you create a run file with terraform plan -out my-changes you can quickly inspect the proposed changes with terraform show my-changes.

Apply

The terraform apply command applies the changes required to reach the desired state of the configuration or the run file from the terraform plan command described above.

Destroy

The terraform destroy command destroys the current infrastructure managed by Terraform.

Terraform Configuration

So, we’re ready to start deploying infrastructure. Below you’ll find the complete Terraform configuration. I’ll walk you through the most critical parts.

Configure vSphere Provider

Here we describe that the provider to be used in deploying our VMs in vSphere. Be sure to set allow_unverified_ssl = true if you’re using self-signed certs.

# Configure the vSphere provider
provider "vsphere" {
    user = var.vsphereuser
    password = var.vspherepass
    vsphere_server = var.vsphere-vcenter
    allow_unverified_ssl = var.vsphere-unverified-ssl
}

Collect information needed to deploy VM resources

We use data sources to collect data for use later in the configuration. For example, we fetch a cluster resource for later use. Maybe we have multiple clusters, and we need to define the one where we want to deploy.

data "vsphere_compute_cluster" "cluster" {
    name = var.vsphere-cluster
    datacenter_id = data.vsphere_datacenter.dc.id
}

Deploy VM Resources

OK, so here’s where the magic happens. Resources are the most crucial part of the configuration. Resources represent infrastructure objects such as VMs or virtual networks.

In this block, we describe three controller nodes to deploy. We use count.index to number the resources created. Besides configuring name, CPU, RAM, we also specify to clone the template we created with Packer. Inside the clone block, we have the customize block that leverages VMware Tools to customize the VM template with the hostname, network configuration, and DNS server.

To deploy worker nodes, we copy and paste this block and create a worker resource block.

resource "vsphere_virtual_machine" "control" {
    count = var.control-count
    name = "${var.vm-prefix}-controller-${count.index + 1}"
    resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
    datastore_id = data.vsphere_datastore.datastore.id
    folder = vsphere_folder.folder.path
    
    num_cpus = var.vm-cpu
    memory = var.vm-ram
    guest_id = var.vm-guest-id

    
    network_interface {
        network_id = data.vsphere_network.network.id
    }

    disk {
        label = "${var.vm-prefix}-${count.index + 1}-disk"
        size  = 25
    }

    clone {
        template_uuid = data.vsphere_virtual_machine.template.id
        customize {
            timeout = 0
            
            linux_options {
            host_name = "${var.vm-prefix}-controller-${count.index + 1}"
            domain = var.vm-domain
            }
            
            network_interface {
            ipv4_address = "10.128.128.11${count.index + 1}"
            ipv4_netmask = 24
            }

            ipv4_gateway = "10.128.128.1"
            dns_server_list = [ "10.128.128.80" ]

        }
    }
}

Output IP addresses

Terraform Output is used to print information to the console. It’s also possible to use output data in other Terraform modules.

output "control_ip_addresses" {
 value = vsphere_virtual_machine.control.*.default_ip_address
}

Running Terraform

In this short video I show you the steps to run the configuration decribed in this blog post, please enjoy! :)

youtube: https://www.youtube.com/watch?v=WjjCpUhBjXQ

Complete Configuration Files

main.tf

# main.tf

# Configure the vSphere provider
provider "vsphere" {
    user = var.vsphereuser
    password = var.vspherepass
    vsphere_server = var.vsphere-vcenter
    allow_unverified_ssl = var.vsphere-unverified-ssl
}

data "vsphere_datacenter" "dc" {
    name = var.vsphere-datacenter
}

data "vsphere_datastore" "datastore" {
    name = var.vm-datastore
    datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_compute_cluster" "cluster" {
    name = var.vsphere-cluster
    datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_network" "network" {
    name = var.vm-network
    datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_virtual_machine" "template" {
    name = var.vm-template-name
    datacenter_id = data.vsphere_datacenter.dc.id
}

# Create VM Folder
resource "vsphere_folder" "folder" {
  path          = "k8s"
  type          = "vm"
  datacenter_id = data.vsphere_datacenter.dc.id
}

# Create Control VMs
resource "vsphere_virtual_machine" "control" {
    count = var.control-count
    name = "${var.vm-prefix}-controller-${count.index + 1}"
    resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
    datastore_id = data.vsphere_datastore.datastore.id
    folder = vsphere_folder.folder.path
    
    num_cpus = var.vm-cpu
    memory = var.vm-ram
    guest_id = var.vm-guest-id

    
    network_interface {
        network_id = data.vsphere_network.network.id
    }

    disk {
        label = "${var.vm-prefix}-${count.index + 1}-disk"
        size  = 25
    }

    clone {
        template_uuid = data.vsphere_virtual_machine.template.id
        customize {
            timeout = 0
            
            linux_options {
            host_name = "${var.vm-prefix}-controller-${count.index + 1}"
            domain = var.vm-domain
            }
            
            network_interface {
            ipv4_address = "10.128.128.11${count.index + 1}"
            ipv4_netmask = 24
            }

            ipv4_gateway = "10.128.128.1"
            dns_server_list = [ "10.128.128.80" ]

        }
    }
}

# Create Worker VMs
resource "vsphere_virtual_machine" "worker" {
    count = var.worker-count
    name = "${var.vm-prefix}-worker-${count.index + 1}"
    resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
    datastore_id = data.vsphere_datastore.datastore.id
    folder = vsphere_folder.folder.path
    
    num_cpus = var.vm-cpu
    memory = var.vm-ram
    guest_id = var.vm-guest-id
    
    network_interface {
        network_id = data.vsphere_network.network.id
    }

    disk {
        label = "${var.vm-prefix}-${count.index + 1}-disk"
        size  = 25
    }

    clone {
        template_uuid = data.vsphere_virtual_machine.template.id
        customize {
            timeout = 0
            
            linux_options {
            host_name = "${var.vm-prefix}-worker-${count.index + 1}"
            domain = var.vm-domain
            }
            
            network_interface {
            ipv4_address = "10.128.128.12${count.index + 1}"
            ipv4_netmask = 24
            }

            ipv4_gateway = "10.128.128.1"
            dns_server_list = [ "10.128.128.80" ]
        }
    }
}

output "control_ip_addresses" {
 value = vsphere_virtual_machine.control.*.default_ip_address
}

output "worker_ip_addresses" {
 value = vsphere_virtual_machine.worker.*.default_ip_address
}

terraform.tfvars

# terraform.tfvars

# First we define how many controller nodes and worker nodes we want to deploy
control-count = "3"
worker-count = "3"

# VM Configuration
vm-prefix = "k8s"
vm-template-name = "ubnt-packer"
vm-cpu = "2"
vm-ram = "4096"
vm-guest-id = "ubuntu64Guest"
vm-datastore = "NVME"
vm-network = "VLAN128"
vm-domain = "virtjo.local"

# vSphere configuration
vsphere-vcenter = "vcsa.virtjo.local"
vsphere-unverified-ssl = "true"
vsphere-datacenter = "SDDC"
vsphere-cluster = "cluster01"

# vSphere username defined in environment variable
# export TF_VAR_vsphereuser=$(pass vsphere-user)

# vSphere password defined in environment variable
# export TF_VAR_vspherepass=$(pass vsphere-password)

variables.tf

# variables.tf
# Declare all the variables needed
variable "vsphereuser" {
    type = string
}
variable "vspherepass" {
    type = string
}
variable "vsphere-vcenter" {
    type = string
}
variable "vsphere-unverified-ssl" {
    type = string
}
variable "vsphere-datacenter" {
    type = string
}
variable "vsphere-cluster" {
    type = string
    default = ""
}
variable "control-count" {
    type = string
    description = "Number of Control VM's"
    default     =  3
}
variable "worker-count" {
    type = string
    description = "Number of Worker VM's"
    default     =  3
}
variable "vm-datastore" {
    type = string
}
variable "vm-network" {
    type = string
}
variable "vm-cpu" {
    type = string
    default = "2"
}
variable "vm-ram" {
    type = string
}
variable "vm-prefix" {
    type = string
}
variable "vm-guest-id" {
    type = string
}
variable "vm-template-name" {
    type = string
}
variable "vm-domain" {
    type = string
}

Build Ubuntu 20.04 VM on vSphere with Packer

2020-12-18T00:00:00+00:00

I have a home lab in my basement for learning purposes and to be able to create useful content on this blog. My next mission is to learn more about Kubernetes and advance my knowledge of IaC and Linux. For this, I need a consistent way to build and deploy Ubuntu VM’s. In this post, I’ll show you how I use Packer to create VM templates on vSphere.

What is Packer?

Let’s hear it from the source:

Packer is an open-source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

packer.io

So basically, you can use Packer to build VM’s from ISO’s based on code. Infrastructure as code, just how we like it. Also, it’s easy to implement in your CI/CD pipeline to go full hipster.

Packer builds the VM from the desired ISO and is also able to perform advanced provision tasks. Also, I’m able to inject my SSH public key to enable key-based authentication on the template.

I’ll later use this VM template as a source when deploying infrastructure resources with Terraform, but more on that in another blog post.

Install Packer

I will not go through the installation process in this blog. Please visit the getting started guide on packer.io if you need help installing. You’ll find it here.

Create the Build

In my home lab, I’m now able to build a fresh template in ~12 minutes. That’s pretty good!

You’ll need these files. The .json file is where you configure settings for Packer to build and provision the VM, connect to vSphere, and so on. The user-data and meta-data files are used for unattended installation of Ubuntu. Although only the user-data files contain information, both must be present. Packer uses these files and mounts them as CDROM’s to the vSphere guest VM.

❯ tree .
.
├── http
│   ├── meta-data
│   └── user-data
└── ubuntu-2004.json

Also, you’ll need to upload an ISO to the vSphere datastore and edit the iso_url variable. I’ve uploaded the ISO to my NVME datastore, as you see here.

Download Ubuntu Server ISO

"iso_url": "[NVME] ISO/ubuntu-20.04.1-live-server-amd64(1).iso"

Important code snippets in the build file

SSH Timeout

Be sure to add this snippet to workaround the issue.

"ssh_timeout": "20m",
"ssh_handshake_attempts": "100",

Generate Password

> docker run --rm -ti alpine:latest mkpasswd -m sha512
Password: 
$6$eZsKuZ3lwrfwvrJ3$FSmB.RPl9doTTe57fHDvbMjyu/wFZOcNfuGZ82id4ZZYbUZ0P9GP6ZWamYK3JZzwnMIjL1WEVczqHf03.IqYS/

Configure Identity

Edit the user-data file with the newly created password.

#user-data
identity:
  hostname: ubuntu-packer
  username: ubnt
  password: $6$RAVeOPg.9rJylcnK$tm6KVpRXxTGb0et.1PQcmLVCFLjIVaLye.Weiizxw3ecGRI2kA0JH7nFxkJCz4oeZphSO4qzYSwncE/s34UHg0

SSH Settings

First, you’ll need your SSH public key if you want to enable key-based authentication in your template.

❯ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com

Secondly, inject the key in the user-data file.

#user-data
ssh:
  install-server: yes
  allow-pw: true
  authorized-keys: 
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com

Build

When you’ve created the required files and filled in all the variables, you’re ready to build the VM. Use the below command to start the build. If you’re overwriting the VM with each iteration, use the -force flag.

> packer build -force ubuntu-2004.json

When you’re done, you’ll find the template in vSphere ready for use!

Required Files

ubuntu-2004.json

{
    
  "variables": {    
    "vsphere-server": "vcsa.virtjo.local",
    "vsphere-user": "administrator@vsphere.local",
    "vsphere-password": "SuperSecurePassword",
    "vsphere-datacenter": "SDDC",    
    "vsphere-cluster": "esxi01.virtjo.local",    
    "vsphere-network": "VLAN128",
    "vsphere-datastore": "NVME",        
    "vm-name": "ubnt-packer",
    "vm-cpu-num": "1",
    "vm-mem-size": "1024",
    "vm-disk-size": "25600",    
    "iso_url": "[NVME] ISO/ubuntu-20.04.1-live-server-amd64(1).iso"  
  },
  "builders": [
    {
      "type": "vsphere-iso",
      "vcenter_server": "{{user `vsphere-server`}}",      
      "username": "{{user `vsphere-user`}}",      
      "password": "{{user `vsphere-password`}}",
      "insecure_connection": "true",
    
      "datacenter": "{{user `vsphere-datacenter`}}",
      "cluster": "{{user `vsphere-cluster`}}",      
          

      "datastore": "{{user `vsphere-datastore`}}",  
      "guest_os_type": "ubuntu64Guest",
      "CPUs": 1,
      "RAM": 1024,
      "RAM_reserve_all": false,
      "disk_controller_type": "pvscsi",
      "storage": {
        "disk_size": 15000,
        "disk_thin_provisioned":true
      },
      "network_adapters": {
        "network": "{{user `vsphere-network`}}", 
        "network_card": "vmxnet3"
      },

      "vm_name": "{{user `vm-name`}}",      
      "notes": "Build via Packer",            
      "convert_to_template": true,

      "ssh_username": "ubnt",      
      "ssh_password": "ubnt",
      "ssh_timeout": "20m",
      "ssh_handshake_attempts": "100",

      "iso_paths": ["{{user `iso_url`}}"], 

      "cd_files": ["./http/user-data", "./http/meta-data"],
      "cd_label": "cidata",
      "boot_wait": "2s",
      "boot_command": [
        "<enter><wait2><enter><wait><f6><esc><wait>",
        " autoinstall<wait2> ds=nocloud;",
        "<wait><enter>"
      ],
      "shutdown_command": "echo 'ubnt'|sudo -S shutdown -P now" 
    }
  ]
}

user-data

#cloud-config
autoinstall:
  version: 1
  locale: en_US
  keyboard:
    layout: en
    variant: us
  network:
    network:
      version: 2
      ethernets:
        ens192:
          dhcp4: true
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubuntu-packer
    username: ubnt
    password: $6$RAVeOPg.9rJylcnK$tm6KVpRXxTGb0et.1PQcmLVCFLjIVaLye.Weiizxw3ecGRI2kA0JH7nFxkJCz4oeZphSO4qzYSwncE/s34UHg0
  ssh:
    install-server: yes
    allow-pw: true
    authorized-keys: 
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi9eAu6KBaShdcL4pxi6/sJp+IS6nCKexcjQdwFLxg+EoiT2MTAnMsjnfi570het+VV+iOigcZLuRwEcAPh6rSQOtpikmpV6WFjzToWq9aUxDrxWsp/iEPHp+sbjrlsdnGvLGY9XhmPs9s5I8xFQbwF6ilhMIQm+RxtGJJuPUWaF+uXo+3CB91A6bK/rjs97iAjrPZRs0vo5hJGqrIGFi3WP9hf8hF9oWz2BiLRYBib3il6lsAl4Ca0sI//gNM0Ztj4gB7qv1+uPz157bk0IZoN285/72l/rUZVSPIwO+QFZFK07FsyVrpAgMlHk65BiSAO4DtolZEArfXRE1g1DH/ mail@example.com
  user-data:
    disable_root: false
  late-commands:
    - echo 'ubnt ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu

Trident - Persistance in an Ephemeral World

2020-10-02T00:00:00+00:00

We all know Kubernetes is winning the container orchestration battle. We’ve seen a shift in application development and operations. Historically, software vendors shipped .EXE files to customers, small and large. The update cycle meant a new .EXE release every 6-12 months, or something similar.

Now, the reality is that more and more software vendors turn to containers as shipping their products. Containers provide the ability to re-create the application regardless of the environment, both in development and production. No more need to install the correct dependencies on each development machine and production environment; just run docker, and you’re ready to go.

All and well, but both you and I know that containers are stateless. The idea with Kubernetes is to spin up as many pods as you need and if one dies, caused by node failure or anything else, k8s spins up a brand new one, and the old one remains dead.

Many organizations approach Kubernetes with the idea that everything stateful lives outside the realm of Kubernetes. Often in traditional VMs in VMware, things like databases, file repositories, and different application backends.

But, why build silos when persistence is a well-functioning feature in Kubernetes?! Many storage providers hook into the Kubernetes API as CSI’s to provide persistent storage. Even if the node dies and brings the pods with it, the CSI plugin will keep the storage persistent, and when a new pod spins up, the persistent volume (PV) will attach to the fresh container/pod.

Persistence in Kubernetes also enables the traditional IT-department to be a part of the application development team (DevOps!!) process.

In this blog post, I’ll specifically talk about the NetApp CSI called Trident, and I’ll show you some benefits of persistent storage in Kubernetes.

Ok, so you may know NetApp from before, or you’re this is the first time, regardless, Trident is powerful when it comes to persistent storage in k8s. Hell, NetApp is the king of storage irrespective of the application!

What is Trident?

Trident is fully open-source and maintained by NetApp. Even if Trident currently works only with NetApp backends, you could, in theory, extend the project to support other backends.

In short, Trident is the middle man when the developer requests a volume that should persist in container failure. Storage administrators specify what storage should be available to the developer (Storage Class, SC), and the developer then requests that storage as a volume (Persistent Volume Claim, PVC).

Why should you go that extra mile to use NetApp storage with Trident instead of just spinning up an NFS-server to handle the storage? Well, NetApp has very compelling features when it comes to managing data.

Just a shortlist of features that add real value to your application environment.

Instantly clone volumes
Resize volume in production
Snapshot the volume
Import volumes (great for DR and portability!)
Cloud Volumes support (deploy Trident in GKE and leverage NetApp Cloud Volumes in GCP)

Besides that, the underlying NetApp storage adds efficiency features to save money, security features like encryption at rest, and a lot more! Talk to your infrastructure team; they know all of these features by heart.

Where do I find more information on Trident?

GitHub Repository: https://github.com/NetApp/trident
Trident Documentation: https://netapp-trident.readthedocs.io/en/stable-v19.01/index.html
Using Trident with Kubernetes and ONTAP Lab: https://labondemand.netapp.com/lab/sl10556
Trident on thePub: https://netapp.io/persistent-storage-provisioner-for-kubernetes/

Nigel Poulton shows how to deploy GKE with Trident and Cloud Volumes

youtube: https://youtu.be/KxUeyaASQVc

Deploy VMware Cloud Director with PowerShell

2020-06-04T00:00:00+00:00

In my home lab, I like to automate as much as possible so I can bring stuff up/down quickly. To achieve this I use several tools such as Ansible, Terraform and PowerShell. This time I decided on using PowerShell to quickly bring up VCD with the same settings every time.

This script deploys the appliance and please, edit the settings to fit your environment.

First, you need to download the VCD Appliance 10.1.1 and point the $ovfPath variable to the .ova file.

Then, adjust the rest of the variables to fit your environment.

Finally, run the script! Should look something like this,

Below, you’ll find the full code.

$viserver = "vcsa.lab.local"
$ovfPath = ".\VMware_Cloud_Director-10.1.1.4916-16288798_OVF10.ova"
$clusterName = "vSphere-Cluster"
$datastoreName = "vsanDatastore"
$VMName = "vcd.lab.local"
$secretPassword = "P@ssw0rd"
$adminEmail = "admin@lab.local"

$networkPortGroup = "sddc-network"
$networkGateway = "192.168.10.1"
$networkNetmask = "255.255.255.0"
$networkVCDAddress1 = "192.168.10.50"
$networkVCDAddress2 = "192.168.10.51"
$networkNTPServer = "192.168.10.40"
$networkDNSServer = "192.168.10.40"
$searchDomain = "lab.local"
$NFSMount = "192.168.10.45:/mnt/nfsexport"


Connect-VIServer $viserver

# VMware Resources
$Cluster = Get-Cluster -Name $clusterName
$VMHost = Get-Cluster $Cluster | Get-VMHost | Select -First 1
$Datastore = Get-Datastore -Name $datastoreName
$DiskFormat = "Thin"


$ovfConfig = Get-OvfConfiguration -Ovf $ovfPath

# ======================================================================
# OvfConfiguration: VMware_Cloud_Director-10.1.1.4916-16288798_OVF10.ova

#    Properties:
#    -----------
#    DeploymentOption
#    IpAssignment
#    NetworkMapping
#    vami
#    vcloudapp
#    vcloudconf
#    vcloudnet

## --- OVF Configuration

$ovfConfig.DeploymentOption.Value = "primary-small"

# Network mapping
$ovfConfig.NetworkMapping.eth0_Network.Value = $networkPortGroup
$ovfConfig.NetworkMapping.eth1_Network.Value = $networkPortGroup

# Network Configuration
$ovfConfig.IpAssignment.IpProtocol.Value = "IPv4"
$ovfConfig.vami.VMware_vCloud_Director.gateway.Value = $networkGateway
$ovfConfig.vami.VMware_vCloud_Director.domain.Value = $VMName
$ovfConfig.vami.VMware_vCloud_Director.searchpath.Value = $searchDomain
$ovfConfig.vami.VMware_vCloud_Director.DNS.Value = $networkDNSServer
$ovfConfig.vami.VMware_vCloud_Director.ip0.Value = $networkVCDAddress1
$ovfConfig.vami.VMware_vCloud_Director.netmask0.Value = $networkNetmask
$ovfConfig.vami.VMware_vCloud_Director.ip1.Value = $networkVCDAddress2
$ovfConfig.vami.VMware_vCloud_Director.netmask1.Value = $networkNetmask

# vCloud App Settings
$ovfConfig.vcloudapp.VMware_vCloud_Director.ntp_server.Value = $networkNTPServer
$ovfConfig.vcloudapp.VMware_vCloud_Director.varoot_password.Value = $secretPassword
$ovfConfig.vcloudapp.VMware_vCloud_Director.expire_root_password.Value = $false
$ovfConfig.vcloudapp.VMware_vCloud_Director.nfs_mount.Value = $NFSMount
$ovfConfig.vcloudapp.VMware_vCloud_Director.enable_ssh.Value = $true

# vCloud Global Conf
$ovfConfig.vcloudconf.VMware_vCloud_Director.ceip_enabled.Value = $false
$ovfConfig.vcloudconf.VMware_vCloud_Director.db_pwd.Value = $secretPassword
$ovfConfig.vcloudconf.VMware_vCloud_Director.admin_pwd.Value = $secretPassword
$ovfConfig.vcloudconf.VMware_vCloud_Director.admin_email.Value = $adminEmail

Import-VApp -Source $ovfPath -OvfConfiguration $ovfConfig -Name $VMName `
-VMHost $VMHost -Location $Cluster -Datastore $Datastore `
-DiskStorageFormat $DiskFormat -Confirm:$false

Enable Hyperthreading in BIOS with Ansible on VMware hosts

2020-03-22T00:00:00+00:00

Recently, I found a configuration issue on a VMware cluster running on DellEMC hardware. Initially, they were vulnerable to the L1 Terminal Fault CVE-2018-3646, to mitigate we disabled HT in the BIOS of the hosts. As time passed, VMware released software updates to mitigate these issues. We did install the patches, but the disabled HT stayed the same. When the problem came to light, I decided to put Ansible to work as I did not want to perform manual work on the entire cluster. So the adventure begins!

First, if you’re on an affected CPU architecture, Intel Skylake in our case, you need to verify that you’ve implemented the Side-Channel-Aware Scheduler in VMware. I’m not going to go into detail in this post, so please read more at https://kb.vmware.com/s/article/55806.

Secondly, you need Ansible installed and also the OpenManage Ansible Modules and the OpenManage Python SDK. I will not invent the wheel for this as jonamiki.com has got an excellent blog post about the entire procedure http://jonamiki.com/2020/02/18/ansible-with-dell-poweredge-servers/.

Let’s get down to it.

Some things to notice in the play, it’s executed in a rolling fashion and takes two hosts at a time, set the hosts into maintenance mode, applies the change, reboots and takes out of maintenance mode. Also, if no changes are made to the BIOS, the play will continue without rebooting the host. The reason for the reboot is that the BIOS changes are only applied upon reboot.

You need to bring your variables into the play, I will not post mine here, but if you are somewhat familiar with Ansible you will figure it out.

One word of caution, I did not find a pretty way of confirming the host was online in vCenter after the reboot, so I just decided to wait 20 minutes and assume the host was online again. As I monitored the entire process, for me, this was not an issue.

---
- hosts: vmware
  connection: local
  name: Enable HT
  gather_facts: False
  serial: 2

  tasks:
  - name: Enable HT in BIOS
    redfish_config:
        category: Systems
        command: SetBiosAttributes
        bios_attributes:
            LogicalProc: "Enabled"
        baseuri: "{{ idrac_ip }}"
        username: "{{ idrac_username }}"
        password: "{{ idrac_password }}"
    register: ht
    
  - name: Enter maintanence mode
    vmware_maintenancemode:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      esxi_hostname: "{{ inventory_hostname }}"
      validate_certs: False
      state: present
    when: ht.changed

  - name: Create BIOS configuration job
    idrac_redfish_command:
        category: Systems
        command: CreateBiosConfigJob
        baseuri: "{{ idrac_ip }}"
        username: "{{ idrac_username }}"
        password: "{{ idrac_password }}"
    when: ht.changed

  - name: Reboot host
    vmware_host_powerstate:
        hostname: '{{ vcenter_hostname }}'
        username: '{{ vcenter_username }}'
        password: '{{ vcenter_password }}'
        validate_certs: no
        esxi_hostname: "{{ inventory_hostname }}"
        state: reboot-host
    register: reboot_host
    when: ht.changed

  - name: Wait
    wait_for:
        timeout: 1200
    when: reboot_host.changed


  - name: Exit maintanence mode
    vmware_maintenancemode:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      esxi_hostname: "{{ inventory_hostname }}"
      validate_certs: False
      state: absent

  - name: Wait
    wait_for:
        timeout: 30

VCAP Design Study Notes

2020-02-19T00:00:00+00:00

In the year 2019(it feels like a million years ago, right?!), I started my VMware Certification journey. In short, I passed my first VCP6, renewed it to 2019, and passed the VMware Cloud Provider Specialist exam. After a successful 2019, I went on parental leave from November to February. Now, when I’m back in action, I need to continue expanding my knowledge and have new goals to achieve in 2020. At first, I thought about going after the AZ-103 exam because some of my colleagues are going down the Azure route, but, as I’m already a VCP, I think it’s wrong not to continue to the advanced certifications. I will start with the VCAP6.5-DCV design, and when I’m successful, I’ll continue with the deploy exam. In this post, I’ll share some of my study notes and links to the resources I use to prepare myself.

Ok, so let us dive in. The design exam verifies that people in the industry understand and can practice VMware architectural principles. I’ll start by explaining the two most essential pieces in architecture, RAMPS, and RCAR’. I’ll also touch on NFR’s and FR’s. If these abbreviations make no sense to you, read on, and I’ll explain.

RAMPS and RCAR’s

So, these two go hand in hand. You could say that each topic in RAMPS maps to an RCAR. Let’s say you have a requirement that dictates that you need to manage all the storage arrays from one centralized console. That requirement fits into the Manageability of RAMPS. Likewise, if you identify one or more risks in the design, you could map them to either of the RAMPS categories.

RAMPS

Let’s start! I’ll cover RAMPS first then we’ll continue with RCAR’s.

R = Requirement

The requirements of the design dictate what the result will be. These are must-haves in the design.

“The solution needs to accommodate 150 concurrent VDI users.”

While we’re talking about requirements, I’ll mention NFR’s and FR’s, more precisely Non-Functional Requirements and Functional Requirements.

Functional Requirement

These requirements are more of the “hard” kind. They describe what the solution must be able to deliver. They are specific and often relate to regulations and compliance.

“The design needs to meet the requirements of HIPAA.”

Non-Functional Requirement

The NFR’s are softer. They are more general and describe how the solution should behave. Often they describe the performance metrics that the solution should achieve.

“We need sub-millisecond latency on the storage array.”

C = Constraint

Constraints limit your design choice. The difference between a constraint and a requirement is that the constraint is limiting and prevents us from choosing a design that also could fill the requirements.

“We need to maintain our relationship with NetApp. You will use NetApp arrays in the solution.”

A = Assumption

Assumptions are something we trust to be true, but we don’t have confirmation that it’s true. The goal is to keep assumptions to zero at the end of the design phase.

“The co-location racks have sufficient cooling and power.”

RCAR

Ok, so let’s continue sorting out the business of RCAR’s

R = Risk

Risks may prevent you from achieving the result. Risks may not be obvious when the project starts, but it’s important to discover all of the risks and mitigate them accordingly. The goal is to mitigate all the risks.

The RCAR’s are rather self-explanatory, but I’ll summarize them for you.

R = Recoverability

Recoverability means you need to think about how to recover from a disaster. Some things you need to think about are the RTO(Recovery Time Objective) and the RPO(Recovery Point Objective), in short, they dictate how much downtime and how far back in time you need to go in case of recovery from disaster.

A = Availability

Availability means that your solution needs to be available, the are different levels of availability often dictated by SLA’s(Service Level Agreements). The SLA’s are defined in nines, ex: 99,99% uptime. You need to take out the calculator and calculate the downtime allowed in minutes each month/year. To keep the solution available, you should look into redundancy, capacity & performance.

M = Manageability

Manageability describes how you manage the solution. The buzzword often used in manageability is “single pane of glass.” Often the customer wants to have easy management of the installed solution. But, also, the management often needs to be secure, trackable, and responsive.

S = Security

Maybe the most self-explanatory of all the RCARs. Security requirements dictate how secure your environment should be. Security risks mean something is not properly set up, and you risk to have availability issues or something like data theft.

Port Forward on vCloud Director with Terraform

2019-12-24T00:00:00+00:00

My Terraform adventures continue! In my last post, I covered how to provision a VM on vCloud Director with the help of Terraform. I’m just at the beginning of my IaC journey, but I already see the benefits of using this method to deploy infrastructure. The main advantage of using IaC is that you easily can re-use the code on multiple similar projects. When you start a new project, it’s just a matter of copying your codebase and modifying it to fit the current project. Other advantages are documentation in code, ability to have CI/CD pipeline that ties into a managed change process with approvals.

In this post, I’ll show you how to use terraform code to open ports in the vCloud Director tenant Edge. In the vCloud Edge, you need to both open the port in the firewall and create the correct NAT rule to translate the external IP to the internal one. In this example, I need to open several ports, so I’ll create a list object and loop over it in the code. Let’s get started!

This post will focus primarily on the code to deploy resources, firewall & NAT-rules. I will not go into the depth of how to get started with terraform or how terraform works. I’ve already written a post on that.

Below you’ll find the code we use:

# variables declared in tfvars-file
variable "vcd_user" {}
variable "vcd_pass" {}
variable "vcd_org" {}
variable "vcd_vdc" {}
variable "vcd_url" {}
variable "network_name" {}
variable "ext_network_name" {}
variable "edge_name" {}
variable "vcd_allow_unverified_ssl" {
    default = true
}

# Create the list of ports to open
variable "ports" {
  type = list(string)
  default = ["443", "80", "25", "587", "995", "110", "465", "143", "993"]
}

# Configure the VMware vCloud Director Provider
provider "vcd" {
  user                 = var.vcd_user
  password             = var.vcd_pass
  org                  = var.vcd_org
  vdc                  = var.vcd_vdc
  url                  = var.vcd_url
  allow_unverified_ssl = var.vcd_allow_unverified_ssl
}

# Create the firewall rules
resource "vcd_nsxv_firewall_rule" "rules-ports" {
  for_each = toset(var.ports)
  edge_gateway = var.edge_name
  
  name = "mailsrv01 allow ${each.value} in"
  source {
    ip_addresses = ["any"]
  }

  destination {
    ip_addresses = ["207.234.154.120"]
  }

  service {
    protocol = "tcp"
    port = each.value
  }
}

# create the NAT-rules
resource "vcd_nsxv_dnat" "nat-ports" {
  for_each = toset(var.ports)

  edge_gateway = var.edge_name
  network_type = "ext"
  network_name = var.ext_network_name

  original_address   = "207.234.154.120"
  original_port = each.value

  translated_address = "10.23.44.16"
  translated_port = each.value

  protocol = "tcp"

}

To deploy the infrastructure, you should use the terraform workflow, as usual, init, validate, plan, apply. I’ll now go over the different script blocks to explain what is happening.

We create a list of strings, and this is all our port numbers to open. Later on, we will loop over this variable to create multiple rules.

variable "ports" {
  type = list(string)
  default = ["443", "80", "25", "587", "995", "110", "465", "143", "993"]
}

Here we create the resource of type firewall rule, and the one thing you may see differs from a usual block of this resource type is the for_each line. The only thing you need to think about is that for_each only takes a set variable as a parameter, so we need to transform the list of strings into a set.

When we have all this in place, Terraform creates one resource for each value in the list. You access the value in the loop by the each.value variable. We do the same for the NAT-rule resource creation.

resource "vcd_nsxv_firewall_rule" "rules-ports" {
  for_each = toset(var.ports)
  edge_gateway = var.edge_name
  
  name = "mailsrv01 allow ${each.value} in"
  source {
    ip_addresses = ["any"]
  }

  destination {
    ip_addresses = ["207.234.154.120"]
  }

  service {
    protocol = "tcp"
    port = each.value
  }
}

Create VM on VMware vCloud Director with Terraform

2019-11-20T00:00:00+00:00

One of the things I enjoy the most in life is to learn new exciting stuff, the part I like the most is how to create. Whether it’s in the physical world such as carpentry or homebrewing(I love to brew beer!), or if it’s in the virtual computer world where I like to create infrastructure. On the other hand, I also enjoy sharing that knowledge with other people! In this post, I’ll share how to create a virtual machine in a software-defined data center(running vCloud Director) with the help of Terraform.

Infrastructure as Code Let’s start by explaining what Infrastructure as Code(often referred to as IaC) is. IaC is like a lifestyle where you choose to define your infrastructure in code. The philosophy is to deploy infrastructure consistently and predictably(and rapidly!). Wikipedia says the following: “The value of IaC can be broken down into three measurable categories: cost (reduction), speed (faster execution) and risk (remove errors and security violations).”

There’s a lot of different IaC tools out there in the wild. The main products are Chef, Puppet, Ansible, SaltStack, & Terraform. They all work in different ways, but all hold the IaC philosophy close at heart. I’ve chosen Terraform as my weapon today because it’s open-source, written in Go, and supports all of the major cloud providers, including VMware vSphere(& vCD!).

Install Terraform

If you’ve not installed Terraform yet it,s no problem, the installation process is straightforward. Because it’s written in Go, all you need is the executable and the correct string in your PATH variable. To make the whole process more comfortable, I use brew on my Mac for this.

Mac users:

brew install terraform

Other users:

I leave it up to you to find out how to install on your system.

You need to create these two files:

jumphost.tf

I’m creating a VM that will be my jump host in the environment. This file is my base Terraform configuration file.
terraform.tfvars

This file contains all of my variable data. You should add this to your .gitignore file! This file is private!

Files created by Terraform:

terraform.tfstate

This file is called the state file. The content of this file is the state of your infrastructure. Do not manipulate this file in any way! The state is used to map real-world resources to your configuration.
terraform.tfstate.backup

Automatic backup of the state file created by the Terraform CLI.

johnhenriksson@workbook jumphost % tree
.
├── jumphost.tf
├── terraform.tfstate
├── terraform.tfstate.backup
└── terraform.tfvars

0 directories, 4 files

Let’s create that config!

Note that these examples are my labs deployed on Konica Minolta’s IaaS KMSky. These are just examples, and I’ve deleted them after this project.

jumphost.tf

Let’s start with the main configuration file. Here I’ve first declared what variables to use. I then later add the variable data to the .tfvars file.

To create a VM, I first need to get the network that I want to connect to the VM. I also need to obtain or provision a vApp for this VM. In this example, I get the network but create the vApp.

# variables
variable "vcd_user" {}
variable "vcd_pass" {}
variable "vcd_org" {}
variable "vcd_vdc" {}
variable "vcd_url" {}
variable "vcd_allow_unverified_ssl" {
    default = true
}

# Configure the VMware vCloud Director Provider
provider "vcd" {
  user                 = var.vcd_user
  password             = var.vcd_pass
  org                  = var.vcd_org
  vdc                  = var.vcd_vdc
  url                  = var.vcd_url
  allow_unverified_ssl = var.vcd_allow_unverified_ssl
}

# Get the vcd network as a data source
data "vcd_network_routed" "net" {
  name = "workplace-network"
}

# Create the vApp
resource "vcd_vapp" "jumphost" {
  name = "jumphost"
}

# Create the VM in the vApp
resource "vcd_vapp_vm" "jumphost" {
  vapp_name = vcd_vapp.jumphost.name
  name = "jumphost01"
  catalog_name = "KMsky IAAS Templates - Sweden"
  template_name = "Windows Server 2016 Datacenter - Basic"
  
  computer_name = "jumphost01"
  memory = 4096
  cpus = 2
  cpu_cores = 1

  # Map the network from the data source to the VM
  network {
      name = data.vcd_network_routed.net.name
      type = "org"
      ip = "10.10.0.10"
      ip_allocation_mode = "MANUAL"
      is_primary = true
  }

  depends_on = [vcd_vapp.jumphost]
}

terraform.tfvars

This file is the variable file where you need to specify the details suitable for your environment. You can name the file whatever you like as long as it’s file-ending is “.tfvars”. Terraform loads all the tfvars files at execution.

vcd_user = "username"

vcd_pass = "secretpassword"

vcd_org = "HomelabOrg"

vcd_vdc = "HomelabVDC" 

vcd_url = "https://vcloud.home.lab/api"

vcd_allow_unverified_ssl = true

Run Terraform

The last step is to run the terraform configuration you’ve created. The steps are simple. You just run the command terraform apply, and Terraform then checks the state, prompts for verification, and then pushes the changes.

There is also the option to run the command terraform plan to perform a sort of dry-run. The plan command tells you what’s about to happen when you apply. This workflow could also be useful in a change management routine where you quickly can provide the changes to your manager.

Revert changes

Now you’ve successfully applied some new infrastructure with the help of code, congratulations! But, how do I perform lifecycle management on this infrastructure? When it’s time for the app to be decommissioned and the infrastructure to be removed, how do you do that? Well. Lucky for you! Terraform has a command to remove the resources you’ve created. To remove the resources, run the command terraform destroy, and everything is as it was before you created the environment.

Video

I’ll include a simple video of me running the terraform apply command. youtube: https://www.youtube.com/watch?v=3TSa0cSpF1o

Screenshots

Empty vCloud Tenant DC - Before Terraform vApp Created from Terraform VM inside vApp created from Terraform

VMware Proactive HA

2019-11-14T00:00:00+00:00

You are a VMware Administrator. In your data center, there’s a vSphere 6.0 cluster running rock-solid! But, with the 6.0 end of support coming up March 12, 2020, you’re forced to look into the newer version of vSphere. You may have noticed that there’s been a couple of improvements and new features since 6.0 came out. In this post, I’ll walk you through the feature called Proactive HA, released in 6.5.

In short, Proactive HA is and extension of DRS that uses the vendor Health Provider to move VM’s off degraded hosts. You may think, but why?? Why does it have HA in the name if it’s not part of vSphere HA?! Well, it is not part of the vSphere HA architecture, and I’ll explain the difference. But, you do enable and configure it in the same GUI window as vSphere HA.

There are a couple of different states reported from the health provider, and it’s up to you as an admin to configure the responses. The severity levels are; Healthy, Moderate Degradation, Severe Degradation, and Unkown. I’ll jump directly to the recommended actions, and they are, Quarantine mode for Moderate and Maintenance mode for severe failure. In reality, the moderate state could be that one of ten fans failed, and the severe level could be a critical memory error.

Quarantine mode is where no new VM’s are allowed to run on the host, but the ones running will be allowed to stay on the host if the cluster resources are overcommitted.

Maintenance mode is straightforward. All VM’s will be moved off the host.

##To enable Proactive HA, you need the following:

vCenter Server & ESXi 6.5 or above
Enterprise Plus or Platinum license
Supported Health Provider plugin from your hardware vendor

The licensing is straight forward, but what the heck is “Health Provider plugin” ?! It’s the hardest part with Proactive HA. The plugin is the middle-man between the hardware layer and vCenter. The plugin is hard to find from most hardware providers. I advise you to reach out to your hardware vendor or partner.

##Steps to enable Proactive HA:

Edit “vSphere Availability” on the cluster
Select “Turn on Proactive HA”
Click “Proactive HA Failures and Responses”
Modify the settings to:
- Automation Level: Automated
- Remediation: Quarantine mode for moderate and Maintenance mode for severe failure (Mixed).

Top 5 NetApp blogs to follow in 2019

2019-11-10T00:00:00+00:00

If you’re like me, eager to learn and absorb all of the knowledge, you need to have a couple of useful blogs to follow. In this post, I list 5 of my favorite NetApp blogs that I follow regularly.

IOPS.ca

This blog by Chris Maki is easy to consume with it’s relatively short and relevant posts. This blog focus on NetApp products and features.

Why do you like WAFLs?

This blog is, in my opinion, the best place to learn fundamental concepts of SAN, NAS & ONTAP. I used this blog as preparation for both my NCDA & NCIE exam.

thePub

The one-stop for DevOps stuff! It covers Kubernetes, Trident & Automation. This blog should be your home page if you’re working as a NetApp DevOps.

WHY IS THE INTERNET BROKEN?

Home of the famous podcast! Besides the pod-updates, the posts are interesting, and often we see sneak peeks from features to come.

EDIT 2019-11-11

I’m honestly quite embarrassed. After posting a link to this post on Reddit, one excellent user, ragingpanda, made me aware that I only listed 4, not 5 links in this post.

This issue is certainly something we need to correct! I did get some extra tips in the Reddit post as well, so I’ll make it easy for myself, I’ll list the blog sites below. If you got some other NetApp blog, I shouldn’t miss, or you feel it should be on this list, hit me up in the comments below!

DerScmitz.com

Words of Steiner

NetApp Influencer of the Year 2019

2019-10-26T00:00:00+00:00

A couple of days ago, the yearly NetApp partner event in Sweden took place. When I got the invitation, I immediately accepted. But, days before, my kids got both chicken-pox and otitis at the same time, I was forced to cancel.

The day after the event took place, I got an email from one of the organizers that I did win the “Influencer of the Year”-award.

Although I’m regretful that I couldn’t attend, I’m incredibly grateful to be recognized for my efforts! I did not expect to win a prize of any kind, so I’m still in chock.

The motivation for the award was something like this(translated from Swedish):

“With a strong interest in the modern data center, he has taken up the flag in the important work of leading and launching the VMware User Group Sweden, and in a nuanced way, highlight NetApp.”

VMUG Sweden Event October

2019-10-04T00:00:00+00:00

First of all, I would like to thank everyone that attended our first VMUG event in Sweden. Also, a BIG thank you to our sponsors!

At first, I was troubled that everyone would be no-shows. Then, I got overwhelmed at how many that showed up! When we had our first meeting planning the event, we had a plan and conference room for about ten people, but we were roughly 35 attendees!

The first of the two sessions were speakers from VMware. They did an excellent presentation on all of the news from VMworld US. Mainly, the first session focused on VMware Tanzu and Project Pacific.

The second and last session was Pontus from Region Uppsala(Swedish healthcare region). He also did an exceptional presentation! He presented on their project, going from Hyper-V to VMware Cloud Foundation in less than a year. The main goal of the entire project from the start was to automate and provide self-service to the internal application owners. It quickly became much more than that, and they replaced the legacy converged infrastructure to modern hyper-converged vxRail systems. It was a fascinating story!

Lastly, VMware got us some pizza and beer! I met a bunch of new people that are facing the same daily challenges as me. The people in the VMware community are great! Hopefully, we will have another event soon! And, please let me know what you would’ve liked to hear about next time! Send an email to sweden@vmug.com with your thoughts and ideas.

On a different note, I started this morning strong! I successfully renewed my VCP-DCV certification. This time I did the delta exam, consisting of 40 questions over 75 minutes. The exam was challenging in many areas, but I enjoyed the experience! I highly recommend that you renew your certification if you haven’t had the time yet.

Here are some pictures from the event:

NetApp NCIE-SAN NS0-509 Review

2019-09-16T00:00:00+00:00

I’ve stepped out of the certification room, super excited to have passed the NS0-509 exam. It was a challenge! I did pass the NCSA (NS0-160) exam before the summer, so I had some experience in how the NetApp certification program works. I’ll share my experience with the NS0-509 exam in this blog post.

Let me give you some background about myself. I started working with storage in about 2015 as a VMware admin. We had an older VMware cluster running iSCSI with HP LeftHand arrays. But, we got the task to install a new VMware-environment running DELL hardware exclusively. As a new VMware admin, it was a challenge to bring up everything, including networking, compute, and storage. Eventually, we sorted everything out and had a VMware cluster running with iSCSI storage.

About a year ago, I stumbled on to NetApp storage. I worked with a project for approx three months as a NetApp admin. The environment consisted of a stretched VMware cluster with NetApp arrays in MetroCluster configuration. After this, I have a real taste for learning NetApp.

So, enough with the jibber-jabber. The NS0-509 exam is created to prove that you have enough knowledge to implement, troubleshoot, and maintain a SAN environment. It has nothing to do with data protection. There is a dedicated certification path just for that.

Sometimes the sentences were a bit tricky, and it felt that they tried to trick you into answering with a wrong answer. But, overall, I thought that the questions were right, challenging, and evenly spread over the subject areas. You need to have a good understanding of SAN infrastructure to pass this exam. The most important things are multipathing, zoning, port specifications, and compatibility.

The things that stood out in this version of the NCIE exam is the NVMe over fabrics protocol. You need to prepare for that and have knowledge of how namespaces, subsystem, NQN, and ANA works.

The NetApp tools you need to study are (and not exclusively) OneCollect, IMT, HWU, Host Utilities, ActiveIQ, and Config Advisor.

Overall, I’m satisfied with the questions and the exam. Here’s a list of resources for you to study if you’re planning to take the NS0-509 exam.

whydoyoulikewafls blog post about FC zoning

NetApp Learning Center - ONTAP SAN Fundamentals

NetApp Exam Preperation Guide

VMUG Sweden Leader Package

2019-09-13T00:00:00+00:00

I am proud to announce myself as a VMUG Leader in the Sweden Community! The journey has just started, and we’ve planned our first event in October! I’m super excited to be a part of the community and to lead the local group forward!

Yesterday I received the welcome package from VMware, and I’m happy to share with you what I got!

I did get these things:

Playing cards
Pens
VMUG Leader pin
Notebooks
VMUG flyers
VMUG Leader polo
Some small speakers with 3,5mm connection (tiny speaker!!)

VAMI Certificate in SSL Hybrid Mode

2019-08-09T00:00:00+00:00

When you’ve configured your vCenter Server Appliance to be a subordinate certificate authority, the VAMI interface at https://vcenter:5480 doesn’t use the new certificate, and it uses a self-signed certificate. In this post, I show you how to resolve this.

First, SSH into the vCSA and locate the certificate file you signed at your CA for the vCSA appliance, it should be named something like vmca.crt or vmca.cer.

Copy and rename that file:

cp vmca.cer /etc/applmgmt/appliance/vmca.crt

Open the configuration file:

vim /opt/vmware/etc/lighttpd/lighttpd.conf

Add this line at the bottom of the file:

ssl.ca-file = "/etc/applmgmt/appliance/vmca.crt"

Restart the HTTP service:

/etc/init.d/vami-lighttp restart

It should look like this if successful:

Now when you browse to the https://vcenter:5480, the certificate is valid and trusted.

Core Services needed to run VMware vSphere

2019-08-04T00:00:00+00:00

Often when we set up vSphere environments, there are already some things running. Sometimes it may be an Active Directory domain with DNS, NTP, CA, and all that setup. However, if you’re setting up a completely new environment, none of that may exist. What should you do then? We’ll cover all of that in this post.

People often discard the importance of core services. For example, they install vSphere with static entries in the host file instead of using a DNS server. It may work, but it’s hell to operate and maintain. When shit hits the fan, and eventually it does, it helps to have followed the best practices described in this post.

In this post, we’ll cover all of the needed core services for vSphere to function correctly.

Design

You do need to put some effort into the overall design of the environment you’re planning to deploy. It involves what core services you need as well as the redundancy and the separation of core services versus production workloads. It’s best practice to have these two separate, but not all companies invest in both a production cluster and a management cluster. Usually, it depends on the size of the deployment.

When you have a large amount of hardware, it’s more motivated to invest in a dedicated management cluster. However, if it’s a small deployment, it’s a significant investment for the company to buy hardware for the management part. The problem comes when everything goes down. What happens then? vSphere needs DNS to function but what if the DNS servers are down? It’s a chicken and egg situation. To mitigate issues you need to use anti-affinity described later in this post. If you have the core services running in a separate cluster, it’s a lot easier.

The primary core services we cover are:

DNS - Domain Name Service

We need at least two DNS servers to achieve redundancy. These usually are part of the Active Directory domain which we are going to cover later on in this post. You typically install a minimum of two domain controllers and both usually have the DNS service installed.

It’s recommended to always configure static IP-addresses on each service ensuring it retains IP after a reboot. When I say services in this sentence, I mean vCenter, ESXi, and other VMware products.

VMware - vSphere DNS requirements.

The requirements cover two things that are important to configure, and it’s to have both forward and reverse lookup zones.

Forward Lookup

Translates names to IP.
Reverse Lookup

The reverse of forwarding zones. The ability to query the DNS server for the FQDN of a specific IP-address.

NTP - Network Time Protocol

Network Time Protocol is a protocol to where the client syncs the time with a server. It’s essential to have the correct time-synchronized throughout the environment. When there’s a mismatch in time on ESXi hosts, the VM’s could get the wrong time if they sync with the system clock. I’ve experienced this several times, and it’s a pain! Also, when you troubleshoot, there’s helpful to have the same time across different services and logs. There could also be regulatory compliance within the environment, a payment system, for example.

Best practices from VMware is to have an authoritative NTP server as a time source, for example, your domain controllers. You also need to have these NTP servers configured to use a reliable source such as an on-premise GPS or one of the trusted external NTP pools. I use se.pool.ntp.org because I live in Sweden :)

Interesting blog post from VMware about time in ESXi.

CA - Certificate Authority

Certificate Authority is a server that issues certificates. The purpose of digital certificates is to maintain a chain of trust and to provide encryption. The certificate contains information about both these things. You know when you browse to a newly installed ESXi host you get certificates errors and manually needs to approve to continue? That’s when you’ve got the encryption, but your computer doesn’t trust the certificate chain! That’s because ESXi creates the certificates and you’ve never come across them before.

The new vCenter appliance contains a certificate mode called “Hybrid Mode” and that when you let the VMware Certificate Authority to be a subordinate certificate authority. That means that you only need to configure hybrid mode on the vCenter appliance and distribute the root certificate to your clients. The subordinate CA takes care of the rest; it signs new certificates to the ESXi hosts and other services inside vCenter.

VMware Docs - Use VMCA as an Intermediate Certificate Authority.

Authentication - Active Directory

Active Directory is(among other things) a service providing identity services. Identity service is the last requirement that we haven’t covered yet. Identity service and authentication are required to implement role-based access(RBAC), which is a way to grant different users different permissions. One example would be that you have a service desk group where users only should be able to use VM actions such as reboot, power on, off, and so on. It is possible with Active Directory combined with vSphere. It’s also possible with integrated users and other identity services, but Active Directory is the most commonly used.

VMware Docs - Join or Leave an Active Directory Domain.

Conclusion

If we summarize the post, we see that AD could help us maintain all of the requirements for a new environment: authentication, DNS, NTP, and CA.

The minimum requirement would be two domain controllers, bot with AD, DNS & NTP but only one with CA. The CA is just a passive service that signs certificates, so it doesn’t need to be online all the time.

You would ideally put these two VM’s either in a management cluster or in the production cluster, but in either case, you would use anti-affinity rules. Anti-affinity rules keep virtual machines on different hosts all the time. Read more about that here.

In the best of worlds, I argue that a physical domain controller to be used alongside the virtual one. It’s one of the most critical servers that is running in your environment, and when you run it physically, you remove a bit of overhead. I’m not saying that there’s anything wrong running virtual machines, but a physical machine doesn’t have that extra layer of virtualization.

Here are some topics covering Active Directory in a virtualized environment. Microsoft Docs - Virtualize Active Directory

VMworld US 2018 - Virtualize your Active Directory the Right Way! (VAP1898BU)

My first post

2019-07-30T00:00:00+00:00

I followed Dave Ceddias awesome guide on how to get a Gatsbyjs blog up and running, you’ll find the guide here.

My requirements was a minimal blog that was easy to manage. I needed to be able to write in markup syntax and have code higlighting.

Gatsbyjs is really aweseom, it lets med do all that and is lightning fast!

Wish me good luck!

johnhenriksson.se

Veeam Immutable Repository with NetApp ONTAP S3

Create the bucket in ONTAP System Manager

Configure the bucket in Veeam

Build Ubuntu 24.04 VM on vSphere with Packer

What is Packer?

Install Packer

Create the Build

Important code snippets in the build file

SSH Timeout

Generate Password

Configure Identity

SSH Settings

Build

Required Files

Block Youtube Shorts on Network

Pi-hole configuration

Set up certificates and nginx proxy on Ubuntu

Configure proxy

Conclusion

Locking Down Your NetApp ONTAP: A Comprehensive Security Guide for Safeguarding Your Data

ONTAP Image Validation

Local Storage Administrator Accounts

Authentication Methods

Login and Password Parameters

System Administration Methods

Login Banners and Message of the Day (MOTD)

Conclusion

Download and Activate ONTAP One

Step 1, pre-reqs

Generate the new license

Activate the new license in ONTAP

Read more

Monitor Trident with Prometheus and Grafana

Prometheus Operator

Trident Metrics

Set up Prometheus

ServiceAccount

ClusterRole

ClusterRoleBinding

Prometheus Instance

Prometheus Service

Trident ServiceMonitor

Deploy and Configure

Deploy Prometheus on the cluster

Deploy the Trident ServiceMonitor

Verify Prometheus and Trident ServiceMonitor

Configure Grafana

Add a Dashboard

Resources

ONTAP Select vCenter Plug-In

What is ONTAP Select?

ONTAP Select Deploy VM

ONTAP Select vCenter Plug-In

Deploy vCenter Plug-in

Deploy K8s VM's with Terraform on vSphere

Manage secrets with Pass

Use Pass in Terraform Configuration

Terraform

main.tf

variables.tf

terraform.tfvars

Terraform CLI Commands

Plan

Apply

Destroy

Terraform Configuration

Configure vSphere Provider

Collect information needed to deploy VM resources

Deploy VM Resources

Output IP addresses

Running Terraform

Complete Configuration Files

main.tf

terraform.tfvars

variables.tf

Build Ubuntu 20.04 VM on vSphere with Packer

What is Packer?

Install Packer

Create the Build