Setting up Nutanix Disaster Recovery (LEAP) – Step by Step

Nutanix DR implementation to protect guest VMs and orchestrate disaster recovery to other Nutanix Cluster when event causing service disruption to occur at primary site.

Nutanix DR Terminologies:

Availability Zone (AZ): A zone that can have one or more independent datacenters inter-connected by low latency links. An AZ can either be in your office premises (on-prem) or in Xi Cloud Services. AZs are physically isolated from each other to ensure that a disaster at one AZ does not affect another AZ. An instance of Prism Central represents an on-prem AZ.

Recovery Availability Zone: An AZ where you can recover the protected guest VMs when a planned or an unplanned event occurs at the primary AZ causing its downtime. You can configure at most two recovery AZs for a guest VM.

Source Virtual Network: The virtual network from which guest VMs migrate during a failover or failback.

Recovery Virtual Network: The virtual network to which guest VMs migrate during a failover or failback operation.

Network Mapping: A mapping between two virtual networks in paired AZs. A network mapping specifies a recovery network for all guest VMs of the source network. When you perform a failover or failback, the guest VMs in the source network recover in the corresponding (mapped) recovery network.

Category: A VM category is a key-value pair that groups similar guest VMs. Associating a protection policy with a VM category ensures that the protection policy applies to all the guest VMs in the group regardless of how the group scales with time. For example, you can associate a group of guest VMs with the Department: Marketing category, where Department is a category that includes a value Marketing along with other values such as Engineering and Sales.

Recovery Point: A copy of the state of a system at a particular point in time.

Recovery Point Objective (RPO): The time interval that refers to the acceptable data loss if there is a failure. For example, if the RPO is 1 hour, the system creates a recovery point every 1 hour. On recovery, you can recover the guest VMs with data as of up to 1 hour ago. Take Snapshot Every in the Create Protection Policy GUI represents RPO.

Recovery Time Objective (RTO): The time period from failure event to the restored service. For example, an RTO of 30 minutes enables you to back up and run the protected guest VMs in 30 minutes after the failure event.

Protection and DR Between on-prem Availability zone:

Leap protects your guest VMs and orchestrates their disaster recovery (DR) to other Nutanix clusters when events causing service disruption occur at the primary AZ.

Before proceeding further let me introduce to my environment, I have two Nutanix Clusters, both clusters are registered with their own prism central hosting on same cluster. Logical design between two cluster will as below

Enabling Nutanix Discovery:

  1. Login to Prism Central on Both Clusters
  2. Click Gear Icon à Click Disaster Recovery

  1. Click Enable

  1. Click Enable

Nutanix Disaster Recovery is enabled.

Connect AZ:

  1. Browse Navigation Bar à Administration à Availability Zones
  1. Click Connect to Availability Zone

  1. Select Physical Location, provide 2nd Prism Central IP, User and Password and click Connect

Connection will be created between both prism Central.

Creating Category:

  1. Browse Navigation Bar à Administration à Categories

  1. Click New Category

  1. Specify the Category Name and enter value (subcategories)

Creating Protection Policy:

  1. Browse Navigation Bar à Data Protection à Protection Policy

  1. Click Create Protection Policy

  1. Specify the Primary Location, Cluster and Click Save.

  1. Specify Recovery Location PC & Cluster and click save

  1. Specify the Snapshot frequency & retention on local and remote.

  1. Specify the desire Category and click add

  1. Click Create to create the Policy

Assigning VM to Category:

  1. Navigate to VM
  2. Select the desire VM à Action à Manage Categories
  1. Select Desire Category and Click Save

Review Protection Summary:

  1. Browse Navigation Bar à Data Protection à Protection Summary

Protection Summary Dashboard, will show the RPO Status

  1. Browse Navigation Bar à VM à Recovery Points will shows the VM recovery points and protection status of selected VM.

Creating Recovery Plans:

  1. Browse Navigation Bar à Data Protection à Recovery Plans

  1. Create New Recovery Plan

  1. Specify Recovery plan name, specify primary and recovery location and click Next.

  1. Click Add VMs

  1. Select the VM and click add

  1. Click Next to proceed

  1. Select Network Type ( Stretch , Non-Stretch) and Specify source and Target Network Subnet and click Done.

Initating Failover:

  1. Select Recovery Plan

  1. Click Failover to initate failover

  1. Select the Desire Failover type, Incase of Planned Failover (Source VM will be shutdown and after finnal sync , VM will be registered in target cluster and powered-on). Incase if unplanned Failover desire recovery points and be selected for restore .

  1. Type Failover and click Failover

  1. Click Tasks for see the Failover status

  1. VM Successfully failed-over to DR successfully.

Nutanix Cross Hypervisor Disaster Recovery

Cross hypervisor disaster recovery provides the ability to migrate VMs from one hypervisor to another (ESXi to AHV or AHV to ESXi) by using protection domain.

Some of the requirements:

  • Only VM Flat files are supported, vm snapshots or delta disks are not supported
  • VMs with attached volume groups or shared disk not supported
  • Different version of AOS is supported
  • NGT need should be installed on all the VMs.

Creating Protection Domain:

  1. Login to Web Console
  2. Click Main Menu à Data Protection

If DR Site is not added, click Remote Site à Physical Cluster to add the remote site.

Provide Remote Site name and remote cluster Virtual IP and click add site.

Map Source and Destination Network & Datastore as per requirement and click Save

Remote Site Connection is creation, let proceed to setup AsyncDR.

Note: For Cross Hypervisor replication, minimum RPO is 1 hours.

Click Protection Domain à Async DR to create protection domain.

Provide Protection Domain Name and click Create

Select the VMs / Object to protect and Click Protect Selected Entries

Once desired VMs are selected click next

Click New Schedule to create the Schedule.

Specify schedule and retention on local, DR Clusters and click Create Schedule.

Once Schedule is created. Click Close

Nutanix 1-Click In-Place Conversion (VMWare to AHV)

Nutanix has built-in capacity to convert Nutanix Cluster running on ESXi to AHV.

Some of the key requirement to proceed before in-place conversion are as below

  • Resolve all NCC health Check alerts
  • HA & DRS must be enabled
  • All hosts should be managed by the Same VCenter
  • VCenter should not be running on same cluster which is being converted
  • NGT Tools should be installed on all the VMs

Some of the Cluster Limitations are as below:

  • Metro Availability protection domain cannot be enabled in you environment
  • If Free ESXi License for your cluster, you can convert the cluster to AHV but you cannot revert the cluster back to ESXi.
  • Nutanix Files deployed on cluster
  • In-Place Hypervisor conversion is not supported for single and two node cluster

Virtual Machines Requirements & Limitations:

  • VMs with flat disk are supported, delta disks are not supported.
  • Only IDE And SCSI storage controllers are supported
  • Virtual machines with attached volume groups or shared virtual disks are not supported.
  • After reverting back to ESXi from AHV, the VMs are converted to the maximum hardware version that is supported by that specific ESXi version.

Let’s proceed with conversion:

  1. Log-in to Prism
  2. Click on gear icon, Main Menu à Settings à Convert Cluster
  3. Click Validate
  4. Specify VCenter IP Address, VCenter Username and Password and click Validate

Once Validation is completed. Click Covert Cluster to start the conversion.

The entire conversion process may take 3 to 4 hours depending on the nodes that are present in your cluster. However, the VM downtime will be less than 5 minutes because all the nodes in the cluster are converted in a rolling manner.

Setting-up Nutanix Metro Availability (VMware ESX) Step by Step

Nutanix Metro is the Nutanix DR Implementation when Zero RPO is desired. Nutanix Metro is supported by VMware ESXi, Hyper Hypervisors.

Nutanix Metro Availability works on the policy applied on datastore. It achieves by pairing a storage container on local Storage container with a storage container in remote site. containers. When metro availability is enabled, everything in the active storage container is replicated synchronously to the remote storage container. Metro availability configurations can include VMs, but they cannot include volume groups.

Metro availability policies apply per storage container, so a cluster can be active for one datastore and standby for another.

Some of the considering before proceeding:

  • There will be 2 Nutanix Clusters
  • It is recommended both Clusters hosts managed by same vcenter.
  • Less than 5ms latency between the 2 Sites.
  • In case of automatic failover 3rd site is required to setup witness

Log in VCenter, Create VMware Cluster and add both cluster hosts.

You can see Both Cluster Nodes are joined under same VCenter & VMware Cluster. Set VMware Cluster Nutanix recommended configuration as below

Enable DRS:

  • DRS Automation
    • Automation Level: Fully automated
    • Migration threshold: 3
    • Predictive DRS: Disabled
    • Virtual Machine Automation: Enabled
  • Additional Options
    • All unchecked
  • Power Management
    • DPM: disabled
  • Advanced Options
    • All unchecked/empty

Enable HA:

  • Failures and responses
  • Enable Host Monitoring: Enabled
    • Host failure response: Restart VMs
    • Response for Host Isolation: Power off and restart VMs
    • Datastore with PDL: Disabled
    • Datastore with APD: Power off and restart VMs – Aggrassive restart policy
      • Response Recovery: Disabled
      • VM Monitoring: VM Monitoring Only
  • Admission Control
    • Host failures cluster tolerates: 1
  • Heartbeat Datastores
    • Select the two METRO datastores
  • Advanced Options
    • Leave empty/default

VM Override Setting:

Make sure to disable DRS and HA under VM Override Settings.

Create two Host Groups

1. Group A (Main-Site)

a. All the Main-Site Hosts

2. Group B (DR-Site)

a. All the DR-Site Hosts


Create two VM Groups:

1. VMs on Main-Site

a. Add All the VMs which are on Main-site Cluster

2. VMs on DR-Site

a. Add all the VMs which are on DR-Site Cluster

With the VM/Host Rules we will decided where the VM should run, and incase of failure they can run from other cluster.

Basically, you will create 2 rules

§ Mian Cluster, all main-site VMs should run from main Site so DRS will no move those VM to DR Hosts

§ DR Cluster, all DR-site VMs should run from DR Cluster so DRS will no move those VM to Main-Site Hosts

We are done with the VMware Configuration, lets proceed to Prism.

Navigate to Data Protection à Remote Site

Click Remote Site à Physical Cluster

Specify Site Name and DR-Site Cluster Virtual IP , click add site to proceed.

Specify Main-site & DR-Site Datastores and save

Note: For Metro both Source & Destination Datastore should have same names.


To Create the Meto Availability Protection Domain, Click Protection Domain à Meto Availability

Click Next

Specify the source storage container and click next

Specify the Target container and click next

Note: My both clusters have starter license, that’s why target is not getting listing. To setup meto adv. Replication license is required.

Now the storage container is in sync between the clusters.

Nutanix Deployment with VMWare VSphere (Part 2)

In the part 1 we had completed Nutanix Block Foundation process, today will proceed with the next activities like setting up Prism Initial Configuration and Nutanix Recommended settings on VMware Cluster end.

  1. Open Prism using any Browser

Login using Default Credentials, default credentials are as below

admin / Nutanix/4u

Specify new Password

Accept User agreement , and click Accept to proceed further.

Enable / Disable Pulse and click continue

Nutanix Pulse:
Nutanix Pulse is the telemetry capability built into all Nutanix clusters that sends key health metrics to the Nutanix Insights service. Nutanix can use the diagnostic system information that Pulse sends to help build better products and provide a great customer experience.

Home Dashboard

To Specify the Nutanix Cluster IP, Click on Cluster Name. Provide Cluster Virtual IP (Virtual IP should be in the CVM network).

Click on drop down Menu and click Setting to proceed with the setting menu.

To update name servers, Click Name Servers and add DNS Name Server IPs.

To update NFS Setting, Click NFS Servers and specify NTP Servers

To create Storage Container, Click Dropdown Menu à Storage

Click + Storage Container to create the storage container. Provide Datastore name, select Mount on all host and click Save.

Datastore has been Created.

Now we will proceed to add Nutanix Hosts in VCenter. Incase if you don’t know how to deploy VCenter please refer my VCenter Deployment Article.

Right Click on VCenter Root and Create VCenter Cluster.

Specify Cluster Name and Click next

To Register Hosts in VCenter. Click Configure à Quickstart à Under Add Hosts à Click ADD

Provide Host IPs and username and password and click Next.

Accept Host SSL Certificate and click ok

Click Next to Proceed Further

Click Finish

Once Host has been added, we will proceed with the Nutanix Recommended HA , DRS and VM Override settings.

VMWare HA Settings:

Select Cluster à Configure à VSphere Availability

  • Host Failure Response: Select Restart VMs from the drop-down list.
  • This option configures the cluster-wide host isolation response settings.
  • Response for Host Isolation: Select Power off and restart VMs from the drop-down list.
  • Datastore with PDL: Select Disabled from the drop-down list.
  • Datastore with APD: Select Disabled from the drop-down list.
  • VM Monitoring: Select Disabled from the drop-down list.

  • Host failures cluster tolerates: Enter 1 or 2 based on the number of nodes in the Nutanix cluster and the replication factor.
  • Define host failover capacity by: Select Cluster resource Percentage from the drop-down list.
  • Performance degradation VMs tolerate: Set the percentage to 100.

Select the Nutanix Datastore. If the cluster has only one datastore, click Advanced Options tab and add das.ignoreInsufficientHbDatastore with Value of true.

Click OK to apply the settings.

DRS Settings:

Select Cluster à Configure à VSphere Availability

Turn on the vSphere DRS switch.

Specify the following information under the Automation tab.

Leave DPM unchecked and click OK

EVC MODE :

Click Configure, and go to Configuration > VMware EVC.

Enable EVC for the CPU vendor and feature set appropriate for the hosts in the Nutanix cluster, and click OK

VM Override:

Click Add

Select CVM VMs and click Next

Select Below override settings.

  • DRS Automation Level: Disabled
  • VM HA Restart Priority: Disabled
  • VM Monitoring: Disabled

Click Finish to Save the setting .

VCenter Registration:

It is required to register VCenter in Prism, To do the VCenter registration Settings à VCenter Registration

Click Register

Specify VCenter Username and Password and click Register

Change Nutanix Default Passwords

The Nutanix Controller VM (CVM) ‘nutanix’ Account

To change the nutanix default password, SSH to any of the CVM and run the below command to change password, it will replicate to all the CVMs.

nutanix@cvm$ sudo passwd nutanix

The Hypervisor Local Accounts

AHV Hypervisor:
To Change root account password on all Hosts in the cluster, run below command from any of the CVM.

nutanix@cvm$ echo -e “CHANGING ALL AHV HOST ROOT PASSWORDS. Note – This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: “; read -s password1; echo “Confirm new password: “; read -s password2; if [ “$password1” == “$password2” ] && [[ ! “$password1” =~ [\\\{\$\^\}\&] ]]; then hostssh “echo -e \”root:${password1}\” | chpasswd”; else echo “The passwords do not match or contain invalid characters (\ $ { } ^ &)”; fi

To Change admin account password on all Hosts in the cluster, run below command from any of the CVM.

nutanix@cvm$ echo -e “CHANGING ALL AHV HOST ADMIN PASSWORDS. Note – This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: “; read -s password1; echo “Confirm new password: “; read -s password2; if [ “$password1” == “$password2” ] && [[ ! “$password1” =~ [\\\{\$\^\}\&] ]]; then hostssh “echo -e \”admin:${password1}\” | chpasswd”; else echo “The passwords do not match or contain invalid characters (\ $ { } ^ &)”; fi

To Change nutanix account password on all Hosts in the cluster, run below command from any of the CVM.

nutanix@cvm$ echo -e “CHANGING ALL AHV HOST NUTANIX PASSWORDS. Note – This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: “; read -s password1; echo “Confirm new password: “; read -s password2; if [ “$password1” == “$password2” ] && [[ ! “$password1” =~ [\\\{\$\^\}\&] ]]; then hostssh “echo -e \”nutanix:${password1}\” | chpasswd”; else echo “The passwords do not match or contain invalid characters (\ $ { } ^ &)”; fi

VMware ESXi

To Change root account password on all Hosts in the cluster, run below command from any of the CVM.

nutanix@cvm$ echo -e “CHANGING ALL ESXi HOST PASSWORDS. Note – This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: “; read -s password1; echo “Confirm new password: “; read -s password2; if [ “$password1” == “$password2” ] && [[ ! “$password1” =~ [\\\{\$\^\}\&] ]]; then hostssh “echo -e \”${password1}\” | passwd root –stdin”; else echo “The passwords do not match or contain invalid characters (\ $ { } ^ &)”; fi

Microsoft Hyper-V

To change local administrator password for all Hyper-V hypervisors in the Nutanix cluster. Run the below command from any from the CVM.

nutanix@cvm$ echo -e “CHANGING ALL HYPER-V HOST PASSWORDS. Note – This script cannot be used for passwords that contain special characters ( $ \ { } ^)\nPlease input new password: “; read -s password1; echo “Confirm new password: “; read -s password2; if [ “$password1” == “$password2” ] && [[ ! “$password1” =~ [\ \”\’\\\{\$\^\}] ]]; then hostssh “net user administrator $password1”; echo “Updating Host and ManagementServer Entries…”; ncli host ls | grep -i id | grep -Eo “::[0-9]*” | cut -c 3- | while read hID; do ncli host edit id=$hID hypervisor-password=$password1;done > /dev/null; ncli host ls | grep “Hypervisor Address” | awk ‘{print $4}’ | while read hIP; do ncli managementserver edit name=$hIP password=$password1;done > /dev/null; else echo “The passwords do not match or contain invalid characters (\ $ { } ^)”; fi

How to create VM Trunk NIC in AHV

VM NIC can works in two modes

  • Access
  • Trunk

Access nic are the default and NIC is associated with one VLAN. Whereas trunk can allow multiple VLAN on single NIC.

Note: Currently Trunk port can be added/ modifed from CLI only from the prism both access & trunk ports look same.

SSH on any of the CVM and run the below command.

nutanix@CVM~$ acli vm.nic_create <vm name> network=<network name> trunked_networks=<comma separated list of allowed VLAN IDs> vlan_mode=kTrunked

if VM name is DEMO & you want to allow VLAN 10,20,30,40 , we need to run below command.

nutanix@CVM~$ acli vm.nic_create DEMO network=network trunked_networks=10,20,30,40 vlan_mode=kTrunked

If incase if you want to convert Trunk NIC to Access NIC , then below is the command syntax.

acli vm.nic_update <vm name> <vm nic mac address> vlan_mode=kAccess update_vlan_trunk_info=true

Nutanix Deployment with VMWare VSphere (Part1)

In This series will discuss the Nutanix with VMWare VSphere from A to Z. Starting from New Cluster Deployment, VSphere Configuration and last not the least Migration.

Foundation is official deployment software of Nutanix. Foundation help to configure pre-imaged or reimage the nodes with the required hypervisor and AOS. Foundation can give the option to create cluster or skip the cluster formation.

Foundation can be download from Nutanix Support portal.

https://portal.nutanix.com/page/downloads?product=foundation

Foundation software is available for

  • Foundation for windows
  • Foundation for MacOS
  • Standalone Foundation VM

Beside Foundation we required desire AOS bundle & Hypervisor ISO (if we are not going with AHV)

In Today’s walkthrough will use Foundation for Windows to do the foundation. Once Foundation Application is installed. Run as admin foundation.exe

Before proceeding with the foundation, some of the keypoints need to be noted. I would recommend the flat unmanaged switch and connect all nodes shared network ports and laptop on the switch.

You can check the shared port information from hardware compatibility list.

https://portal.nutanix.com/page/documents/list?type=compatibilityList

incase if share port is not available we need to connect IPMI & ethernet ports from each node.

Select the hardware platform type, Provide CVM & IPMI Subnet and Gateway IP. Click Next to Proceed.

Provide each Node’s IPMI, HOSTIPs, CVM IPs and hostname and click Next.

Provide Cluster Name, Specify Time Zone & CVM Memory and Click Next.

Provide AOS Bundle and Click Next

By default AHV is bundle within AOS Package, as I want to install VMware ESXI as hypervisor so we need to provide VMware ESXi ISO.

Provide IPMI Credentials and Click Next

Foundation Process will take approx. 1hrs.

Foundation is finished. You can access the prism from any of the CVM IPs.

Default Credentials are as below:

User Password
CVM nutanix nutanix/4u
Hypervisor root nutanix/4u
Prism admin Nutanix/4u
IPMI ADMIN Node Serial Number

Create Citrix NetScaler ADC VM in Nutanix AHV

Nutanix AHV is certified to run Citrix NetScaler. Below is the procedure to create Citrix NetScaler VM on AHV.

  • Download NetScaler Virtual Appliance from Citrix Portal
  • Extract .tgz file , it will give you .qcow file
  • Login to Prism

  • Click Setting à Image Configuration
  • Click Upload Image and specify below information
    • Name
    • Image Type
    • Storage Container
    • Upload a file / From URL

  • Click Save to start the upload.
  • Navigate to VM page
  • Create VM , in the VM Creation Dialogue page

    • Provide VM Name
    • Specify VCPU = 2 (recommended by Citrix)
    • Specify Ram = 2G (recommended by Citrix)
    • Click Add Disk , Select Clone from Image Service and select the Citrix NetScaler Image and click add

    • Add Network and click Add

    • Click Save to create the VM.
  • NetScaler required Serial Port on the VM, to add the Serial Port on the VM
    • SSH to CVM
    • Run the below command to add the serial port
      • acli vm.serial_port_create <vm_name> type=kServer index=0

  • Power-On the VM and open console from Prism.

Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can’t Convert “22010 10001”

FIP-FS Scan Engine

FIP-FS is probably the anti-malware virus scanner that has been on board since Exchange Server 2013. This is supposed to scan the on-premises Exchange Server installation for malicious content. This anti-malware scan engine seems to cause problems the problem.

Temporary Solution is to disable anti-malware filter . You can disable it as below

Set-MalwareFilteringServer exch-19 -BypassFiltering $true

Happy New Year 😊