Tag Archives: VCHA

How to deploy vCenter server HA using vSphere Python SDK pyVmomi?: Part-2

In part 1, we explored new vCenter HA APIs for managing vCenter HA. In part 2, we will focus on how to deploy vCenter HA using pyVmomi. As I pointed in part-1, managed object FailoverClusterConfigurator is responsible for deploying vCenter HA & below are the methods exposed by this managed object.

  • deployVcha_Task
  • configureVcha_Task
  • createPassiveNode_Task
  • createWitnessNode_Task
  • destroyVcha_Task

Let us start with first and important method i.e.  deployVcha_Task() : This method is really handy when user wants to deploy as well as configure vCenter HA in single call. I will encourage you to read vSphere API reference for more details on this method. This method assumes that you already deployed vCenter server,which would act as active node and created required vCenter HA network. As I wanted to deploy vCenter HA in basic mode, I created my self managed vCenter server as shown below & configured required vCenter HA network.

Before you start deploying your active node, please take a look at vCenter HA software & hardware requirement. I would also recommend you to just overview this & this documentation. If you need any help on deploying active node, please let me know, I would be happy to help you.

Lets take a look at code now. Note that detailed code documentation is added inside script itself for your easy reference & all below scripts are available on my github repo as well.


from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
# Deploying vCenter HA in basic mode using self managed VC 

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Parameters required are hardcoded below, please do change as per your environment.

vcha_network_name="VCHA" #port group name, I am using standard switch.
vcha_dc_name="IndiaDC"    #Datacenter name 
vcha_subnet_mask="255.255.255.0"  #Subnect mask for vCenter HA/Private network
active_vcha_ip="192.168.0.1"    # Active node vCenter HA IP
passive_vcha_ip="192.168.0.2"   # Passive node vCenter HA IP
witness_vcha_ip="192.168.0.3"   # Witness node vCenter HA IP
active_vcha_vm_name="vThinkBVM-VC1" #Active node/VC VM name
active_vc_username="Administrator@vsphere.local"  #Active VC username
active_vc_password="VMware#23"  #Active VC password
active_vc_url="https://10.61.34.35"  #Active VC public IP
active_vc_thumbprint="55:A9:C5:7E:0C:CD:46:26:D3:5C:C2:92:B7:0F:A7:91:E5:CD:0D:5D" #Active VC thumbprint
passive_vc_datastore="SharedVMFS-1"  #Passive node datastore
witness_vc_datastore="SharedVMFS-2"  #Witness node datastore

vcha=si.content.failoverClusterConfigurator #Getting managed object responsible for vCenter HA deployment

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
        obj = None
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for c in container.view:
                if name:
                        if c.name == name:
                                obj = c
                                break
                        else:
                                obj = c
                                break
        return obj


vcha_network=get_obj(content,[vim.Network],vcha_network_name) 
vcha_dc=get_obj(content,[vim.Datacenter],vcha_dc_name)

#I would highly recommend  to read vSphere API reference for "failoverClusterConfigurator", this will help to understand below specs.
 
deployment_spec=vim.vcha.FailoverClusterConfigurator.VchaClusterDeploymentSpec()

#Active node related data/parameter population
active_nw_config_spec=vim.vcha.FailoverClusterConfigurator.ClusterNetworkConfigSpec()
active_nw_config_spec.networkPortGroup=vcha_network
active_ipSettings=vim.vm.customization.IPSettings()
active_ipSettings.subnetMask = vcha_subnet_mask
active_ip_spec=vim.vm.customization.FixedIp()
active_ip_spec.ipAddress= active_vcha_ip
active_ipSettings.ip=active_ip_spec
active_nw_config_spec.ipSettings=active_ipSettings
deployment_spec.activeVcNetworkConfig=active_nw_config_spec

#Active node service locator 
active_vc_spec=vim.vcha.FailoverClusterConfigurator.SourceNodeSpec()
active_vc_vm=get_obj(content,[vim.VirtualMachine],active_vcha_vm_name)
active_vc_spec.activeVc=active_vc_vm
service_locator=vim.ServiceLocator()
cred=vim.ServiceLocator.NamePassword()
cred.username=active_vc_username
cred.password=active_vc_password
service_locator.credential=cred
service_locator.instanceUuid=si.content.about.instanceUuid
service_locator.url=active_vc_url  #Source active VC
service_locator.sslThumbprint=active_vc_thumprint
active_vc_spec.managementVc=service_locator
deployment_spec.activeVcSpec=active_vc_spec

#Passive node configuration spec
passive_vc_spec=vim.vcha.FailoverClusterConfigurator.PassiveNodeDeploymentSpec()
passive_ipSettings=vim.vm.customization.IPSettings()
passive_ipSettings.subnetMask = vcha_subnet_mask
passive_ip_spec=vim.vm.customization.FixedIp()
passive_ip_spec.ipAddress= passive_vcha_ip
passive_ipSettings.ip=passive_ip_spec
passive_vc_spec.ipSettings=passive_ipSettings
passive_vc_spec.folder=vcha_dc.vmFolder
passive_vc_spec.nodeName= active_vcha_vm_name+"-passive"
passive_datastore=get_obj(content,[vim.Datastore],passive_vc_datastore)
passive_vc_spec.datastore=passive_datastore
deployment_spec.passiveDeploymentSpec=passive_vc_spec

#Witness node configuration spec
witness_vc_spec=vim.vcha.FailoverClusterConfigurator.NodeDeploymentSpec()
witness_ipSettings=vim.vm.customization.IPSettings()
witness_ipSettings.subnetMask = vcha_subnet_mask
witness_ip_spec=vim.vm.customization.FixedIp()
witness_ip_spec.ipAddress= witness_vcha_ip
witness_ipSettings.ip=witness_ip_spec
witness_vc_spec.ipSettings=witness_ipSettings
witness_vc_spec.folder=vcha_dc.vmFolder
witness_vc_spec.nodeName=active_vcha_vm_name+"-witness"
witness_datastore=get_obj(content,[vim.Datastore],witness_vc_datastore)
witness_vc_spec.datastore=witness_datastore
deployment_spec.witnessDeploymentSpec=witness_vc_spec

# Calling the method we aimed to invoke by passing complete deployment spec

task= vcha.deployVcha_Task(deployment_spec)

if(task.info.state == "running"):
        print "VCHA deployment is started, it will take few minutes, please monitor web client for its completion"

As soon as we invoke deployVcha_Task() method, it deploys vCenter HA in basic mode where it also creates passive and witness node. Below is how it looks when looked into web client recent task pane.

We can see that “Deploy VCHA” task is running, it has in-turn cloned passive/witness nodes and initiated powereON operations. I waited for some time for deploy VCHA task to get completed. After completion, below is how vCenter HA VMs look from web client.

Note: You might have noticed that we need to have vCenter active node thumbprint to be passed in VCHA spec. I could get this thumbprint by using openssl command as shown here. In fact, there are some other ways to get it as well.

Now that we deployed VCHA successfully, let us take a look at how to destroy the vCenter HA using method “destroyVcha_Task()”: This method can be invoked when VCHA is disabled, isolated or its configuration is failed. In my case, I chose to disable the existing VCHA by using method “setClusterMode_Task()” , which we already explored in Part-I. Lets take a look at the code.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Destroy vCenter server HA
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Getting VCHA configurator managed object
vcha=si.content.failoverClusterConfigurator

#Getting VCHA cluster manager
vcha_cluster_manager=si.content.failoverClusterManager

# Setting vCenter HA to "disabled" mode.
task = vcha_cluster_manager.setClusterMode_Task("disabled")
while(task.info.state != "success"):
        continue

#Getting VCHA cluster mode
VCHA_mode=vcha_cluster_manager.getClusterMode()
if (VCHA_mode == "disabled"):
        vcha.destroyVcha_Task() #Destroing it
else:
        print "VCHA must be in disabled mode before destrying it"

When I executed above script, I could see from web client that vCenter HA mode got set to disabled followed by vCenter HA was getting destroyed. Please take a look at below screenshot to confirm it

I waited for few seconds & I could see, vCenter HA is destroyed as shown in below screenshot.

Note that once we destroy vCenter HA using above method, active node will continue to work fine & serve client requests. This API just destroys the vCenter HA configuration. It does not delete passive & witness node by itself. To delete passive & witness node, you will have to use regular VM destroy API (This option is provided in UI as part of vcenter HA destroy UI workflow)

Moving on to other methods exposed by managed object “FailoverClusterConfigurator”, it seems these methods (listed below) are directly suited when vCenter HA is deployed using advanced option

configureVcha_Task(),
createPassiveNode_Task(),
createWitnessNode_Task()

Since I had deployed vCenter HA using basic option, I will talk about above methods in my future post or will add samples on my github repo. On the other side, you can refer all the scripts discussed in part-1,2 & easily code around those as well. If you ask me, it would be really good exercise to understand vCenter HA APIs.

Notes:
1. All above scripts are available on my github repository as well. Even, I have a plan to write one nice python module for the same.
2. Note that, for the sake of simplicity, I have hard-coded some values & disabled certificate validation, hence please do appropriate changes as per your environment.
2. I highly recommend you to read Part 1
3. Finally, if you haven’t yet set up “pyVmomi” environment, refer my blog post

How to manage vCenter server HA using vSphere python SDK pyVmomi? : Part 1

Recently I was undergoing “vSphere 6.5: Optimize and Scale” course and I was excited to see a dedicated module on “vCenter Server High Availability and Performance”. Without a doubt, vCenter Server HA was one of the highlights of vSphere 6.5 release. If you quickly want to understand vCenter HA, I highly recommend you to watch innovative white board style presentation by our own Adam Eckerle on VMware’s official youtube channel here

After completing the vCenter HA course lab, when I looked into vSphere 6.5 SOAP API reference, I could see there are 2 managed objects exposed as part of vCenter Server High Availability as follows.

1. FailoverClusterConfigurator
2. FailoverClusterManager

In part 1, I am going to demonstrate how can we manage vCenter HA by using 2nd managed object listed above i.e. FailoverClusterManager. This managed object provides operations to manage a vCenter High Availability Cluster (VCHA Cluster) and it assumes that vCenter HA is already deployed. Accordingly, in my case, basic vCenter HA was already deployed. Let’s now see how can we invoke methods exposed by this managed object and fetch/set some cool properties using pyVmomi. If you still haven’t set pvVmomi environment, please do configure it using my earlier post “getting started with pyVmomi”. Here we go.

If you look into API reference for managed object “FailoverClusterManager”, it has exposed total 4 methods i.e. getClusterMode(), GetVchaClusterHealth(), initiateFailover_Task() & setClusterMode_Task(). Let’s go one by one.

1. getClusterMode(): This method gets us the current mode that vCenter HA cluster is in: vCenter HA can be in one of the these 3 modes i.e. Enabled, Disabled or Maintenance mode. Let us look at what is the mode my vCenter HA cluster is in.


from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to get vCenter Server High Availability (VCHA) mode
# Below is Python 2.7.x code, which can be easily converted to python 3.x version

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.45.10", user="Administrator@vsphere.local", pwd="VMware!1",sslContext=s)
vcha=c.content.failoverClusterManager

VCHA_mode=vcha.getClusterMode()
print "VCHA Cluster mode::", VCHA_mode

Line 11 to 14: We have got “FailoverClusterManager” object, invoked getClusterMode() using it and finally printed the VCHA mode.

Output:
vmware@localhost:~$ python getVCHA_cluster_mode.py
VCHA Cluster mode:: enabled

Same is the VCHA cluster mode shown on below vSphere web client screenshot, is it not pretty simple to call this method?

2. GetVchaClusterHealth(): This method gives us overall health of vCenter HA.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to Get vCenter server HA health information

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.20.30", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha = c.content.failoverClusterManager

VchaClusterHealth = vcha.GetVchaClusterHealth()

vcha_health_Messages = VchaClusterHealth.healthMessages
print "VCHA Health messages::"
for health_data in vcha_health_Messages:
        print health_data.message

print "\nAdditional Information::",VchaClusterHealth.additionalInformation

vcha_runtime_info = VchaClusterHealth.runtimeInfo
print "\nVCHA Cluster Mode::",vcha_runtime_info.clusterMode
print "\nVCHA Cluster State::",vcha_runtime_info.clusterState

vcha_node_info = vcha_runtime_info.nodeInfo

print "\nVCHA Node information:"
for node in vcha_node_info:
        print node.nodeRole+":"+node.nodeIp+":"+node.nodeState

Line 13: We have invoked method “GetVchaClusterHealth()” & it returns VchaClusterHealth object.
Line 15 to 18: We are fetching the property “healthMessages” offered by object returned above. Since “healthMessages” are multiple (array), we iterated and printed each one of them.
Line 20 to 30: We fetched remaining properties of the object ” VchaClusterHealth”, such as vcha mode, vcha state & vcha nodeinfo etc. and printed them one by one.

Output:
vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
active:192.168.111.151:up
passive:192.168.111.152:up
witness:192.168.111.153:up

Note: Additional Information property returns “empty” array since VCHA cluster is in healthy mode. If cluster is not in healthy mode, it will provide additional info on what is causing the issue etc.

If you see below web client screenshot, it is matching perfectly with above output. For IPs, you can refer the 1st screenshot already posted above.

3.initiateFailover_Task(): By invoking this method, user can initiate a fail-over from active node to passive node.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

#Script to get initiate vCenter Server High Availability failover

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.160.20.40", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager
task = vcha.initiateFailover_Task(True)

while(task.info.state != "success"):
        continue
print "Initiate Failover task is completed"

Line 12 to 16: We invoked method “initiateFailover_Task(True)” and waited for task to complete. Please take a note that we are passing boolean “True” into this method, which indicates that this method will wait till on-going active-passive state replication to finish and finally it initiates the failover, other wise it will force the failover, which can lead to data loss.

As soon as I executed above script, I could see, as expected web client session was down and when I logged out of web client, I was happy to see below message on web client.

If you want to know how to initiate failover directly from web client, please take a look at below screenshot.

I waited for some more time and I could see web client was up & running again. Now I thought to check overall VCHA health to see whether VCHA is in healthy condition and active/passive nodes are actually exchanged or not. Accordingly I executed the script #2, which was written for getting VCHA health using GetVchaClusterHealth() and below was the output.

vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
passive:192.168.111.151:up
active:192.168.111.152:up
witness:192.168.111.153:up

As expected, VCHA was in enabled mode and healthy state, even active & passive node IPs were exchanged. Same was confirmed from web client as follows.

4.setClusterMode_Task(): Using this method user can change VCHA mode. Since my current mode is “enabled” & state as “healthy”, I can either set mode to “disabled” or “maintenance”. I decided to change VCHA cluster mode to “maintenance”. In maintenance mode, VCHA active/passive state replication is enabled but automatic failover is not allowed.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Script to set vCenter Server High Availability mode
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.1.2", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager

task = vcha.setClusterMode_Task("maintenance")

while(task.info.state != "success"):
        continue
print "VCHA mode is set to ::", vcha.getClusterMode()

Line 11 to 15: invoked “setClusterMode_Task()” method, waited till it completes and finally checked cluster mode by calling “getClusterMode()” method.

Output:
vmware@localhost:~$ python setVCHA_cluster_mode.py
VCHA mode is set to :: maintenance

Then immediately I looked into web client and I see VCHA mode was changed to “maintenance”as follows.

That is all, How cool is that! I really enjoyed writing this post and I hope you will enjoy as well. Update: I recently published Part-2 on “FailoverClusterConfigurator” managed object as well.

Further learning on vCenter Server HA

1. All above scripts are available on my github repository as well. I have a plan to write one nice python module for the same. Note that for the sake of simplicity I have hard-coded some values & disabled certificate validation, please do appropriate changes as per your environment.
2. VCHA walkthroughs
3. If you prefer PowerCLI to play around VCHA instead of pyVmomi, please take a look at nice VCHA PowerCLI module written by none other than “William Lam
4. VCHA performance white paper
5. Finally, if you want to start learning “pyVmomi”, refer my blog post