Category Archives: HOME

PowerCLI script : How to confirm a vCenter server, ESXi VMware hypervisor & microcode patches are applied?: Spectre vulnerability

Some time back, VMware has released one of important KBs on “Hypervisor-Assisted Guest Mitigation for branch target injection”. Please take a look at this KB here. In this KB, there is one important section on “how to confirm a host has both patched microcode and patched VMware hypervisor?” and below is what is posted on KB.

Confirmation of Correct Operation
To confirm a host has both patched microcode and patched VMware hypervisor, use the following steps:
1.Power on a Virtual Machine which is configured to use Virtual Hardware Version 9 or later.
2.Examine the vmware.log file for that VM and look for one of the following entries:
“Capability Found: cpuid.IBRS”
“Capability Found: cpuid.IBPB”
“Capability Found: cpuid.STIBP”
3. Any of the above log entries indicates that both the CPU microcode and hypervisor are properly updated.

I thought it would be really handy if we automate it using PowerCLI including right vCenter server patch. Accordingly, I started looking for any existing cmdlets which can help me to retrieve the VM logs (specially vmware.log) and I could see PowerCLI guru Luc Dekens had written a article on retrieving VM logs long back. I leveraged the same and came up with one quick PowerCLI script to confirm whether a host has both patched microcode and patched VMware hypervisor or not. Here we go.

Update 01/13: As per latest update from VMware on KB, I have published another PowerCLI script, I recommend you to take a look at it before reading this post further.

Script execution steps:
i) First it validates whether vCenter is patched to correct build or not (builds are compared with build specified in respective release notes).
ii) After that, it gets all the connected hosts inside the specified cluster. It can be easily modified to consider host from particular datacenter as mentioned above.
iii) It will then iterate through each host in sequence and create a dummyVM.
iv) DRS automation on created dummyVM will be disabled since we need to powerON the VM and if DRS is enabled on the cluster, as part of DRS initial placement workflow, it may place/powerON the VM on some other host inside the cluster.
v) Now it will powerON the dummyVM
vi) Download the vmware.log file at specified location
vii) Scan for the log lines specified in KB (posted above) & record the results.
viii) PowerOff the created VM (to avoid any disk space utilization, you can choose to delete this VM as well)
ix) Step iii) through viii) will repeated for every host inside the cluster.
x) Finally log the result into one CSV file. CSV file will have HostName & Status (Patched or Un-Patched). Refer CSV sample below.

Update 01/10: Added vCenter server patch validation code as well. Now this script can provide you report on whether vCenter, ESXi VMware hypervisor & ESXi host microcode patches are applied or not. It is already added into below script.

Update 01/11: Below code will consider all hosts from particular datacenter (it will cover all clusters from that DC as well) instead of just cluster. By default script will take hosts from specified cluster, to consider all hosts from datacenter, please replace line #106 through #110 with below lines and do change “ResourcePool” parameter passed to New-VM cmdlet, which creates the VM.

$DCName="IndiaDC" # VC datacenter name, script assumes that datacenter is available
$dc= Get-Datacenter -Name $DCName

#Get connected hosts from above datacenter
$esxhosts = Get-Datacenter $dc | Get-VMHost -State Connected

Below script is available on my git-hub repo as well

<# .SYNOPSIS PowerCLI script:For spectre vulnerability : "How to confirm whether vCenter server, ESXi hypervisor & CPU microcode patches are applied or not?" .NOTES Author: Vikas Shitole .NOTES Site: www.vThinkBeyondVM.com .NOTES Please add the vCenter server IP/credetails as per your environment .NOTES Relese notes: VC 6.5 U1e: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-65u1e-release-notes.html VC 6.0 U3d: https://docs.vmware.com/en/VMware-vSphere/6.0/rn/vsphere-vcenter-server-60u3d-release-notes.html VC 5.5 U3g: https://docs.vmware.com/en/VMware-vSphere/5.5/rn/vsphere-vcenter-server-55u3g-release-notes.html #>

#vCenter Connection, please modify as per your env.

Connect-VIServer -Server 10.160.20.30 -Protocol https -User administrator@vsphere.local -Password VMware!32


function Get-VMLog{
<# .SYNOPSIS Retrieve the virtual machine logs .DESCRIPTION The function retrieves the logs from one or more virtual machines and stores them in a local folder .NOTES Author: Luc Dekens .PARAMETER VM The virtual machine(s) for which you want to retrieve the logs. .PARAMETER Path The folderpath where the virtual machines logs will be stored. The function creates a folder with the name of the virtual machine in the specified path. .EXAMPLE PS> Get-VMLog -VM $vm -Path "C:\VMLogs"
.EXAMPLE
	PS> Get-VM | Get-VMLog -Path "C:\VMLogs"
#>
 
	param(
	[parameter(Mandatory=$true,ValueFromPipeline=$true)]
	[PSObject[]]$VM,
	[parameter(Mandatory=$true)]
	[string]$Path
	)
 
	process{
		foreach($obj in $VM){
			if($obj.GetType().Name -eq "string"){
				$obj = Get-VM -Name $obj
			}
		}
		$logPath = $obj.Extensiondata.Config.Files.LogDirectory
		$dsName = $logPath.Split(']')[0].Trim('[')
		$vmPath = $logPath.Split(']')[1].Trim(' ')
		$ds = Get-Datastore -Name $dsName
		$drvName = "MyDS" + (Get-Random)
		New-PSDrive -Location $ds -Name $drvName -PSProvider VimDatastore -Root '\' | Out-Null
		Copy-DatastoreItem -Item ($drvName + ":" + $vmPath + "vmware.log") -Destination ($Path + "\" + $obj.Name + "\") -Force:$true
		Remove-PSDrive -Name $drvName -Confirm:$false
	}
}
$report = @()

#As per release notes: vCenter builds which provide part of the hypervisor-assisted guest remediation of CVE-2017-5715 for guest operating systems 
$vc65u1eBuild = "7515524" #VC 6.5 U1e build

$vc60u3dBuild = "7464194" #VC 6.0 u3e build

$vc55u3gBuild = "7460778" #VC 5.5 u3g build

#getting service-Instance object to get vCenter build details
$Si = Get-View ServiceInstance
$vcVersion = $Si.Content.About.ApiVersion
$vcBuild = $Si.Content.About.Build
$vcBuildVersion=$Si.Content.About.FullName

if (($vcBuild -eq $vc65u1eBuild) -or ($vcBuild -eq $vc60u3deBuild) -or ($vcBuild -eq $vc55u3gBuild )) {
	Write-Host "vCenter build is matching with build specified on release notes"
	Write-Host "vCenter build number::" $vcBuild
	Write-Host "vCenter build & version ::" $vcBuildVersion
	Write-Host "VC is patched to correct build"
	$row = '' | select HostName, Status
 	$row.HostName = $vcBuildVersion
 	$row.Status="Patched"
 	$report += $row
}Else {
    Write-Host "vCenter build is NOT matching with build specified on release notes"
	Write-Host "vCenter build number::" $vcBuild
	Write-Host "vCenter build & version ::" $vcBuildVersion
	Write-Host "VC is NOT patched to correct build, please upgrade vCenter server."
	$row = '' | select HostName, Status
 	$row.HostName = $vcBuildVersion
 	$row.Status="Un-Patched"
 	$report += $row
}

#Location where vmware.log file gets downloaded
$drive="C:\"
$vmname="MyVM"

# Any of below lines must be found in vmware.log file to confirm microcode & VMware hypervisor patch
$pat1='Capability Found: cpuid.IBRS'
$pat2='Capability Found: cpuid.IBPB'
$pat3='Capabliity Found: cpuid.STIBP'

$clusterName="EVCCluster" #Your cluster name, script assumes that cluster is available
$cluster= Get-Cluster -Name $clusterName

#Get connected hosts from above cluster
$esxhosts = Get-Cluster $cluster | Get-VMHost -State Connected

#Counter used to give unique name to dummyvm, you can use any number of your choice.
$i=45

#Iterating through each host for VM creation and scanning vmware.log file
Foreach ($ESXHost in ($esxhosts)){
	$vm=$vmname+$i
#Creating dummy vm with below configuration
	New-VM -Name $vm -VMHost $ESXHost -ResourcePool $cluster -DiskGB 1 -MemoryGB 1 -DrsAutomationLevel Disabled -DiskStorageFormat Thin
	Start-VM -VM $vm -RunAsync -Confirm:$false  #DRS may powerON this VM on some other host inside the cluster
	$dest=$drive+$vm
	Get-VMLog -VM $vm -Path "C:\"
	if (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat1}){
 		$row = '' | select HostName, Status
 		$row.HostName = $($ESXHost.name)
 		$row.Status="Patched"
 		$report += $row
 		Write-Host "Matched pattern:"+$pat1+":"+$vm+":"+$($ESXHost.name) 
 }
 ElseIf (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat2}){
  	$row = '' | select HostName, Status
 	$row.HostName = $($ESXHost.name)
	$row.Status ="Patched"
    $report += $row
    Write-Host "Matched pattern:"+$pat2+":"+$vm+":"+$($ESXHost.name) 
 }
  ElseIf (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat3}){
   	$row = '' | select HostName, Status
  	$row.HostName = $($ESXHost.name)
 	$row.Status ="Patched"
 	$report += $row
   	Write-Host "Matched pattern:"+$pat3+":"+$vm+":"+$($ESXHost.name) 
 }
 Else{
  	$row = '' | select HostName, Status
 	$row.HostName = $($ESXHost.name)
	$row.Status ="Un-Patched"
  	$report += $row
 	Write-Host "Nothing matched on for VM on host:"$($ESXHost.name) 
 }
 Stop-VM -VM $vm -RunAsync -Confirm:$false
 #you can delete this VM as well to avoid any disk space consumption or minimize the disk size
 $i++
}
#Log the report into this CSV file, you can provide your name
$report | Sort HostName | Export-Csv -Path "D:PatchStatus.csv"

#Disconnect the vCenter server
 Disconnect-VIServer -Confirm:$false
 

Sample CSV file result:
#TYPE Selected.System.String
“HostName”,”Status”
“10.20.30.20”,”Patched”
“10.20.30.21”,”Un-Patched”
“VMware vCenter Server 6.5.0 build-7515524″,”Patched”

Notes:
1. For the sake of simplicity I have hardcoded some values, please do change as per your environment.
2. If you have any comment/feedback on above script, please do provide.
3. There can be other ways to automate the same. Since I started with this approach, I continued. I haven’t yet thought about other approach (probably we may be able to just use datastore browser APIs to navigate/scan vmware.log file instead of downloading it outside).
4. I would recommend you to use this script as sample and make changes as needed. Let me know what changes you did or probably any other way you found.
5. I have tested this script on vCenter/ESXi 6.5 & it worked as expected for me. I will update this article as I update above script

Note: I would like you to understand “vMotion and EVC Information” section from KB, specially below lines. Let me know if you have any doubts I can share my understanding.

“In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated”

Some must read links:
1. VC 6.5 U1e release notes
2. VC 6.0 U3d release notes
3. VC 5.5 U3g release notes
4. Microsoft has released similar validation script for their Guest OS

I hope this will be helpful.

How to deploy vCenter HA using pyVmomi?

In part 1, we explored new vCenter HA APIs for managing vCenter HA. In part 2, we will focus on how to deploy vCenter HA using pyVmomi. As I pointed in part-1, managed object FailoverClusterConfigurator is responsible for deploying vCenter HA & below are the methods exposed by this managed object.

  • deployVcha_Task
  • configureVcha_Task
  • createPassiveNode_Task
  • createWitnessNode_Task
  • destroyVcha_Task

Let us start with first and important method i.e.  deployVcha_Task() : This method is really handy when user wants to deploy as well as configure vCenter HA in single call. I will encourage you to read vSphere API reference for more details on this method. This method assumes that you already deployed vCenter server,which would act as active node and created required vCenter HA network. As I wanted to deploy vCenter HA in basic mode, I created my self managed vCenter server as shown below & configured required vCenter HA network.

Before you start deploying your active node, please take a look at vCenter HA software & hardware requirement. I would also recommend you to just overview this & this documentation. If you need any help on deploying active node, please let me know, I would be happy to help you.

Lets take a look at code now. Note that detailed code documentation is added inside script itself for your easy reference & all below scripts are available on my github repo as well.


from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
# Deploying vCenter HA in basic mode using self managed VC 

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Parameters required are hardcoded below, please do change as per your environment.

vcha_network_name="VCHA" #port group name, I am using standard switch.
vcha_dc_name="IndiaDC"    #Datacenter name 
vcha_subnet_mask="255.255.255.0"  #Subnect mask for vCenter HA/Private network
active_vcha_ip="192.168.0.1"    # Active node vCenter HA IP
passive_vcha_ip="192.168.0.2"   # Passive node vCenter HA IP
witness_vcha_ip="192.168.0.3"   # Witness node vCenter HA IP
active_vcha_vm_name="vThinkBVM-VC1" #Active node/VC VM name
active_vc_username="Administrator@vsphere.local"  #Active VC username
active_vc_password="VMware#23"  #Active VC password
active_vc_url="https://10.61.34.35"  #Active VC public IP
active_vc_thumbprint="55:A9:C5:7E:0C:CD:46:26:D3:5C:C2:92:B7:0F:A7:91:E5:CD:0D:5D" #Active VC thumbprint
passive_vc_datastore="SharedVMFS-1"  #Passive node datastore
witness_vc_datastore="SharedVMFS-2"  #Witness node datastore

vcha=si.content.failoverClusterConfigurator #Getting managed object responsible for vCenter HA deployment

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
        obj = None
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for c in container.view:
                if name:
                        if c.name == name:
                                obj = c
                                break
                        else:
                                obj = None                      
        return obj


vcha_network=get_obj(content,[vim.Network],vcha_network_name) 
vcha_dc=get_obj(content,[vim.Datacenter],vcha_dc_name)

#I would highly recommend  to read vSphere API reference for "failoverClusterConfigurator", this will help to understand below specs.
 
deployment_spec=vim.vcha.FailoverClusterConfigurator.VchaClusterDeploymentSpec()

#Active node related data/parameter population
active_nw_config_spec=vim.vcha.FailoverClusterConfigurator.ClusterNetworkConfigSpec()
active_nw_config_spec.networkPortGroup=vcha_network
active_ipSettings=vim.vm.customization.IPSettings()
active_ipSettings.subnetMask = vcha_subnet_mask
active_ip_spec=vim.vm.customization.FixedIp()
active_ip_spec.ipAddress= active_vcha_ip
active_ipSettings.ip=active_ip_spec
active_nw_config_spec.ipSettings=active_ipSettings
deployment_spec.activeVcNetworkConfig=active_nw_config_spec

#Active node service locator 
active_vc_spec=vim.vcha.FailoverClusterConfigurator.SourceNodeSpec()
active_vc_vm=get_obj(content,[vim.VirtualMachine],active_vcha_vm_name)
active_vc_spec.activeVc=active_vc_vm
service_locator=vim.ServiceLocator()
cred=vim.ServiceLocator.NamePassword()
cred.username=active_vc_username
cred.password=active_vc_password
service_locator.credential=cred
service_locator.instanceUuid=si.content.about.instanceUuid
service_locator.url=active_vc_url  #Source active VC
service_locator.sslThumbprint=active_vc_thumprint
active_vc_spec.managementVc=service_locator
deployment_spec.activeVcSpec=active_vc_spec

#Passive node configuration spec
passive_vc_spec=vim.vcha.FailoverClusterConfigurator.PassiveNodeDeploymentSpec()
passive_ipSettings=vim.vm.customization.IPSettings()
passive_ipSettings.subnetMask = vcha_subnet_mask
passive_ip_spec=vim.vm.customization.FixedIp()
passive_ip_spec.ipAddress= passive_vcha_ip
passive_ipSettings.ip=passive_ip_spec
passive_vc_spec.ipSettings=passive_ipSettings
passive_vc_spec.folder=vcha_dc.vmFolder
passive_vc_spec.nodeName= active_vcha_vm_name+"-passive"
passive_datastore=get_obj(content,[vim.Datastore],passive_vc_datastore)
passive_vc_spec.datastore=passive_datastore
deployment_spec.passiveDeploymentSpec=passive_vc_spec

#Witness node configuration spec
witness_vc_spec=vim.vcha.FailoverClusterConfigurator.NodeDeploymentSpec()
witness_ipSettings=vim.vm.customization.IPSettings()
witness_ipSettings.subnetMask = vcha_subnet_mask
witness_ip_spec=vim.vm.customization.FixedIp()
witness_ip_spec.ipAddress= witness_vcha_ip
witness_ipSettings.ip=witness_ip_spec
witness_vc_spec.ipSettings=witness_ipSettings
witness_vc_spec.folder=vcha_dc.vmFolder
witness_vc_spec.nodeName=active_vcha_vm_name+"-witness"
witness_datastore=get_obj(content,[vim.Datastore],witness_vc_datastore)
witness_vc_spec.datastore=witness_datastore
deployment_spec.witnessDeploymentSpec=witness_vc_spec

# Calling the method we aimed to invoke by passing complete deployment spec

task= vcha.deployVcha_Task(deployment_spec)

if(task.info.state == "running"):
        print "VCHA deployment is started, it will take few minutes, please monitor web client for its completion"

As soon as we invoke deployVcha_Task() method, it deploys vCenter HA in basic mode where it also creates passive and witness node. Below is how it looks when looked into web client recent task pane.

We can see that “Deploy VCHA” task is running, it has in-turn cloned passive/witness nodes and initiated powereON operations. I waited for some time for deploy VCHA task to get completed. After completion, below is how vCenter HA VMs look from web client.

Note: You might have noticed that we need to have vCenter active node thumbprint to be passed in VCHA spec. I could get this thumbprint by using openssl command as shown here. In fact, there are some other ways to get it as well.

Now that we deployed VCHA successfully, let us take a look at how to destroy the vCenter HA using method “destroyVcha_Task()”: This method can be invoked when VCHA is disabled, isolated or its configuration is failed. In my case, I chose to disable the existing VCHA by using method “setClusterMode_Task()” , which we already explored in Part-I. Lets take a look at the code.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Destroy vCenter server HA
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Getting VCHA configurator managed object
vcha=si.content.failoverClusterConfigurator

#Getting VCHA cluster manager
vcha_cluster_manager=si.content.failoverClusterManager

# Setting vCenter HA to "disabled" mode.
task = vcha_cluster_manager.setClusterMode_Task("disabled")
while(task.info.state != "success"):
        continue

#Getting VCHA cluster mode
VCHA_mode=vcha_cluster_manager.getClusterMode()
if (VCHA_mode == "disabled"):
        vcha.destroyVcha_Task() #Destroing it
else:
        print "VCHA must be in disabled mode before destrying it"

When I executed above script, I could see from web client that vCenter HA mode got set to disabled followed by vCenter HA was getting destroyed. Please take a look at below screenshot to confirm it

I waited for few seconds & I could see, vCenter HA is destroyed as shown in below screenshot.

Note that once we destroy VCHA using above method, active node will continue to work fine & serve client requests. This API just destroys the VCHA configuration. It does not delete passive & witness node by itself. To delete passive & witness node, you will have to use regular VM destroy API (This option is provided in UI as part of vcenter HA destroy UI workflow)

Moving on to other methods exposed by managed object “FailoverClusterConfigurator”, it seems these methods (listed below) are directly suited when VCHA is deployed using advanced option

configureVcha_Task(),
createPassiveNode_Task(),
createWitnessNode_Task()

Since I had deployed vCenter HA using basic option, I will talk about above methods in my future post or will add samples on my github repo. On the other side, you can refer all the scripts discussed in part-1,2 & easily code around those as well. If you ask me, it would be really good exercise to understand vCenter HA APIs.

Notes:
1. All above scripts are available on my github repository as well. Even, I have a plan to write one nice python module for the same.
2. Note that, for the sake of simplicity, I have hard-coded some values & disabled certificate validation, hence please do appropriate changes as per your environment.
2. I highly recommend you to read Part 1
3. Finally, if you haven’t yet set up “pyVmomi” environment, refer my blog post

How to manage vCenter HA using vSphere python SDK pyVmomi? : Part 1

Recently I was undergoing “vSphere 6.5: Optimize and Scale” course and I was excited to see a dedicated module on “vCenter Server High Availability and Performance”. Without a doubt, vCenter HA was one of the highlights of vSphere 6.5 release. If you quickly want to understand vCenter HA, I highly recommend you to watch innovative white board style presentation by our own Adam Eckerle on VMware’s official youtube channel here

After completing the vCenter HA course lab, when I looked into vSphere 6.5 SOAP API reference, I could see there are 2 managed objects exposed as part of vCenter Server High Availability as follows.

1. FailoverClusterConfigurator
2. FailoverClusterManager

In part 1, I am going to demonstrate how can we manage vCenter HA by using 2nd managed object listed above i.e. FailoverClusterManager. This managed object provides operations to manage a vCenter High Availability Cluster (VCHA Cluster) and it assumes that vCenter HA is already deployed. Accordingly, in my case, basic vCenter HA was already deployed. Let’s now see how can we invoke methods exposed by this managed object and fetch/set some cool properties using pyVmomi. If you still haven’t set pvVmomi environment, please do configure it using my earlier post “getting started with pyVmomi”. Here we go.

If you look into API reference for managed object “FailoverClusterManager”, it has exposed total 4 methods i.e. getClusterMode(), GetVchaClusterHealth(), initiateFailover_Task() & setClusterMode_Task(). Let’s go one by one.

1. getClusterMode(): This method gets us the current mode that vCenter HA cluster is in: vCenter HA can be in one of the these 3 modes i.e. Enabled, Disabled or Maintenance mode. Let us look at what is the mode my vCenter HA cluster is in.


from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to get vCenter Server High Availability (VCHA) mode
# Below is Python 2.7.x code, which can be easily converted to python 3.x version

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.45.10", user="Administrator@vsphere.local", pwd="VMware!1",sslContext=s)
vcha=c.content.failoverClusterManager

VCHA_mode=vcha.getClusterMode()
print "VCHA Cluster mode::", VCHA_mode

Line 11 to 14: We have got “FailoverClusterManager” object, invoked getClusterMode() using it and finally printed the VCHA mode.

Output:
vmware@localhost:~$ python getVCHA_cluster_mode.py
VCHA Cluster mode:: enabled

Same is the VCHA cluster mode shown on below vSphere web client screenshot, is it not pretty simple to call this method?

2. GetVchaClusterHealth(): This method gives us overall health of vCenter HA.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to Get vCenter server HA health information

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.20.30", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha = c.content.failoverClusterManager

VchaClusterHealth = vcha.GetVchaClusterHealth()

vcha_health_Messages = VchaClusterHealth.healthMessages
print "VCHA Health messages::"
for health_data in vcha_health_Messages:
        print health_data.message

print "\nAdditional Information::",VchaClusterHealth.additionalInformation

vcha_runtime_info = VchaClusterHealth.runtimeInfo
print "\nVCHA Cluster Mode::",vcha_runtime_info.clusterMode
print "\nVCHA Cluster State::",vcha_runtime_info.clusterState

vcha_node_info = vcha_runtime_info.nodeInfo

print "\nVCHA Node information:"
for node in vcha_node_info:
        print node.nodeRole+":"+node.nodeIp+":"+node.nodeState

Line 13: We have invoked method “GetVchaClusterHealth()” & it returns VchaClusterHealth object.
Line 15 to 18: We are fetching the property “healthMessages” offered by object returned above. Since “healthMessages” are multiple (array), we iterated and printed each one of them.
Line 20 to 30: We fetched remaining properties of the object ” VchaClusterHealth”, such as vcha mode, vcha state & vcha nodeinfo etc. and printed them one by one.

Output:
vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
active:192.168.111.151:up
passive:192.168.111.152:up
witness:192.168.111.153:up

Note: Additional Information property returns “empty” array since VCHA cluster is in healthy mode. If cluster is not in healthy mode, it will provide additional info on what is causing the issue etc.

If you see below web client screenshot, it is matching perfectly with above output. For IPs, you can refer the 1st screenshot already posted above.

3.initiateFailover_Task(): By invoking this method, user can initiate a fail-over from active node to passive node.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

#Script to get initiate vCenter Server High Availability failover

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.160.20.40", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager
task = vcha.initiateFailover_Task(True)

while(task.info.state != "success"):
        continue
print "Initiate Failover task is completed"

Line 12 to 16: We invoked method “initiateFailover_Task(True)” and waited for task to complete. Please take a note that we are passing boolean “True” into this method, which indicates that this method will wait till on-going active-passive state replication to finish and finally it initiates the failover, other wise it will force the failover, which can lead to data loss.

As soon as I executed above script, I could see, as expected web client session was down and when I logged out of web client, I was happy to see below message on web client.

If you want to know how to initiate failover directly from web client, please take a look at below screenshot.

I waited for some more time and I could see web client was up & running again. Now I thought to check overall VCHA health to see whether VCHA is in healthy condition and active/passive nodes are actually exchanged or not. Accordingly I executed the script #2, which was written for getting VCHA health using GetVchaClusterHealth() and below was the output.

vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
passive:192.168.111.151:up
active:192.168.111.152:up
witness:192.168.111.153:up

As expected, VCHA was in enabled mode and healthy state, even active & passive node IPs were exchanged. Same was confirmed from web client as follows.

4.setClusterMode_Task(): Using this method user can change VCHA mode. Since my current mode is “enabled” & state as “healthy”, I can either set mode to “disabled” or “maintenance”. I decided to change VCHA cluster mode to “maintenance”. In maintenance mode, VCHA active/passive state replication is enabled but automatic failover is not allowed.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Script to set vCenter Server High Availability mode
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.1.2", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager

task = vcha.setClusterMode_Task("maintenance")

while(task.info.state != "success"):
        continue
print "VCHA mode is set to ::", vcha.getClusterMode()

Line 11 to 15: invoked “setClusterMode_Task()” method, waited till it completes and finally checked cluster mode by calling “getClusterMode()” method.

Output:
vmware@localhost:~$ python setVCHA_cluster_mode.py
VCHA mode is set to :: maintenance

Then immediately I looked into web client and I see VCHA mode was changed to “maintenance”as follows.

That is all, How cool is that! I really enjoyed writing this post and I hope you will enjoy as well. Update: I recently published Part-2 on “FailoverClusterConfigurator” managed object as well.

Further learning on vCenter Server HA

1. All above scripts are available on my github repository as well. I have a plan to write one nice python module for the same. Note that for the sake of simplicity I have hard-coded some values & disabled certificate validation, please do appropriate changes as per your environment.
2. VCHA walkthroughs
3. If you prefer PowerCLI to play around VCHA instead of pyVmomi, please take a look at nice VCHA PowerCLI module written by none other than “William Lam
4. VCHA performance white paper
5. Finally, if you want to start learning “pyVmomi”, refer my blog post

Getting started with vCenter server REST APIs using python

You already might have noticed, as part of vSphere 6.5, VMware introduced vCenter Server REST APIs. I really enjoyed playing around them using vCenter apiexplorer as well as Postman REST client. Recently, I wanted to code around these APIs using one of the programming languages and I am happy that I was able to do it using Python. I thought it is worth to share with you. In this blog post, I will take you through all the steps required to get started with vCenter REST API using python. Here we go.

Step 1. First important thing is to get familiar with vCenter server REST API documentation. Similar documentation is available from vCenter apiexplorer as well. I would recommend you to play with apiexplorer, which will not only make you familiar with documentation but also will enable you to quickly invoke these APIs against your vCenter server. I am sure you will be pretty impressed on REST API documentation when compared to older vSphere SOAP API documentation.

Step 2. Once you explore the first step , we can get started with environment creation for REST APIs using python. If you ask me, it is fairly easy and quick, we just need to have python installed machine, where we need to install “requests” python module as follows

$ pip install requests

Step 3. Now let us take a look at below python module developed to simplify REST API usage.

# Author: Vikas Shitole
# Website: www.vThinkBeyondVM.com
# Product: vCenter server
# Description: Python module for vCenter server REST APIs
# Reference:https://code.vmware.com/apis/191/vsphere-automation
# How to setup vCenter REST API environment?: Just have VM with python and install "requests" python library using pip

import requests
import json
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

s=requests.Session()
s.verify=False

# Function to get the vCenter server session
def get_vc_session(vcip,username,password):
         s.post('https://'+vcip+'/rest/com/vmware/cis/session',auth=(username,password))
         return s

# Function to get all the VMs from vCenter inventory
def get_vms(vcip):
        vms=s.get('https://'+vcip+'/rest/vcenter/vm')
        return vms

#Function to power on particular VM
def poweron_vm(vmmoid,vcip):
        s.post('https://'+vcip+'/rest/vcenter/vm/'+vmmoid+'/power/start')

# Function to power off particular VM
def poweroff_vm(vmmoid,vcip):
        s.post('https://'+vcip+'/rest/vcenter/vm/'+vmmoid+'/power/stop')

Above vcrest.py module is available on my github repo.

Let us understand above code.

Line 8: Imported powerful “requests” python library required to make API calls
Line 9: Imported “json” library required to parse json response we get from REST APIs
Line 10/11: Here we are disabling warnings related to SSL connection. In production, we should not disable it.
Line 13/14: Here we are creating Session object to have session persisted during the current request. If you see “s.verify” is set to False, it does mean that we are ignoring verifying SSL certificates. If you want to set it to true, please take a look at SSL Cert Verification section
Line 16 to 32: I have added 4 methods i.e. get_vc_session(), get_vms(), poweron_vm() & poweroff_vm(). We would be calling these methods from below sample script. If you see, in all the methods, I have used REST API documentation and called these APIs using “requests” library.

Step 4. Now that we understood above “vcrest.py” module, let us import above module into our script to demonstrate its usage.

# Description: Python sample to get VMs and its moid using vCenter server REST API.
# Reference:https://code.vmware.com/apis/191/vsphere-automation
#  Make sure you have "rest.py" file into your python directory.

import vcrest
import json
vcip="10.192.23.143" # vCenter server ip address/FQDN

#Get vCenter server session and can be used as needed. pass vcenter username & password
vcsession = vcrest.get_vc_session(vcip,"Administrator@vsphere.local","VMware1!")

#Get all the VMs from inventory using below method from "vcrest" module.
vms = vcrest.get_vms(vcip)

# Parsing the JSON response we got from above function call (it has all the Vms present in inventory
vm_response=json.loads(vms.text)
json_data=vm_response["value"]

print "VM names and its unique MOID"
print "============================"
for vm in json_data:
        print vm.get("name")+" :: "+vm.get("vm")
        #We are powering on all the VMs those are in powered off state
        if vm.get("power_state") == "POWERED_OFF":
                vcrest.poweron_vm(vm.get("vm"),vcip)

Above script i.e. vcrestsample.py is available on my github repo as well

Output :
vmware@localhost:~$ python vcrestsample.py
VM names and its unique MOID
============================
NTP-India-1 :: vm-42
NTP-PA-2 :: vm-43
WebApp-1 :: vm-44
vThinkBeyondVM :: vm-45
vmware@localhost:~$

Let us understand above script.

Line 5: Imported “vcrest” module we just discussed above.
Line 10: We are getting vCenter server session by calling function defined in “vcrest” module. We can use this session object as needed.
Line 13: We are getting all the VMs from inventory using “get_vms() function defined in “vcrest” module. Note that with this call, we will get JSON response as shown below, which we need to parse to fetch useful information.

{
  "value": [
    {
      "memory_size_MiB": 512,
      "vm": "vm-42",
      "name": "NTP-India-1",
      "power_state": "POWERED_OFF",
      "cpu_count": 1
    },
    {
      "memory_size_MiB": 512,
      "vm": "vm-43",
      "name": "NTP-PA-2",
      "power_state": "POWERED_OFF",
      "cpu_count": 1
    },
    {
      "memory_size_MiB": 512,
      "vm": "vm-44",
      "name": "WebApp-1",
      "power_state": "POWERED_ON",
      "cpu_count": 1
    },
    {
      "memory_size_MiB": 512,
      "vm": "vm-45",
      "name": "vThinkBeyondVM",
      "power_state": "POWERED_ON",
      "cpu_count": 1
    }
  ]
}

Line 16/17: As we got JSON response as pointed above, here we parsed it so that we can easily access as part of python dictionary.
Line 21 to 25: Iterating through dictionary and printing vm names & its moid (managed object id). Finally powering on VMs those are off.

That is all, is not it cool? Since we have REST APIs available for vCenter VM life cycle, VCSA, content library, tagging etc, there is lot to learn and play around. I will keep adding more methods to vcrest.py module. If you are interested in contributing to this module, let me know, it would be really great. In case, you would like to explore vCenter SOAP based APIs, please refer my last post