Category Archives: Vmware private AI and IaaS platform

Blog posts on VMware vSphere Kubernetes Service (VKS) and Private AI foundation

Tutorial: How to manage Enhanced vMotion Compatibility (EVC) using vSphere Python SDK pyVmomi? Part-I

HOME, Vmware private AI and IaaS platform, vSphere API, , , , ,

As part of recent security issues i.e. Spectre & Meltdown, I got an opportunity to play around  one of famous vSphere features i.e. Enhanced vMotion Compatibility (EVC) and  I thought this is the right time to explore EVC vSphere APIs . I see that EVC APIs are made public as part of vSphere 6.0 release though feature itself was introduced in vSphere 4.0. Since there is lot to learn about EVC vSphere APIs, I thought it is better to divide this topic in 2 parts. These parts will be kind of tutorial on EVC APIs and at the same time, we will touch upon EVC functionality as well. Before we start exploring EVC APIs, it is important that you understand what EVC is all about & why one needs to enable it on cluster? For quick learning, I think, no better resource than this EVC FAQ KB. In short, EVC makes it possible to mix different generations of CPU servers inside a cluster, which enables vMotion across these servers. Once EVC is enabled on vSphere cluster level, it ensures that all the servers across different CPU generation exposes same cpuid bits to its virtual machines.

When it comes to EVC APIs, ClusterEVCManager is the major managed object, which exposes 4 important methods as follows.

  •  ConfigureEvcMode_Task()
  •  DisableEvcMode_Task()
  • CheckConfigureEvcMode_Task()
  • CheckAddHostEvc_Task()

1. Getting max EVCMode key

Before we look into above methods, we need to get max EVCMode supported by each ESXi host to be added into EVC enabled cluster. Let us take a look at simple pyVmomi script for the same.

Below script is available on my github repo getMaxEVCMode.py
[python]
#Script to get Max EVC Mode supported on all the hosts in the cluster

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.160.50.60", user="Administrator@vsphere.local", pwd="VMware#12",sslContext=s)
content=si.content

# Your cluster name
cluster_name="vThinkBVMCluster"

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
obj = None
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name:
if c.name == name:
obj = c
break
else:
obj = None
return obj

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
print "ClusterName::"+cluster.name

# Get all the hosts available inside cluster
hosts = cluster.host

#Iterate through each host to get MaxEVC mode supported on the host
for host in hosts:

host_max_evcmode = host.summary.maxEVCModeKey

if host_max_evcmode == None:
print host.name+" does not support EVC"
else:
print host.name+"::"+host_max_evcmode

#Disconnect the vCenter server session.
atexit.register(Disconnect, si)
[/python]

Above script takes cluster name as input, get all the hosts inside the cluster and iterate through each host in order to get max EVCMode required to enable the EVC on right mode. Line #37 is what gets us “maxEVCModeKey”. Let us take look at how output looks like

Output :
vmware@localhost:~$ python getMaxEVCModeonHost.py
ClusterName::vThinkBVMCluster
10.160.20.30::intel-broadwell
10.160.20.31::intel-sandybridge
vmware@localhost:~$

You could see, there are 2 hosts inside the cluster and their max EVCMode keys are intel-broadwell and intel-sandybridge respectively. We need to choose right max EVCMode key to enable EVC on the cluster.

You may be wondering, can EVC modes supported per host be viewed from vSphere web client? Yes, it is, please take a look at below screenshot.

2. Enable EVC on cluster

Now that we have got max EVC mode per host, let us take a look at EVC API responsible to enable EVC on the cluster. i.e. ConfigureEvcMode_Task(). This API require EVC mode to be passed as parameter.

To start with, we will pass max EVCMode as “intel-broadwell” and see what happens. Lets take a look at script below.

Below script is available on my github repo enableEVC.py

[python]
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys

#Script to enable/disable EVC on cluster

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.160.50.60", user="Administrator@vsphere.local", pwd="VMware#12",sslContext=s)
content=si.content

#Your cluster input
cluster_name="vThinkBVMCluster"

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
obj = None
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name:
if c.name == name:
obj = c
break
else:
obj = None
return obj

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)

# Major EVC cluster manager object
evc_cluster_manager=cluster.EvcManager()

print "ClusterName::"+cluster.name

# Configure EVC mode by passing EVCMode
task=evc_cluster_manager.ConfigureEvcMode_Task("intel-broadwell")

atexit.register(Disconnect, si)

[/python]

Notice the line #40, where method is called to enable EVC. Right after executing above script, I looked into web client to see whether EVC is really enabled or not and below is what I observed.

Error displayed above is expected since we are enabling EVC on cluster with “intel-broadwell” EVCMode, which has additional cpuid bits (CPU feature set) those are NOT available on other host inside the cluster. Since “intel-broadwell” is latest intel CPU model, “intel-sandybridge” host does not have new features available. In other words, host with “intel-sandybridge” EVCMode has all the cpu features are available on “intel-broadwell” based host but not the vice-versa, hence we can only enable EVC on cluster with “intel-sandybridge or lower EVCMode.

Now before I execute above script with “intel-sandybridge” EVCMode, I powered on all the VMs available inside the cluster, once VMs are powered ON, I executed above script by editing EVCMode as “intel-sandybridge” on line #40. Since “intel-sandybridge” cpu feature set is available on “intel-broadwell”, you might be hoping that EVC will be successfully enabled on cluster. Let us take a look at web client screenshot.

You could see EVC is NOT enabled again. You might be wondering why is EVC not enabled even though EVCMode is right? As error states, since cluster has Powered-ON VMs those can potentially be using CPU features, which can not be available after enabling EVC on “intel-sandybridge” mode. Reason is EVC will mask additional CPU features (from broadwell host) to have same CPU features across all the hosts inside cluster. I hope you are following.

Now I powered-off and executed the script to enable EVC on “intel-sandybridge” mode. Let us take a look at below screenshot.

You can see EVC is enabled successfully, isn’t it cool?

Note: It is recommended to enable EVC on cluster with min of all hosts maxEVCMode key Ex. min (intel-sandybridge, intel-broadwell) = intel-sandybridge

3. Disabling EVC on cluster

Disabling EVC on cluster is fairly simple using DisableEvcMode_Task() , we have to just replace line #40 with below line.

[python]
task=evc_cluster_manager.DisableEvcMode_Task()
[/python]

After making change as specified above, I confirmed from web client that EVC is successfully disabled. Please take a look at below screenshot.

EVC disabled

You can use Task object returned by ConfigureEvcMode_Task() & DisableEvcMode_Task() methods for tracking the task progress, any errors, state etc.

Now that you learned how to configure, enable & disable EVC on cluster, let us now learn how to know what cpu features are masked by EVC, what cpu features are available on each hosts inside EVC enabled cluster. If you ask me, again it is very simple.

4. Exploring EVC Cluster state

How to get current EVCMode enabled on cluster ? How to know whether EVC is disabled on cluster or not? How to get cpu features available & features masked on EVC cluster? All these questions are answered as part of below script.

Below script available on my github repo getEVCClusterState.py
[python]
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
#Script to get Max EVC Mode supported on all the hosts in the cluster
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.160.50.60", user="Administrator@vsphere.local", pwd="VMware#12",sslContext=s)
content=si.content
cluster_name="vThinkBVMCluster"

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
obj = None
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name:
if c.name == name:
obj = c
break
else:
obj = None
return obj

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
evc_cluster_manager=cluster.EvcManager()

evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
print "Current EVC Mode::"+current_evcmode_key
else:
print "EVC is NOT enabled on the cluster"
quit()

feature_capabilities = evc_state.featureCapability

for capability in feature_capabilities:
print "Feature Capability\n"
print capability.featureName
print capability.key
print capability.value
print "————-"

features_masked = evc_state.featureMask

for mask in features_masked:
print "Feature Masked\n"
print mask.featureName
print mask.key
print mask.value
print "————-"
atexit.register(Disconnect, si)

[/python]

That is all for Part-1, I hope you enjoyed.

Update : Part-2 is ready as well, where we will explore remaining EVC APIs i.e. CheckConfigureEvcMode_Task() & CheckAddHostEvc_Task().

For more learning, EVC official documentation is here

How to deploy vCenter HA using pyVmomi?

HOME, Vmware private AI and IaaS platform, vSphere API, , ,

In part 1, we explored new vCenter HA APIs for managing vCenter HA. In part 2, we will focus on how to deploy vCenter HA using pyVmomi. As I pointed in part-1, managed object FailoverClusterConfigurator is responsible for deploying vCenter HA & below are the methods exposed by this managed object.

  • deployVcha_Task
  • configureVcha_Task
  • createPassiveNode_Task
  • createWitnessNode_Task
  • destroyVcha_Task

Let us start with first and important method i.e.  deployVcha_Task() : This method is really handy when user wants to deploy as well as configure vCenter HA in single call. I will encourage you to read vSphere API reference for more details on this method. This method assumes that you already deployed vCenter server,which would act as active node and created required vCenter HA network. As I wanted to deploy vCenter HA in basic mode, I created my self managed vCenter server as shown below & configured required vCenter HA network.

Before you start deploying your active node, please take a look at vCenter HA software & hardware requirement. I would also recommend you to just overview this & this documentation. If you need any help on deploying active node, please let me know, I would be happy to help you.

Lets take a look at code now. Note that detailed code documentation is added inside script itself for your easy reference & all below scripts are available on my github repo as well.

[python]

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
# Deploying vCenter HA in basic mode using self managed VC

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Parameters required are hardcoded below, please do change as per your environment.

vcha_network_name="VCHA" #port group name, I am using standard switch.
vcha_dc_name="IndiaDC" #Datacenter name
vcha_subnet_mask="255.255.255.0" #Subnect mask for vCenter HA/Private network
active_vcha_ip="192.168.0.1" # Active node vCenter HA IP
passive_vcha_ip="192.168.0.2" # Passive node vCenter HA IP
witness_vcha_ip="192.168.0.3" # Witness node vCenter HA IP
active_vcha_vm_name="vThinkBVM-VC1" #Active node/VC VM name
active_vc_username="Administrator@vsphere.local" #Active VC username
active_vc_password="VMware#23" #Active VC password
active_vc_url="https://10.61.34.35" #Active VC public IP
active_vc_thumbprint="55:A9:C5:7E:0C:CD:46:26:D3:5C:C2:92:B7:0F:A7:91:E5:CD:0D:5D" #Active VC thumbprint
passive_vc_datastore="SharedVMFS-1" #Passive node datastore
witness_vc_datastore="SharedVMFS-2" #Witness node datastore

vcha=si.content.failoverClusterConfigurator #Getting managed object responsible for vCenter HA deployment

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
obj = None
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name:
if c.name == name:
obj = c
break
else:
obj = None
return obj

vcha_network=get_obj(content,[vim.Network],vcha_network_name)
vcha_dc=get_obj(content,[vim.Datacenter],vcha_dc_name)

#I would highly recommend to read vSphere API reference for "failoverClusterConfigurator", this will help to understand below specs.

deployment_spec=vim.vcha.FailoverClusterConfigurator.VchaClusterDeploymentSpec()

#Active node related data/parameter population
active_nw_config_spec=vim.vcha.FailoverClusterConfigurator.ClusterNetworkConfigSpec()
active_nw_config_spec.networkPortGroup=vcha_network
active_ipSettings=vim.vm.customization.IPSettings()
active_ipSettings.subnetMask = vcha_subnet_mask
active_ip_spec=vim.vm.customization.FixedIp()
active_ip_spec.ipAddress= active_vcha_ip
active_ipSettings.ip=active_ip_spec
active_nw_config_spec.ipSettings=active_ipSettings
deployment_spec.activeVcNetworkConfig=active_nw_config_spec

#Active node service locator
active_vc_spec=vim.vcha.FailoverClusterConfigurator.SourceNodeSpec()
active_vc_vm=get_obj(content,[vim.VirtualMachine],active_vcha_vm_name)
active_vc_spec.activeVc=active_vc_vm
service_locator=vim.ServiceLocator()
cred=vim.ServiceLocator.NamePassword()
cred.username=active_vc_username
cred.password=active_vc_password
service_locator.credential=cred
service_locator.instanceUuid=si.content.about.instanceUuid
service_locator.url=active_vc_url #Source active VC
service_locator.sslThumbprint=active_vc_thumprint
active_vc_spec.managementVc=service_locator
deployment_spec.activeVcSpec=active_vc_spec

#Passive node configuration spec
passive_vc_spec=vim.vcha.FailoverClusterConfigurator.PassiveNodeDeploymentSpec()
passive_ipSettings=vim.vm.customization.IPSettings()
passive_ipSettings.subnetMask = vcha_subnet_mask
passive_ip_spec=vim.vm.customization.FixedIp()
passive_ip_spec.ipAddress= passive_vcha_ip
passive_ipSettings.ip=passive_ip_spec
passive_vc_spec.ipSettings=passive_ipSettings
passive_vc_spec.folder=vcha_dc.vmFolder
passive_vc_spec.nodeName= active_vcha_vm_name+"-passive"
passive_datastore=get_obj(content,[vim.Datastore],passive_vc_datastore)
passive_vc_spec.datastore=passive_datastore
deployment_spec.passiveDeploymentSpec=passive_vc_spec

#Witness node configuration spec
witness_vc_spec=vim.vcha.FailoverClusterConfigurator.NodeDeploymentSpec()
witness_ipSettings=vim.vm.customization.IPSettings()
witness_ipSettings.subnetMask = vcha_subnet_mask
witness_ip_spec=vim.vm.customization.FixedIp()
witness_ip_spec.ipAddress= witness_vcha_ip
witness_ipSettings.ip=witness_ip_spec
witness_vc_spec.ipSettings=witness_ipSettings
witness_vc_spec.folder=vcha_dc.vmFolder
witness_vc_spec.nodeName=active_vcha_vm_name+"-witness"
witness_datastore=get_obj(content,[vim.Datastore],witness_vc_datastore)
witness_vc_spec.datastore=witness_datastore
deployment_spec.witnessDeploymentSpec=witness_vc_spec

# Calling the method we aimed to invoke by passing complete deployment spec

task= vcha.deployVcha_Task(deployment_spec)

if(task.info.state == "running"):
print "VCHA deployment is started, it will take few minutes, please monitor web client for its completion"

[/python]

As soon as we invoke deployVcha_Task() method, it deploys vCenter HA in basic mode where it also creates passive and witness node. Below is how it looks when looked into web client recent task pane.

We can see that “Deploy VCHA” task is running, it has in-turn cloned passive/witness nodes and initiated powereON operations. I waited for some time for deploy VCHA task to get completed. After completion, below is how vCenter HA VMs look from web client.

Note: You might have noticed that we need to have vCenter active node thumbprint to be passed in VCHA spec. I could get this thumbprint by using openssl command as shown here. In fact, there are some other ways to get it as well.

Now that we deployed VCHA successfully, let us take a look at how to destroy the vCenter HA using method “destroyVcha_Task()”: This method can be invoked when VCHA is disabled, isolated or its configuration is failed. In my case, I chose to disable the existing VCHA by using method “setClusterMode_Task()” , which we already explored in Part-I. Lets take a look at the code.

[python]
from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Destroy vCenter server HA
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.34.35", user="Administrator@vsphere.local", pwd="VMware#23",sslContext=s)
content=si.content

#Getting VCHA configurator managed object
vcha=si.content.failoverClusterConfigurator

#Getting VCHA cluster manager
vcha_cluster_manager=si.content.failoverClusterManager

# Setting vCenter HA to "disabled" mode.
task = vcha_cluster_manager.setClusterMode_Task("disabled")
while(task.info.state != "success"):
continue

#Getting VCHA cluster mode
VCHA_mode=vcha_cluster_manager.getClusterMode()
if (VCHA_mode == "disabled"):
vcha.destroyVcha_Task() #Destroing it
else:
print "VCHA must be in disabled mode before destrying it"

[/python]

When I executed above script, I could see from web client that vCenter HA mode got set to disabled followed by vCenter HA was getting destroyed. Please take a look at below screenshot to confirm it

I waited for few seconds & I could see, vCenter HA is destroyed as shown in below screenshot.

Note that once we destroy VCHA using above method, active node will continue to work fine & serve client requests. This API just destroys the VCHA configuration. It does not delete passive & witness node by itself. To delete passive & witness node, you will have to use regular VM destroy API (This option is provided in UI as part of vcenter HA destroy UI workflow)

Moving on to other methods exposed by managed object “FailoverClusterConfigurator”, it seems these methods (listed below) are directly suited when VCHA is deployed using advanced option

configureVcha_Task(),
createPassiveNode_Task(),
createWitnessNode_Task()

Since I had deployed vCenter HA using basic option, I will talk about above methods in my future post or will add samples on my github repo. On the other side, you can refer all the scripts discussed in part-1,2 & easily code around those as well. If you ask me, it would be really good exercise to understand vCenter HA APIs.

Notes:
1. All above scripts are available on my github repository as well. Even, I have a plan to write one nice python module for the same.
2. Note that, for the sake of simplicity, I have hard-coded some values & disabled certificate validation, hence please do appropriate changes as per your environment.
2. I highly recommend you to read Part 1
3. Finally, if you haven’t yet set up “pyVmomi” environment, refer my blog post

How to manage vCenter HA using vSphere python SDK pyVmomi? : Part 1

HOME, Vmware private AI and IaaS platform, vSphere API, , , ,

Recently I was undergoing “vSphere 6.5: Optimize and Scale” course and I was excited to see a dedicated module on “vCenter Server High Availability and Performance”. Without a doubt, vCenter HA was one of the highlights of vSphere 6.5 release. If you quickly want to understand vCenter HA, I highly recommend you to watch innovative white board style presentation by our own Adam Eckerle on VMware’s official youtube channel here

After completing the vCenter HA course lab, when I looked into vSphere 6.5 SOAP API reference, I could see there are 2 managed objects exposed as part of vCenter Server High Availability as follows.

1. FailoverClusterConfigurator
2. FailoverClusterManager

In part 1, I am going to demonstrate how can we manage vCenter HA by using 2nd managed object listed above i.e. FailoverClusterManager. This managed object provides operations to manage a vCenter High Availability Cluster (VCHA Cluster) and it assumes that vCenter HA is already deployed. Accordingly, in my case, basic vCenter HA was already deployed. Let’s now see how can we invoke methods exposed by this managed object and fetch/set some cool properties using pyVmomi. If you still haven’t set pvVmomi environment, please do configure it using my earlier post “getting started with pyVmomi”. Here we go.

If you look into API reference for managed object “FailoverClusterManager”, it has exposed total 4 methods i.e. getClusterMode(), GetVchaClusterHealth(), initiateFailover_Task() & setClusterMode_Task(). Let’s go one by one.

1. getClusterMode(): This method gets us the current mode that vCenter HA cluster is in: vCenter HA can be in one of the these 3 modes i.e. Enabled, Disabled or Maintenance mode. Let us look at what is the mode my vCenter HA cluster is in.

[python]

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to get vCenter Server High Availability (VCHA) mode
# Below is Python 2.7.x code, which can be easily converted to python 3.x version

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.45.10", user="Administrator@vsphere.local", pwd="VMware!1",sslContext=s)
vcha=c.content.failoverClusterManager

VCHA_mode=vcha.getClusterMode()
print "VCHA Cluster mode::", VCHA_mode

[/python]

Line 11 to 14: We have got “FailoverClusterManager” object, invoked getClusterMode() using it and finally printed the VCHA mode.

Output:
vmware@localhost:~$ python getVCHA_cluster_mode.py
VCHA Cluster mode:: enabled

Same is the VCHA cluster mode shown on below vSphere web client screenshot, is it not pretty simple to call this method?

2. GetVchaClusterHealth(): This method gives us overall health of vCenter HA.

[python]
from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Script to Get vCenter server HA health information

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.20.30", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha = c.content.failoverClusterManager

VchaClusterHealth = vcha.GetVchaClusterHealth()

vcha_health_Messages = VchaClusterHealth.healthMessages
print "VCHA Health messages::"
for health_data in vcha_health_Messages:
print health_data.message

print "\nAdditional Information::",VchaClusterHealth.additionalInformation

vcha_runtime_info = VchaClusterHealth.runtimeInfo
print "\nVCHA Cluster Mode::",vcha_runtime_info.clusterMode
print "\nVCHA Cluster State::",vcha_runtime_info.clusterState

vcha_node_info = vcha_runtime_info.nodeInfo

print "\nVCHA Node information:"
for node in vcha_node_info:
print node.nodeRole+":"+node.nodeIp+":"+node.nodeState

[/python]

Line 13: We have invoked method “GetVchaClusterHealth()” & it returns VchaClusterHealth object.
Line 15 to 18: We are fetching the property “healthMessages” offered by object returned above. Since “healthMessages” are multiple (array), we iterated and printed each one of them.
Line 20 to 30: We fetched remaining properties of the object ” VchaClusterHealth”, such as vcha mode, vcha state & vcha nodeinfo etc. and printed them one by one.

Output:
vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
active:192.168.111.151:up
passive:192.168.111.152:up
witness:192.168.111.153:up

Note: Additional Information property returns “empty” array since VCHA cluster is in healthy mode. If cluster is not in healthy mode, it will provide additional info on what is causing the issue etc.

If you see below web client screenshot, it is matching perfectly with above output. For IPs, you can refer the 1st screenshot already posted above.

3.initiateFailover_Task(): By invoking this method, user can initiate a fail-over from active node to passive node.

[python]
from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

#Script to get initiate vCenter Server High Availability failover

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.160.20.40", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager
task = vcha.initiateFailover_Task(True)

while(task.info.state != "success"):
continue
print "Initiate Failover task is completed"
[/python]

Line 12 to 16: We invoked method “initiateFailover_Task(True)” and waited for task to complete. Please take a note that we are passing boolean “True” into this method, which indicates that this method will wait till on-going active-passive state replication to finish and finally it initiates the failover, other wise it will force the failover, which can lead to data loss.

As soon as I executed above script, I could see, as expected web client session was down and when I logged out of web client, I was happy to see below message on web client.

If you want to know how to initiate failover directly from web client, please take a look at below screenshot.

I waited for some more time and I could see web client was up & running again. Now I thought to check overall VCHA health to see whether VCHA is in healthy condition and active/passive nodes are actually exchanged or not. Accordingly I executed the script #2, which was written for getting VCHA health using GetVchaClusterHealth() and below was the output.

vmware@localhost:~$ python get_VCHA_health.py
VCHA Health messages::
PostgreSQL replication mode is Synchronous.
Appliance configuration is in sync.
Appliance state is in sync.
Appliance sqlite db is in sync.

Additional Information:: (vmodl.LocalizableMessage) []

VCHA Cluster Mode:: enabled

VCHA Cluster State:: healthy

VCHA Node information:
passive:192.168.111.151:up
active:192.168.111.152:up
witness:192.168.111.153:up

As expected, VCHA was in enabled mode and healthy state, even active & passive node IPs were exchanged. Same was confirmed from web client as follows.

4.setClusterMode_Task(): Using this method user can change VCHA mode. Since my current mode is “enabled” & state as “healthy”, I can either set mode to “disabled” or “maintenance”. I decided to change VCHA cluster mode to “maintenance”. In maintenance mode, VCHA active/passive state replication is enabled but automatic failover is not allowed.

[python]
from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Script to set vCenter Server High Availability mode
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
c= SmartConnect(host="10.192.1.2", user="Administrator@vsphere.local", pwd="VMW!23A",sslContext=s)

vcha=c.content.failoverClusterManager

task = vcha.setClusterMode_Task("maintenance")

while(task.info.state != "success"):
continue
print "VCHA mode is set to ::", vcha.getClusterMode()

[/python]

Line 11 to 15: invoked “setClusterMode_Task()” method, waited till it completes and finally checked cluster mode by calling “getClusterMode()” method.

Output:
vmware@localhost:~$ python setVCHA_cluster_mode.py
VCHA mode is set to :: maintenance

Then immediately I looked into web client and I see VCHA mode was changed to “maintenance”as follows.

That is all, How cool is that! I really enjoyed writing this post and I hope you will enjoy as well. Update: I recently published Part-2 on “FailoverClusterConfigurator” managed object as well.

Further learning on vCenter Server HA

1. All above scripts are available on my github repository as well. I have a plan to write one nice python module for the same. Note that for the sake of simplicity I have hard-coded some values & disabled certificate validation, please do appropriate changes as per your environment.
2. VCHA walkthroughs
3. If you prefer PowerCLI to play around VCHA instead of pyVmomi, please take a look at nice VCHA PowerCLI module written by none other than “William Lam”
4. VCHA performance white paper
5. Finally, if you want to start learning “pyVmomi”, refer my blog post

One of cool vSphere features: Tutorial:How SDRS integration with storage profiles works?

HOME, Vmware private AI and IaaS platform, , , , , ,

SDRS integration with storage profiles was one of the cool features I was waiting for since vSphere 5.5. In vSphere 6.0 GA, there were some public references of its support but no detailed documentation was available. As per this KB I am happy to know that in vCenter Server 6.0.0b and above, Storage DRS is fully supported to have storage profile enforcement. Now SDRS is aware of storage policies in SPBM (Storage Policy Based Management). Recently I got opportunity to play with this feature and I thought to write a detailed post which can help all. Here we go

As part of this SDRS integration with storage profiles/policy, One SDRS cluster level advanced option is introduced i.e. “EnforceStorageProfiles”. Advanced option EnforceStorageProfiles takes one of these integer values, 0,1 or 2 where the default value is 0.

When option is set to 0, it is mean that NO storage profile enforcement on the SDRS cluster.

When option is set to 1, it is mean that there is storage profile SOFT enforcement on the SDRS cluster. SDRS will try its best to comply with storage profile/policy. However if required, SDRS will violate the storage profile compliant.

When option is set to 2, it is mean that there is storage profile HARD enforcement on the SDRS cluster. In any case, SDRS will not violate the storage profile.

Refer KB 2142765 in order to know how to configure SDRS advanced option “EnforceStorageProfiles” using vSphere web client and vSphere client.

Now I will walk you through vSphere web client workflows as follows in order to play with this cool feature. This is kind of Tutorial.

1. Configuring SDRS advanced option to enable SOFT (1) storage profile enforcement with Storage DRS.

-Create a SDRS cluster (aka POD) with 3 datastores namely DS1, DS2 and DS3.
-Go to SDRS Cluster >> Manage >> Settings >> Storage DRS web client workflow. Click on Edit and configure the option “EnforceStorageProfiles” as shown in below screenshot.

Adding SDRS option

You could see I had set this option to 1 i.e. SOFT enforcement

2. Creating 2 Tags named “Gold” and Silver”.

Tags are required to attach to all the datastores in datastore cluster (as per datastore capability) as well as to create tag based VM storage policies.

– Go to Home >> Tags >> click on New tag and create a “Gold” tag with “FC SAN” category as shown in below screenshot

Create Tag

Similar to Gold tag, please create a “Silver” tag with “iSCSI SAN” category. At this point make sure you have 2 tags, Gold and Silver.

3. Assign the tags to datastores in SDRS POD.

-Go to SDRS POD >>DS1 >> Manage >> Tags >> Click on Assign tags as shown in below screenshot

Assign tags to Datastore

Please assign “Gold” tag to the DS1 as well as DS2. Finally assign “Silver” tag to datastore DS3. You can see the assigned tags for each datastore as shown below screenshot.

Assiged-tag-on-DS3

You can see in above screenshot, DS3 was assigned with “Silver” tag.

4. Creating VM storage policy

– Go to Home >> Policies and profiles >> VM storage policy >> Click on the create VM storage policy as specified in below screen shot.

Create VM storage policy

-Once you click on “Create a new VM storage policy”, specify policy name as “Gold VM Storage Policy” and Click next.
– On Rule-Sets window, click on Add tag-based rule and select “Gold” tag under “FC SAN” category as shown in below screenshot.
Adding tag based rule
– On Storage compatibility UI page, keep default and click next. Finally click on finish at the end as shown below screenshot.
VM storage policy finish

– Similar to “Gold VM Storage Policy” creation, repeat above steps to create “Silver VM Storage Policy” based on “Silver” tag. At this moment you will have 2 VM storage policies “Gold VM Storage Policy” and “Silver VM Storage Policy”.

Now we are ready to play with SDRS with Storage profile integration feature. Lets try out some workflows from SDRS perspective to verify whether profiles are being considered by SDRS. Note that in the very first step we have set the “EnforceStorageProfiles” SDRS option to 1 i.e. SOFT profile enforcement.

1. Create VM workflow: i.e. SDRS initial placement workflow.

– From web client. Start create VM workflow >> Give VM name as “GoldVM” >> select compute resource >> under select storage, select VM storage policy as “Gold VM Storage Policy” that we created in last section and storage as SDRS POD we created initially as shown in below screenshot

Warning on selecting policy create VM

If you see in above screenshot, SDRS POD is listed under “incompatible” storage. Note that this is expected as SDRS POD has 3 datastores and 2 of those have “Gold” tag attached and 3rd has “Silver” attached. Warning shown in above screenshot also shows the same. i.e. Datastore does not satisfy compatibility since it does not support one or more required properties. Tags with name “Gold” not found on datastore”. As I said, this is expected as we have selected “Gold VM Storage policy” and NOT all the datastores in SDRS POD have “Gold” tag attached. Overall, no panic, just click next.

-Finally on the finish page of VM creation, we can see SDRS initial placement recommendations as shown in below screenshotSDRS recommendations

From above screen shot you can understand that SDRS placement recommendations on DS2 are absolutely spot on as DS2 datastore has “Gold” tag attached.

Now you can check whether the created VM “GoldVM” is placed on the datastore compatible with Gold VM storage policy by SDRS. You could see in below screenshot, SDRS has placed the GoldVM on right datastore and VM storage policies are compliant.

VM-complaint

Below is the screenshot for the “GoldVM” VM files. You can see all the VM files are placed on datastore DS2 which has Gold tag attached. is not it cool?

Datastore files on DS2

Based on above screenshots we can say that SDRS is aware of VM storage policies. Now lets try another SDRS workflow i.e. Putting datastore in maintenance mode.

2. Putting datastore in maintenance mode.

As we know that all the “GoldVM” VM files are in datastore DS2 as expected, now we will put datastore DS2 into maintenance mode and we expect SDRS will storage vMotion the VM to DS1 as DS1 is the only other datastore where “Gold” tag is attached.

From web client, Go to SDRS POD >> DS2 >> right click on DS2 and select “Enter Maintenance mode”. As soon as we click Enter Maintenance mode we get SDRS migration recommendations in order to evacuate the DS2 datastore as shown in below screenshot.

After putting into MM

You could see in above screenshot, SDRS has recommended to migrate VM files to DS1 as expected. How cool is that?

To see if SOFT profile integration is working fine, I went ahead and put DS1 also into maintenance mode and I observed VM files were moved to DS3 as expected as we have set the “EnforceStorageProfiles” SDRS option to 1 i.e. SOFT profile enforcement (DS2 is already in maintenamce mode.).

3. Configuring SDRS affinity rules.

I tested first case with VMDK anti affinity (Intra VM) and second case with VM anti affinity(Inter VM), in both the cases, I have observed affinity rules will have higher precedence over storage profiles attached to the VMs/datastores. SDRS will first obey the affinity rules first.

Finally I even tested some SDRS workflows where I filled the datastores so that they cross the SDRS space threshold and finally invoked SDRS to see if SDRS really does consider storage profiles while generating space migration recommendations. I observed SDRS does consider storage profiles as expected.

Overall, above tutorial will help you to get started. Testing SDRS with HARD storage profile enforcement I leave it to you. Enjoy!

One caveat : As noted in this KB, vCloud Director (vCD) backed by SDRS cluster does NOT support Soft (1) or Hard (2) storage profile enforcements. vCloud Director (vCD) will work well with Default (0) option.

Let me know if you have comments.

Want to vMotion a VM from one vCenter server to another vCenter using vSphere API?

HOME, Vmware private AI and IaaS platform, vSphere API, , ,

As part of one of customers case, I was testing vMotion across two 6.0 vCenter servers those are connected to the same SSO domain. As we know already, when 2 vCenter servers are connected to the same SSO domain, you can manage both VCs using web client. Initially I tested vMotion from web client and later in order to automate bulk VMs migration across vCenter, I was looking for vSphere APIs and I could find below 2 vSphere APIs.

1. RelocateVM_Task() under VirtualMachine Managed object
2. placevm() under ClusterComputeResource managed object

I went through above vSphere API reference and I could see RelocateVM_Task() API was already there. However, in vSphere 6.0, it is improved greatly in order to support vMotion between 2 vCenters. On the other hand, placeVM() API was new brand API introduced in vSphere 6.0. Initially, I decided to try RelocateVM_Task() API for automating vMotion across 2 vCenters with same SSO domain. After automating this, I tested my java SDK script on vCenters with same SSO domain and it worked with no any issues. Later, I just thought lets give a try across vCenters those are connected to the different SSO domains and to my surprise, it worked like charm. is it not super cool? How easy it is now to migrate your workloads across data-centers/VCs/Clusters!.

So vMotion between 2 completely different vCenter is supported in vSphere 6.0 but there is NO UI support from web client at the moment. If you want to this functionality i.e. vMotion between 2 VCs with different SSO domains. vSphere API is the only way.

After successful vMotion across different SSO domains, I was really excited and thought to play with this little more by creating a DRS cluster in both VCs. I created a VM-VM affinity rule with couple of VMs in DRS cluster in VC1 as shown in below screenshot.

VC1_Rule_before_vMotion

Now I initiated vMotion from first VC to DRS cluster in second VC using same Java SDK script and I could see the DRS rule associated the migrated VM also got migrated as shown in below screen shot. How cool is that!

VC2_Rule_After_vMotion

Below is the complete code sample which can help you quickly to vMotion from a VC to other VC.

Note that this sample works fine with VCs with same SSO or VCs with different SSO. You do not even need to have shared storage between both VCs. This script will work fine within the same VC as well (with some change).

Same script is available on my git hub repository: ExVC_vMotion.java

[java]<code lang="java">
//:: # Author: Vikas Shitole
//:: # Website: www.vThinkBeyondVM.com
//:: # Product/Feature: vCenter Server/DRS/vMotion
//:: # Description: Extended Cross VC vMotion using RelocateVM_Task() API. vMotion between vCenters (with same SSO domain or different SSO domain)</code>

package com.vmware.yavijava;

import java.net.MalformedURLException;
import java.net.URL;
import com.vmware.vim25.ManagedObjectReference;
import com.vmware.vim25.ServiceLocator;
import com.vmware.vim25.ServiceLocatorCredential;
import com.vmware.vim25.ServiceLocatorNamePassword;
import com.vmware.vim25.VirtualDevice;
import com.vmware.vim25.VirtualDeviceConfigSpec;
import com.vmware.vim25.VirtualDeviceConfigSpecOperation;
import com.vmware.vim25.VirtualDeviceDeviceBackingInfo;
import com.vmware.vim25.VirtualEthernetCard;
import com.vmware.vim25.VirtualMachineMovePriority;
import com.vmware.vim25.VirtualMachineRelocateSpec;
import com.vmware.vim25.mo.ClusterComputeResource;
import com.vmware.vim25.mo.Datastore;
import com.vmware.vim25.mo.Folder;
import com.vmware.vim25.mo.HostSystem;
import com.vmware.vim25.mo.InventoryNavigator;
import com.vmware.vim25.mo.ServiceInstance;
import com.vmware.vim25.mo.VirtualMachine;

public class ExVC_vMotion {

public static void main(String[] args) throws Exception {
if(args.length!=7)
{
//Parameters you need to pass
System.out.println("Usage: ExVC_vMotion srcVCIP srcVCusername srcVCpassword destVCIP destVCusername destVCpassword destHostIP");
System.exit(-1);
}

URL url1 = null;
try
{
url1 = new URL("https://"+args[0]+"/sdk");
} catch ( MalformedURLException urlE)
{
System.out.println("The URL provided is NOT valid. Please check it.");
System.exit(-1);
}

String srcusername = args[1];
String srcpassword = args[2];
String DestVC=args[3];
String destusername=args[4];
String destpassword = args[5];
String destvmhost=args[6];

//Hardcoded parameters for simplification
String vmName="VM1"; //VM name to be migrated
String vmNetworkName="VM Network"; //destination vSphere VM port group name where VM will be migrated
String destClusterName="ClusterVC2"; //Destination VC cluster where VM will be migrated
String destdatastoreName="DS1"; //destination datastore where VM will be migrated
String destVCThumpPrint="c7:bc:0c:a3:15:35:57:bd:fe:ac:60:bf:87:25:1c:07:a9:31:50:85"; //SSL Thumbprint (SHA1) of the destination VC

// Initialize source VC
ServiceInstance vc1si = new ServiceInstance(url1, srcusername,
srcpassword, true);
URL url2 = null;
try
{
url2 = new URL("https://"+DestVC+"/sdk");
} catch ( MalformedURLException urlE)
{
System.out.println("The URL provided is NOT valid. Please check it.");
System.exit(-1);
}

// Initialize destination VC

ServiceInstance vc2si = new ServiceInstance(url2, destusername,
destpassword, true);
Folder vc1rootFolder = vc1si.getRootFolder();
Folder vc2rootFolder = vc2si.getRootFolder();

//Virtual Machine Object to be migrated
VirtualMachine vm = null;
vm = (VirtualMachine) new InventoryNavigator(vc1rootFolder)
.searchManagedEntity("VirtualMachine", vmName);

//Destination host object where VM will be migrated
HostSystem host = null;
host = (HostSystem) new InventoryNavigator(vc2rootFolder)
.searchManagedEntity("HostSystem", destvmhost);
ManagedObjectReference hostMor=host.getMOR();

//Destination cluster object creation
ClusterComputeResource cluster = null;
cluster = (ClusterComputeResource) new InventoryNavigator(vc2rootFolder)
.searchManagedEntity("ClusterComputeResource", destClusterName);

//Destination datastore object creation
Datastore ds=null;
ds = (Datastore) new InventoryNavigator(vc2rootFolder)
.searchManagedEntity("Datastore", destdatastoreName);
ManagedObjectReference dsMor=ds.getMOR();

VirtualMachineRelocateSpec vmSpec=new VirtualMachineRelocateSpec();
vmSpec.setDatastore(dsMor);
vmSpec.setHost(hostMor);
vmSpec.setPool(cluster.getResourcePool().getMOR());

//VM device spec for the VM to be migrated
VirtualDeviceConfigSpec vdcSpec=new VirtualDeviceConfigSpec();
VirtualDevice[] devices= vm.getConfig().getHardware().getDevice();
for(VirtualDevice device:devices){

if(device instanceof VirtualEthernetCard){

VirtualDeviceDeviceBackingInfo vddBackingInfo= (VirtualDeviceDeviceBackingInfo) device.getBacking();
vddBackingInfo.setDeviceName(vmNetworkName);
device.setBacking(vddBackingInfo);
vdcSpec.setDevice(device);
}

}

vdcSpec.setOperation(VirtualDeviceConfigSpecOperation.edit);
VirtualDeviceConfigSpec[] vDeviceConSpec={vdcSpec};
vmSpec.setDeviceChange(vDeviceConSpec);

//Below is code for ServiceLOcator which is key for this vMotion happen
ServiceLocator serviceLoc=new ServiceLocator();
ServiceLocatorCredential credential=new ServiceLocatorNamePassword();
((ServiceLocatorNamePassword) credential).setPassword(destpassword);
((ServiceLocatorNamePassword) credential).setUsername(destusername);
serviceLoc.setCredential(credential);

String instanceUuid=vc2si.getServiceContent().getAbout().getInstanceUuid();
serviceLoc.setInstanceUuid(instanceUuid);
serviceLoc.setSslThumbprint(destVCThumpPrint);
serviceLoc.setUrl("https://"+DestVC);
vmSpec.setService(serviceLoc);
System.out.println("VM relocation started….please wait");
boolean flag=false;
vm.relocateVM_Task(vmSpec, VirtualMachineMovePriority.highPriority);
flag=true;
if(flag){
System.out.println("VM is relocated to 2nd vCenter server");
}
vc1si.getServerConnection().logout();
vc2si.getServerConnection().logout();
}
}

[/java]

Notes:
– There is one imp parameter you need to pass into migration spec i.e. Destination VC Thumbprint. There are several ways to get it as shown here. I personally used below way to get the destination VC thumbprint.
From google chrome browser : URL box of the VC (besides https:// lock symbol)>>view site information >> Certificate information >> Details >> Scroll down till last as shown in below screenshot

Thumbprint

– For the sake of simplicity, I have hard-coded some parameters, you can change it based on your environment.
– You can scale the same code to vMotion multiple VMs across vCenter server
– As I said, there is another API for vMotion across VCs, please take a look at this post on  placevm()

If you want to automate the same use case using PowerCLI, here is great post by William Lam.

If you have still not setup your YAVI JAVA Eclipse environment:Getting started tutorial

Important tutorials to start with: Part I & Part II