Category Archives: vSphere API

posts on vSphere API/SDK samples/scripts

vSphere 6.7: What is per-VM EVC and How to configure & manage it using pyVmomi?

Enhanced vMotion Compatibility (EVC) is without a doubt has been one of the famous vSphere features. As you know already,  EVC is a cluster level feature, which makes it possible to have vMotion across different generations of CPU within the cluster.  vSphere 6.7 has taken EVC to the next level.  This latest release has introduced one of cool features around EVC i.e. per-VM EVC.  Recently I got an opportunity to explore this cool feature and I thought to share my learning with you. In this blog post, I will take a through below items around per-VM EVC.

  • What is per-VM EVC?
  • Notes on per-VM EVC
  • per-VM EVC UI workflow
  • Configuring per-VM EVC using pyVmomi
  • Playing around featureMask using pyVmomi
  • Disabling per-VM EVC using pyVmomi
What is per-VM EVC?

As name indicates, per -VM EVC can be enabled on individual VM. Great thing about per VM EVC is that, it not only works on VMs inside the cluster but also VMs outside of the cluster.  Unlike cluster level EVC, this feature enables vMotion across clusters, standalone hosts, cross vCenters & hybrid clouds such as VMware cloud on AWS (from on-prem vCenter). How cool is that?

Notes on per-VM EVC
  • User needs to have vSphere 6.7
  • VM hardware version must be vmx-14
  • User must powered off the VM before configuring per-VM EVC
  • If user is enabling per -VM EVC on a VM, which is inside EVC enabled cluster, EVC mode on the VM should be equal or lower than that of EVC mode configured on cluster.
  • If  user wants to enable this feature from UI, only vSphere H5 client  supports it and not the flex based client.
  • per-VM EVC works fine with vSphere DRS
  • If user clones per-VM EVC configured VM, it will retain per-VM EVC configuration on cloned VM.
  • If underlying host does not support EVC mode configured on VM, VM can not power ON
  • User can enable per-VM EVC on a VM though VM is already part of a EVC enabled cluster.
per-VM EVC UI workflow

If you ask me, it is pretty easy to configure per-VM EVC from vSphere H5 client (flex client does not support). User just needs to click on the VM >> Configure >> VMware EVC >>Edit >> Configure EVC mode of your choice (of course EVC mode supported by underlying host/Cluster). Please take a look at below H5 client screenshots.

Per VM EVC UI configuration workflow

I configured “Intel ivy-bridge” EVC mode and below is how it looks like post configuration.

per VM EVC UI post configuration
Configuring per-VM EVC using pyVmomi

vSphere 6.7 has exposed a vSphere API to configure per-VM EVC i.e. ApplyEvcModeVM_Task(). We need to pass right “featureMask” to this API in order to configure appropriate EVC mode on the VM. Note that every EVC mode defines its own set of featureMask. Ex. intel-sandybridge EVC mode will have corresponding set of featureMask, so is for intel-ivybridge and so on..

Since we need to pass “featureMask” for particular EVC mode, first important thing user needs to get hold of is right EVC mode. Once we get hold of right EVC mode, we need to get corresponding  “featureMask”.  In our case, lets configure per-VM EVC on a VM residing  on a standalone host.  As I said, first we need to find max EVC mode supported on the host. This can be easily found using the script I discussed in my blog post here (Refer section on “Getting max EVCMode key“). It can be quickly found from vSphere web client or H5 client host summary as well. In my case, my host “max EVCMode”was “intel-haswell”. It does mean that I can enable per-VM EVC with featureMask for either “intel-haswell” EVC mode or lower EVCMode such as “intel-ivybridge” , “intel-sandybridge” etc.  I chose to enable per-VM EVC on “intel-ivybridge” EVC mode.

Now that we have finalized EVC mode i.e. intel-ivybridge, we now need to get hold of corresponding “featureMask”. Please take a look at below code snippet.

si= SmartConnect(host=args.host, user=args.user, pwd=args.password, sslContext=s)
supported_evc_mode=si.capability.supportedEVCMode
for evc_mode in supported_evc_mode:
	if(evc_mode.key == "intel-ivybridge"):
		ivy_mask=evc_mode.featureMask
		break

If you take a look above snippet, it is fairly easy to get featuremask for particular EVC Mode.

putting it together

This script is available on my github-repo here

 
# Author: Vikas Shitole
# Website: www.vThinkBeyondVM.com
# Product: vCenter server/ per-VM EVC (Enhanced Compatibility Mode)
# Description: Script to get enbale/disable per-VM EVC on VM
# Reference: 
# How to setup pyVmomi environment?:
# On linux:  http://vthinkbeyondvm.com/how-did-i-get-started-with-the-vsphere-python-sdk-pyvmomi-on-ubuntu-distro/
# On windows: http://vthinkbeyondvm.com/getting-started-with-pyvmomi-on-windows-supports-vsphere-6-7/


from pyVim.connect import SmartConnect
import ssl
from pyVmomi import vim
import atexit
import sys
import argparse
import getpass
 
 
def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-v', '--vmname',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the VM to be configured per VM EVC')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args
 
 # Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj
 
 
args = get_args() 
s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
 
si= SmartConnect(host=args.host, user=args.user, pwd=args.password, sslContext=s)
content=si.content
vm= get_obj(content, [vim.VirtualMachine],args.vmname)

if(vm and vm.capability.perVmEvcSupported):
        print ("VM available in vCenter server and it supports perVm EVC, thats good")
else:
        print ("VM either NOT found or perVMEvc is NOT supported on the VM")
        quit()

supported_evc_mode=si.capability.supportedEVCMode
for evc_mode in supported_evc_mode:
	if(evc_mode.key == "intel-ivybridge"):
		ivy_mask=evc_mode.featureMask
		break
	 
vm.ApplyEvcModeVM_Task(ivy_mask,True)
print ("ApplyEvcModeVM_Task() API is invoked, check out your H5 client")

Line #80: This confirms whether VM found or not and it also confirms whether VM supports per-VM EVC or not.
#86-90: It is about getting right featureMask corresponding to particular EVC Mode.
#92: Finally we called the API “ApplyEvcModeVM_Task()”

How to run this script
“python perVMEVCNew.py -s 10.161.81.159 -u Administrator@vsphere.local -v DRSPerVMEVC”, user needs to enter vCenter password. Below is how it looks like.

I looked into vSphere H5 client and this is how it looks like. isn’t it cool?

Playing with featureMask

In line #86, we learned how to get hold of featureMask. is it the only way we can get featureMask and pass it to this API? Answer is NO. There are multiple ways. However, I recommend to follow the way I did on line #86 above.
Sometime you may want to configure per-VM EVC with the same featureMask as that of EVC enabled cluster.  This can be handy specially for the VMs those are outside of the EVC cluster. In such case, you can simply copy the featureMask from EVC cluster and pass it to the per-VM EVC API. Let us take look at below code snippet.

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource], args.cluster)

if(cluster):
        print ("Cluster available in vCenter server, thats good")
else:
        print ("Cluster is NOT available in vCenter server, please enter correct name")
        quit()
		
evc_cluster_manager=cluster.EvcManager()

evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
        print ("Current EVC Mode::"+current_evcmode_key)
else:
        print ("EVC is NOT enabled on the cluster")
        quit()
features_masked = evc_state.featureMask

Above EVC API property I had already discussed in my post on cluster level EVC APIs here (Refer: section #4 : Exploring EVC Cluster state). Here is the complete script, where featureMask is copied from EVC cluster. In addition, you can also choose to copy the featureMask from already per-VM EVC configured VM but if you do not document it properly, it can lead to confusion in future, hence it is always better to follow one way consistently to avoid any issues in future.

Disabling per-VM EVC

If you ask me, it is pretty easy. We just need to invoke the same API but without any featureMask as follows


vm.ApplyEvcModeVM_Task(None,True)

Some useful resources on EVC

1. Part-1: Managing EVC using pyVmomi
2. Part 2: Managing EVC using pyVmomi
3. Tutorial on getting started pyVmomi  on linux
4. Tutorial on getting started pyVmomi on Windows

I hope you enjoyed this post, let me know if you have any questions/doubts.

Tutorial: Getting started with pyVmomi on Windows : Supports vSphere 6.7

Couple of months back, I wrote an article on “How did I get started with pyVmomi on ubuntu distro?”, which was very well received.  Recently I got several requests from readers  on how to set pyVmomi on Windows platform and I thought to write this tutorial, which will take you through steps required to set pyVmomi on my Windows 10.

Step 1: Install latest python for windows.

Latest 3.x python version available is 3.6.5 on python official portal . Download the python installer as pointed out in below screenshot.

Once downloaded, while installing python, make sure to select checkbox “Add python 3.6 to PATH” as shown below.

As soon as installation is completed, I checked environment variable and I see that python 3.6 path is set properly as shown below.

I see that python is installed as expected at directory i.e. C:\Users\shitolev\AppData\Local\Programs\Python\Python36, which is default directory. I also checked python version using “python -V” command and I see correct python version is installed.

Step 2: Install pyvmomi and pyvim python modules

As I already discussed in my pyVmomi on ubuntu article, we need “pip” to install these python modules. Note that “pip” itself gets installed as part of python installation. Let us go ahead for “pyvmomi” and pyvim” modules installation by running this command
“python.exe -m pip install pyvmomi pyvim” . You can run this command from any directory of your choice.

Below is how it looks like

you can see that latest “pyvmomi” i.e. the one supports vSphere 6.7  and “pyvim” python modules got installed successfully. I also see that these modules are installed at python  directory i.e. C:\Users\shitolev\AppData\Local\Programs\Python\Python36\Lib\site-packages

Now that we are done with setting up pyVmomi on Windows, its time to run a vSphere API sample to confirm everything is working fine.

Step 3: Running vSphere API sample.

For the sake of simplicity I chose the same sample I used in my ubuntu article

# VMware vSphere Python SDK, pyvmomi
# Simple script to get vCenter server product details

from pyVim.connect import SmartConnect
import ssl

s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE

si= SmartConnect(host="10.192.1.2", user="Administrator@vsphere.local", pwd="VMware#123", sslContext=s)
aboutInfo=si.content.about

print ("Product Name:",aboutInfo.fullName)
print("Product Build:",aboutInfo.build)
print("Product Unique Id:",aboutInfo.instanceUuid)
print("Product Version:",aboutInfo.version)
print("Product Base OS:",aboutInfo.osType)
print("Product vendor:",aboutInfo.vendor)

We can keep this sample file at any location of our choice and execute it as follows.

Output shows that this small pyvmomi script got executed on vSphere 6.7 successfully. How cool is that?

Note: I would like you to take a look at line #7, where I had to use “ssl.PROTOCOL_SSLv23” protocol for VC 6.7 connection. For VC 6.0 and 6.5, I used “ssl.PROTOCOL_TLSv1”.

Useful resources

1. Please take a look at tutorial on how to get hold of vSphere objects and play around. Take a look at some cool samples on VC HA

2. If you want to contribute to pyVmomi, refer pyvmomi and its samples github repo

3. Not interested in Python vSphere SDK? no problem, take a look at vSphere Java SDK and PowerCLI posts.

4. Getting started with vCenter server REST APIs using python

Tutorial: Part-2: How to simulate Proactive HA using pyVmomi?

In part-1, we learned “What is Proactive HA and how to configure Proactive HA using pyVmomi?”. To understand part-2, I highly recommend you to take a look at part-1. In this post, we will learn how to simulate a failure to see Proactive HA in action.

Before going further, let us review what we configured already in last post.
– We registered a health provider.
– Added entities to be monitored i.e. hosts inside cluster
– Verified above 2 steps.
– Enabled Proactive HA on cluster in automated mode and configured remediation.

After performing above steps, you might have observed that all of the hosts are showing one red icon. This red icon indicates that initial health status for hosts is unknown i.e. gray, which is absolutely expected since Proactive HA provider must initialize health status of all ESXi hosts for the first time. In order to initialize health status of all ESXi, first we need to set status to “green”. This is where Proactive HA API i.e. PostHealthUpdates() comes to rescue. In fact, the same API will help us simulate a “red” health status, which will make Proactive HA to take configured action. Let us take a look at below pyVmomi script, which will initialize ESXi hosts to “green” health status.

This script is available on my git-hub repo as well i.e. push_health_updates.py

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import time
#Script to push meaningful health updates to Proactive HA 
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.161.20.30", user="Administrator@vsphere.local", pwd="VMware#123",sslContext=s)
content=si.content

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
        obj = None
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for c in container.view:
                if name:
                        if c.name == name:
                                obj = c
                                break

        return obj

#Cluster where ProactiveHA is enabled, this parameter is NOT required for this script
cluster_name="ClusterProHA"
# initialize health status of below host, we need to repeat for all the hosts inside the cluster. 
host_name="10.160.20.15" 
host = get_obj(content,[vim.HostSystem],host_name)
# Managed object that exposes all the methods to configure & manage proactiveHA
heath_update_mgr=content.healthUpdateManager

#Provider ID for the registered provider
providerid="52 b5 17 d2 2f 46 7e 9b-5f 4e 1a 25 a3 db 49 85"
health_update=vim.HealthUpdate()
health_update.entity=host
health_update.healthUpdateInfoId="1000"
health_update.id="vThinkBeyondVM"
health_update.remediation =""
health_update.status="green"
#Below is the array of health updates
updates = [health_update]

#Method to post health updates for Proactive HA to consume
heath_update_mgr.PostHealthUpdates(providerid,updates)

#Disconnecting vCenter session
atexit.register(Disconnect, si)

We must understand line #32 to line #45.
Line #34: This is provider id for the health provider we registered in part 1 using QueryProviderName() API
Line #35: HealthUpdate is the data object to be passed to the Proactive HA API we are talking about i.e. PostHealthUpdates()
Line #36: This is the host object for which we wanted to set status to “green”. For the sake of simplicity, this script changes health status per host. You can either modify this script to have for all the hosts or can run it multiple times.
Line #37: This is the healthUpdateInfoId we passed while registering health provider i.e. register_provider.py
Line #38: This id can be any of our choice but it should be unique per health update being pushed.
Line #39: Note that remediation set here was blank. This is required when we are setting health updates as “green”.
Line #40: Here we set the health status as “green”. Other valid values are “yellow” and “red”.
Line #42: We can have multiple health updates in single API call.
Line #45: This is the API we are interested to call i.e. PostHealthUpdates().

Since we have initialized health status for all ESXi hosts to “green”, now is the time to simulate a fake health update so that Proactive HA will trigger “quarantine mode” on a host. We can use exactly the same script with little change to line #38,39 & 40. Note that simulating fake health update is only for educational purpose . Take a look at the code snippet for pushing “red”(fake) health status a host.

#Provider ID for the registered provider
providerid="52 b5 17 d2 2f 46 7e 9b-5f 4e 1a 25 a3 db 49 85"
health_update=vim.HealthUpdate()
health_update.entity=host
health_update.healthUpdateInfoId="1000"
health_update.id="vThink-BVM"
health_update.remediation ="Please replace power unit"
health_update.status="red"
#Below is the array of health updates
updates = [health_update]

#Method to post health updates for Proactive HA to consume
heath_update_mgr.PostHealthUpdates(providerid,updates)

Once I executed above script, I refreshed web client after 3-4 seconds and below is what I see. How cool is that?

Now you can set the host health status back to “green” the way did above, which will make host to exit from quarantine mode. Finally you can leverage my Part-1 scripts to disable Proactive HA on the cluster and un-register the fake health provider we registered initially. In addition to APIs we explored in part-1 & 2, there are few more utility APIs Proactive HA has exposed. I will leave them for your exercise. Let me know if you have any doubts/comments.

Further learning:

1. If you haven’t started with vSphere python SDK i.e. pyVmomi. Here is my tutorial post on the same.

2.For the sake of simplicity, I use to write individual API scripts & hard-code some parameters but I have a plan to consolidate above scripts to have good Proactive HA pyVmomi module. If you would like to contribute to module, please let me know, we can work together.

pyVmomi script: How to confirm ESXi hypervisor & microcode patches are applied?: Spectre vulnerability

Some time back, VMware has updated one of KBs on “Hypervisor-Assisted Guest Mitigation for branch target injection”. Please take a look at this KB here. In this KB, there is one important section on “how to confirm a host has both patched microcode and patched VMware hypervisor?” In fact, there are multiple ways to confirm it and one of the ways is specified in the KB itself i.e. scanning VM vmware.log file for new cpuids. Last time, I had automated exactly the same way using PowerCLI. If you prefer PowerCLI, you can use mine or you can also refer nice PowerCLI script written by William for the same. If there are PowerCLI scripts available for confirming these patches, you may wonder, what is that I am going to write in this post? Since there is a very handy host level vSphere API property available to confirm these patches, I thought to write a quick pyVmomi script. This will be very handy for people who prefer pyVmomi. Also as it uses native vSphere APIs, it will be faster than its PowerCLI equivalent.

This script i.e. hosts_patched.py available on my git repo as well.

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-c', '--cluster',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the cluster you wish to check')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args


# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj



args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
 print ("Cluster not found, please enter correct EVC cluster name")
 quit()

print ("Cluster Name:"+cluster.name)

# Get all the hosts available inside cluster
hosts = cluster.host

#Iterate through each host to get MaxEVC mode supported on the host
for host in hosts:
 print ("----------------------------------")
 print ("Host:"+host.name)
 feature_capabilities = host.config.featureCapability
 flag=False
 for capability in feature_capabilities:
  if(capability.key in ["cpuid.STIBP", "cpuid.IBPB","cpuid.IBRS"] and capability.value=="1"):
   print ("Found::"+capability.key)
   flag=True
 if(not flag):
  print ("No new  cpubit found, hence "+host.name+" is NOT patched")
 else:
  print ("New CPU bit is found, hence "+host.name+" is patched")
	

atexit.register(Disconnect, si)


This script takes VCIP, username, password and cluster-name (with or without EVC) as parameter. Let us take a look at below output.

Output:
vmware@localhost:~$ python hosts_patched.py -s 10.192.30.40 -u Administrator@vsphere.local -c EVCCluster
Enter vCenter password:
Cluster Name:EVCCluster
———————————-
Host: 10.20.43.35
Found ::cpuid.IBPB
Found ::cpuid.IBRS
Found ::cpuid.STIBP
New CPU bit is found, hence 10.20.43.35 is patched
———————————-
Host:10.20.43.36
No new cpubit found, hence 10.20.43.36 is NOT patched
vmware@localhost:~$

Based on above script, it is clear that there are 2 hosts inside the cluster and only one of them has exposed new cpuids, that confirms it is patched with both ESXi hypervisor & microcode.

Further learning:

1. I highly recommend you take a look at last post on EVC cluster.
2. Empty EVC cluster issue discussed here is fixed with latest vCenter Patches
3. My PowerCLI script to achieve the same
4. Tutorial on getting started pyVmomi

I hope you enjoyed this post, let me know if you have any questions/doubts.