Tag Archives: EVC using pyVmomi

pyVmomi script: How to confirm whether EVC cluster is patched or not ? Spectre vulnerability

As per latest update from VMware, new vCenter and ESXi patches are released. These patches primarily will help enable “Hypervisor-Assisted Guest mitigation”. With these patches, one of the important aspects is to make sure whether entire EVC cluster is patched/upgraded or not. In this blog post, we will take a look at the pyVmomi script, which will confirm whether EVC cluster is patched with both hypervisor & microcode patch or not. Before we take a look at the script, let me quote statement from VMware KB on EVC cluster behavior.

“The vCenter patches enable vMotion compatibility to be retained within an EVC cluster.
In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated. At that time, the cluster will automatically upgrade its capabilities to expose the new features.”

As per above quote, new features should be available on EVC cluster once all the hosts inside the cluster are patched. Now lets take a look at the script.

This script i.e. is_evc_cluster_patched.py is available on my git-hub repo.


from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-c', '--cluster',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the cluster you wish to check')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args




# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj


args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
 print ("Cluster not found, please enter correct EVC cluster name")
 quit()

print ("Cluster Name:"+cluster.name)
evc_cluster_manager=cluster.EvcManager()


evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
 print ("Current EVC Mode::"+current_evcmode_key)
else:
 print ("EVC is NOT enabled on the cluster, please enable it first")
 quit()

feature_capabilities = evc_state.featureCapability

flag=False
for capability in feature_capabilities:


    if(capability.key in ["cpuid.STIBP", "cpuid.IBPB","cpuid.IBRS"] and capability.value=="1"):
        print ("Found::"+capability.key)
        flag=True

if(not flag):
    print ("No new cpubit found on EVC cluster,hence cluster is NOT fully patched/upgraded")
else:
    print ("EVC cluster is patched, enjoy!, this also confirms all the hosts inside this EVC cluster are patched as well")


atexit.register(Disconnect, si)


If you take a look at line 81-85, these are exact same EVC vSphere API properties that we discussed in my tutorial here. Same properties I have used to confirm whether EVC cluster is patched/upgraded or not. This script takes VCIP, username, password and EVC cluster name as parameter. Let us take a look at the output.

Output:

From output, it is clear that EVC cluster is patched successfully. It does mean that all the hosts inside the clusters are patched with both microcode and hypervisor patch. If at-least one of the hosts inside this EVC cluster is NOT patched, EVC cluster is NOT going to expose new cpuids on cluster and will be called as NOT patched.

Please note that since EVC APIs got introduced from vSphere 6.0, above script will be applicable for vSphere 6.0 and 6.5. Also, microcode/BIOS support is available for few Intel/AMD CPU models, please check out VMware KB for more details on it.

Some useful resources on EVC

1. If you want to confirm each host inside the cluster (with or without EVC) is patched or not, please take a look at my latest blog post
2. Empty EVC cluster issue discussed here is fixed with latest vCenter Patches
3. Part-1: Managing EVC using pyVmomi
4. Part 2: Managing EVC using pyVmomi
5. Tutorial on getting started pyVmomi

I hope you enjoyed this post, let me know if you have any questions/doubts.

Simple workaround: Adding a host to an empty EVC cluster fails on latest vCenter patch

This article is motivated by this VMTN thread posted by one of VMware users. I see that several users have reported this usability issue. In this post, I thought to share my understanding about this “empty EVC cluster” behavior along with simple workaround to fix this usability issue.

Update 03/20: Great news! Empty EVC cluster issue discussed in this post is fixed with latest vCenter Patches. With new vCenter patch, empty EVC cluster will NOT expose new cpuids by-default, rather it will expose cpu features based on the first host getting added into EVC cluster. For your learning, once you patch your vCenter server, I would recommend you try below script to understand the new empty EVC cluster behavior.

1. What is the issue?

Users are NOT able to add a host into an empty EVC cluster after applying latest vCenter patch, which was released as part of Hypervisor-Assisted Guest Mitigation on 9th Jan, 2018. In order to explore this further, I created a empty EVC cluster on vCenter 6.5 U1e as shown below.

You can see, I have created empty EVC cluster, where EVC mode configured is “Intel-IvyBridge”. Now I tried adding a host, where corresponding ESXi patch is NOT applied and as posted on VMTN thread, I see below error on web client.

 2. Why does it fail with this vCenter 6.5 U1e patch?

You might know that, as part of VMware patches released for Hypervisor-Assisted Guest Mitigation, there are three new cpuids got exposed i.e. cpuid.IBRS , cpuid.IBPB & cpuid.STIBP. Now we need to know whether these new cpuids are exposed by the “empty EVC cluster” that we just created but you might be wondering how to go about it? It is fairly simple to know it by using one of EVC API properties as posted in my tutorial post on EVC vSphere APIs. As part of section #4 of that tutorial i.e. “Exploring EVC Cluster state”, I had written a pyVmomi script to get the EVC cluster state i.e. getEVCClusterState.py. I little modified this script to know whether new cpuids are exposed by cluster or not. Below is the code snippet.

#Cluster object for empty EVC cluster
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
#EVCManager which exposes EVC APIs
evc_cluster_manager=cluster.EvcManager()

evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
        print "Current EVC Mode::"+current_evcmode_key
else:
        print "EVC is NOT enabled on the cluster"
        quit()

#getting feature capabilities available on cluster
feature_capabilities = evc_state.featureCapability

flag=False
for capability in feature_capabilities:

        if(capability.key=="cpuid.STIBP" and capability.value=="1"):
                print "Found new cpuid::"+ capability.key
                flag=True

        if(capability.key=="cpuid.IBPB" and capability.value=="1"):
                print "Found new cpuid::"+ capability.key
                flag=True

        if(capability.key=="cpuid.IBRS" and capability.value=="1"):
                print "Found new cpuid::"+ capability.key
                flag=True
if(not flag):
        print "No new cpubit found"

Once I executed above script on “empty EVC cluster” and I see output as follows

Output:
vmware@localhost:~$ python emptyEVCCluster.py
Current EVC Mode::intel-ivybridge
Found new cpuid::cpuid.IBPB
Found new cpuid::cpuid.IBRS
Found new cpuid::cpuid.STIBP
vmware@localhost:~$

Output clearly shows that new cpuids are exposed on the “empty EVC cluster” that we created on latest vCenter patch. This concludes that when you try to add a non-patched host into this cluster, it will fail since that host does NOT support these new cpuids. On the other hand, if you have applied corresponding ESXi patch (which is pulled back at the moment), then you will not face this issue as new cpuids are available on that patched host. Now that you understood this behavior, let us take a look at the simple workaround to fix this issue.

3. What is the workaround?

I understand, this is a problem from usability perspective but good news is that we have very simple workaround available i.e. Instead of upfront enabling EVC on the empty cluster, you can create non-EVC cluster first , add all possible hosts into it and finally enable EVC as per available EVC mode. If you ask me, it is always good idea to create non-EVC cluster and only after adding all possible hosts, we should finally enable EVC on cluster.

I would also recommend you to read “vMotion and EVC Information” section from VMware KB to understand EVC and vMotion behavior post applying these patches.

4. Some useful resources on EVC

1. Part-1: Managing EVC using pyVmomi
2. Part 2: Managing EVC using pyVmomi
3. Tutorial on getting started pyVmomi

I hope you enjoyed this post, let me know if you have any questions/doubts.

Tutorial: Part-2 : How to Manage EVC using vSphere Python SDK pyVmomi?

In part-1, we explored EVC enable, disable APIs & some important EVC properties using pyVmomi. In part-2, we would explore left out EVC APIs from part-1 i.e.CheckConfigureEvcMode_Task() & CheckAddHostEvc_Task(). In addition, we will also take a look at some more important EVC properties required to manage EVC end to end.

1. Checking whether EVC can be configured on existing cluster or not?
CheckConfigureEvcMode_Task() is responsible for checking exactly the same. If you have existing cluster with several hosts and if you would like to evaluate whether EVC can be enabled on cluster or not on particular EVCMode key, this API is really handy. If it is not possible to enable EVC on particular cluster, this API will return not only the set of reasons why it can fail to configure EVC but also the set of hosts those are causing it to fail. Before we start, below are the set of hosts present in cluster already.

vmware@localhost:~$ python getMaxEVCModeonHost.py
ClusterName::vThinkBVMCluster
10.160.20.30::intel-broadwell
10.160.20.31::intel-sandybridge
vmware@localhost:~$

To start with, the way we did in part-1, we will check whether EVC can be enabled on “intel-broadwell” EVCMode key. As we learned in last post, EVC should not be able to configure on this cluster with EVCMode “intel-broadwell”. Let us see whether API properly returns right reason and host that is causing it.

Below script is available on my git-hub repo as well

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import time

#Script to check whether EVC can be enabled on cluster with given EVCMode

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.160.50.60", user="Administrator@vsphere.local", pwd="VMware#12",sslContext=s)
content=si.content
cluster_name="vThinkBVMCluster";

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
        obj = None
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for c in container.view:
                if name:
                        if c.name == name:
                                obj = c
                                break
                        else:
                                obj = None                          
        return obj

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
evc_cluster_manager=cluster.EvcManager()

print "ClusterName::"+cluster.name

task=evc_cluster_manager.CheckConfigureEvcMode_Task("intel-broadwell")
time.sleep(5)
checkResult= task.info.result

if(checkResult):
         print "EVC can not be enabled on this cluster, please take look at below reasons and hosts causing this issue"
        for result in checkResult:
                print result.error.msg
                for h in result.host:
                        print h.name
                print "---------------------"
else:
        print "EVC can be successfully enabled on the cluster with no issues"

atexit.register(Disconnect, si)

vmware@localhost:~$ python checkEVC.py
ClusterName::vThinkBVMCluster
EVC can not be enabled on this cluster, please take look at below reasons and hosts causing this issue
The host’s CPU hardware does not support the cluster’s current Enhanced vMotion Compatibility mode. The host CPU lacks features required by that mode.
10.160.20.31
———————

isn’t this error matching with this screenshot from Part-1?

Note: If you notice on line #38, CheckConfigureEvcMode_Task() API returns ClusterEVCManagerCheckResult only when EVC can NOT be configured on the cluster.

Now I did powerON one of the VMs on host, which has max EVCMode as “intel-broadwell” and executed above script but this time passed EVCMode key as “intel-sandybridge”. Let us take a look at output.

Output:
vmware@localhost:~$ python checkEVC.py
ClusterName::vThinkBVMCluster
EVC can not be enabled on this cluster, please take look at below reasons and hosts causing this issue
The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.
10.160.20.30
———————
vmware@localhost:~$

You can see, EVC configuration failed since one of VM was powered ON, which is using some cpu features those are NOT available on “intel-sandybridge” EVC mode. This is expected and error is exactly matching with web client screenshot from Part-1.

Now I powered-off the VM residing on “intel-broadwell” based host & Powered-ON VM on host, which support max EVCMOde as “intel-sandybridge”. Finally executed same checkEVC.py script and below is the output.

Output:
vmware@localhost:~$ python checkEVC.py
ClusterName::vThinkBVMCluster
EVC can be succesfully enabled on the cluster with no issues
vmware@localhost:~$

wasn’t output cool? Note that VM was up on host with “intel-sandybridge” host and EVCMode used was also “intel-sandybridge”, hence EVC will be configured successfully. Now let us move on to next EVC API.

2. Checking whether host can be added into existing EVC enabled cluster or not?
CheckAddHostEvc_Task() API allows user to check exactly the same. This API accepts HostSpec as paramter for the host to be checked and below is the code snippet. I will leave it to you in order to try this API in your environment.

host_conn_spec=vim.host.ConnectSpec()
host_conn_spec.force=True
host_conn_spec.hostName=host.name
host_conn_spec.userName=host_user
host_conn_spec.password=host_pass
thumb_print=host.summary.config.sslThumbprint
host_conn_spec.sslThumbprint=thumb_print
print host_conn_spec
task=evc_cluster_manager.CheckAddHostEvc_Task(host_conn_spec)

That is all about EVC APIs. I hope Part-1 & Part-2 have given you good insight into how to play around EVC APIs. Stay tuned for my next post. Please comment if you have any query.