Tag Archives: vsphere

PowerCLI script: How to hide the speculative-execution control mechanism for VMs running on affected processors ?

Here is the latest KB update from VMware on Hypervisor-Assisted Guest Mitigation for branch target injection. I would highly recommend you to read this latest KB to understand all minute details. As per KB, it is recommended NOT to apply recently released ESXi patches i.e. ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG on servers based out of affected processors. However, if you have already applied aforementioned patches on affected servers/hosts, VMware recommends to do following in order to hide the speculative-execution control mechanism for virtual machines(quoting from KB)

Update 01/23: As per latest update on KB, Intel Sightings just not only applicable to broadwell & haswell CPUs but also some additional CPUs (Please take a look at Table 1 in KB). Accordingly you can easily edit the script to remove the check for broadwell & haswell CPU servers.

Quote started

For servers using affected Intel processors (see Table 1.) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following::

On each affected ESXi host, add the following line in the /etc/vmware/config file:
cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-”
This will hide the speculative-execution control mechanism for virtual machines which are power-cycled afterwards on the ESXi host.
This line will need to be removed after applying a future fixed microcode from Intel in order to enable the full guest OS mitigations for CVE-2017-5715.
When convenient, power-cycle virtual machines on the affected ESXi hosts; rebooting of the ESXi host is not required.

Quote ended

The way I wrote a PowerCLI script for confirming patch validation, I thought it would be really handy to have another quick PowerCLI script to hide the speculative-execution control mechanism for virtual machines as suggested by VMware in KB.

What this script does?

1. Get all the connected hosts from specified vCenter server datacenter.
2. Start iterating through each ESXi host.
3. Check whether ESXi host max supported EVC mode is “intel-haswell” or “intel-broadwell”. If yes, script will then check whether new cpuids are really exposed to these hosts.
4. If new cpuids are exposed on haswell or broadwell hosts, it picks that host for adding “cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” into “/etc/vmware/config” file.
5. Before updating above file, script does check whether SSH is enabled on picked (haswell or broadwell) host, if SSH is not enabled, it does enable it.
6. If SSH on picked host is already enabled, it will create SSH session & takes the backup of the file /etc/vmware/config, which gets named as per time stamp.
7. Once file is backed up, it will go ahead and add “cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” line at the end of file “/etc/vmware/config”
8. Once file edit operation is done, script does disable SSH on the host where it was not enabled before editing. It also confirms whether SSH is really disabled.
9. Above steps will be repeated only for intel-haswell or intel-broadwell based hosts, where new cpuids are exposed.
10. It keeps logging (on console) the hosts which are edited/not-edited
11. Finally it disconnects vCenter server session.

This script is available on my on my git repo: hideNewCpuids.ps1

After executing above script, it is critical to do VM power-cycle (PowerOFF the VM & PowerON) to get this change affected. Once you do powerCycle, you may wonder how to verify whether speculative-execution control mechanism for virtual machines is actually got hidden or not. When we say “hide the speculative-execution control mechanism for virtual machines”, as earlier blog post pointed, it exposes new CPUIDs to the VM i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP. So it is meant that after script execution, once user does VM power-cycle, these CPUIDs should NO more be exposed to VMs.

How to quickly verify these cpuids are hidden from VM? It can easily done using vCenter MOB (Managed object browser) as follows:

1. To check whether host is having new cpuid capability: please use vCenter MOB (https:// your VC IP/mob): https://10.192.10.20/mob/?moid=host-10&doPath=config (replace your VC IP , host moid and check for “featureCapability” property, you will get list of new cpuids this host is capable of.

Note: Patched ESXi hosts continue to show new CPUIDs (value as 1) even after editing /etc/vmware/config file. This is expected since update to file /etc/vmware/config just hides speculative-execution control mechanism for virtual machines.

2. To check whether new cpuids are hidden from VM:
https://10.192.10.20/mob/?moid=vm-19&doPath=runtime (replace your VC IP , vm moid and check for “featureRequirement” property.

Result:
Before editing file /etc/vmware/config: “”featureRequirement” property of VM shows new CPUIDs i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP

After editing file /etc/vmware/config : “”featureRequirement” property of VM does NOT show new CPUIDs (PowerCycle needed). i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP

How to restore the host to old state? i.e. before editing /etc/vmware/config file
– Before doing this, do make sure that you completely understand the impact it can have on your environment. KB #52345 gives more details.
– To restore, you can either remove added line from /etc/vmware/config or you can comment out using “#” as prefix. To verify this change, you will have to do powercycle. No host reboot required as specified in KB post file update.

Notes:
1. For the sake of simplicity I have hardcoded some values, please do change as per your environment.
2. If you have any comment/feedback on above script, please do provide.
3. I have tested this script on vCenter/ESXi 6.0 & it worked as expected for me.

Additional references:
1. If you are new to MOB, refer official doc

2. I would like to acknowledge this post on Posh-SSH module. I leveraged SSH part of it.

3. If you want to verify above change using PowerCLI, you can leverage nice script written by our own William Lam here.

4. I may write a python script using pyVmomi for verification similar to what we did using MOB above.

I hope this post was useful, please share as appropriate.

pyVmomi tutorial : How to get all the core vCenter server inventory objects and play around?

Recently I posted about “how did I get started with pyvmomi?”. I continued exploring this further and I thought to share my learning so far. After setting up the pyvmomi environment, first thing I wanted to learn was to get hold of all core vCenter inventory objects so that I can play around each of them. vCenter core inventory includes objects such as Datacenters, Clusters, Virtual Machines, Hosts, Datastores & Networks etc.
As an initial step, I was looking for a way to get below objects from VC inventory
-All Virtual Machines
-All Clusters
-All Hosts
-All Datacenters

There could be multiple ways to get above objects but below is the way I could achieve this.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

# Get all the Vms from vCenter server inventory and print its name
# Below is Python 2.7.x code, which can be easily converted to python 3.x version

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.192.3.2", user="Administrator@vsphere.local", pwd="$h1vKamal",sslContext=s)
content=si.content

# Method that populates objects of type vimtype
def get_all_objs(content, vimtype):
        obj = {}
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for managed_object_ref in container.view:
                obj.update({managed_object_ref: managed_object_ref.name})
        return obj

#Calling above method
getAllVms=get_all_objs(content, [vim.VirtualMachine])

#Iterating each vm object and printing its name
for vm in getAllVms:
        print vm.name

Output:
vmware@localhost:~$ python getAllVms.py
Database-server
NTP-Bangalore
Web-server
NTP-Pune
vmware@localhost:~$

You could see above, script listed all the VMs available in inventory. Let us look at another example, here we will get all the clusters from vCenter Inventory.

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl

#Get all the Clusters from vCenter invetory and printing its name
#Below is Python 2.7.x code, which can be easily converted to python 3.x version

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.192.3.2", user="Administrator@vsphere.local", pwd="$h1vKamal",sslContext=s)
content=c.content

# Method that populates objects of type vimtype
def get_all_objs(content, vimtype):
        obj = {}
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for managed_object_ref in container.view:
                obj.update({managed_object_ref: managed_object_ref.name})
        return obj

#Calling above method
clusters=get_all_objs(content, [vim.ClusterComputeResource])

#Iterating each cluster object and printing its name
for cluster in clusters:
        print (cluster.name)

Output:
vmware@localhost:~$ python getClusters.py
DRSCluster-USA
DRSCluster-Europe
DRSCluster-India
vmware@localhost:~$

If you notice, only important change from getAllVms and getClusters script is line #22, is not it cool? So now we were able to get all the VMs as well as all the clusters. Similarly we can get other core inventory objects such as datacenter, hosts, datastores, networks etc. Below is how line #22 will look for other vCenter/ESXi objects.

# For ESXi host
hosts=get_all_objs(content, [vim.HostSystem])

#For datacenters
dcs = get_all_objs(content, [vim.Datacenter])

#For datastores 
datastores = get_all_objs(content, [vim.Datastore])

By now we know how to get hold of vSphere inventory objects, let’s do something meaningful with these objects. I chose to call a method on one of clusters that we retrieved above. What is that method?
vCenter 6.0 has introduced one of interesting APIs on vSphere DRS cluster rules. i.e. findRulesForVM(). This method is called on individual cluster and it returns set of DRS VM-VM affinity rules (enabled or disabled) those are associated with given/input VM. Since this method returns VM-VM affinity rules, I first created some VM-VM rules on one of my clusters as follows.

Cluster with DRS enabled
DRS VM-VM rules

Now that DRS rules are created, let us pass VM name as parameter to “findRulesForVM()” API and call this API on that cluster.

Updated code is available on VMware sample exchange portal here

from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
#Get DRS VM-VM rules associated with a VM.

s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host="10.192.3.2", user="Administrator@vsphere.local", pwd="$vabc325pqr",sslContext=s)
content=c.content

def get_all_objs(content, vimtype):
        obj = {}
        container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
        for managed_object_ref in container.view:
                obj.update({managed_object_ref: managed_object_ref.name})
        return obj

# Scanning a input VM inside inventory using special  python construct i.e. List comprehension
# It will get all the Vms and check whether input VM is available inside inventory or not, finally it returns list with matching condition

vmToScan = [vm for vm in get_all_objs(content,[vim.VirtualMachine]) if "NTP-Bangalore" == vm.name]

# Scanning a input cluster inside invetory the way we did for VM above. here also we used list comprehension.

cluster = [cluster for cluster in get_all_objs(content,[vim.ClusterComputeResource]) if "DRSCluster-India" == cluster.name]

# Now we can call the method on input cluster by passing input VM as parameter, it returns array of rule objects associated with input VM.
ClusterRuleInfo=cluster[0].FindRulesForVm(vmToScan[0])

# Now iterate through rule objects and print the rule name

for rule in ClusterRuleInfo:
        print rule.name

Output:
vmware@localhost:~$ python findRules.py
Anti-Affine-NTP-Servers
Affine-Web-DB-Servers

Here is how vSphere web client “recent task” looks, you could see “Find Rules for vms” task.

Notes:
-For the sake of simplicity, I have hard-coded some values and simplified the code, please make changes as per your requirement.
-I have added required documentation inside above code itself.
-This code follows mostly python 2.7.x version, please make minor changes to work with python 3.x

If you need any help, please let me know and yes, please stay tuned for my upcoming blog posts on pyVmomi.