Category Archives: vCenter Server

Posts on vCenter server

pyVmomi script to confirm Speculative Store Bypass Disable (SSBD) mitigation on vSphere patches

Few hours back, VMware released vSphere patches to mitigate “Speculative Store Bypass Disable (SSBD)” security issue. Please take a look at this KB for more details. In this post, as I did in the past, I am going to provide you a pyVmomi script to confirm whether vCenter server, ESXi hypervisor and microcode patches are applied or not to mitigate this critical security issue. Before we look into script, one of the important points you should note that, these latest vSphere (both vCenter server and ESXi) patches  are cumulative &  if you haven’t applied earlier spectre vulnerability patches [released as on 20th March] yet, you can directly apply these patches to get earlier fixes as well.

pyVmomi script to confirm SSBD mitigation

Notes:

  • This script works for all vSphere releases i.e. 5.5, 6.0, 6.5, 6.7.
  • This script i.e. confirm_ssbd_patch.py is available on my github repo as well.
  • Since this patch is cumulative, focus in this script is only SSBD cpubit
  • This script takes VCIP, username, password and cluster-name (with or without EVC) as parameter.
  • Please take a note of line #72 on SSL/TLS protocol
  • As specified in the KB, you need to perform VM power-cycle post patch application.
# Author: Vikas Shitole
# Product: vCenter server
# Description: Script to confirm whether vCenter server, hypervisor and microcode patches are applied or not : vCenter/ESXi patches for Speculative Store Bypass Disable vulnerability.
# Reference: https://kb.vmware.com/s/article/55111
# How to setup pyVmomi environment?: 
# Linux: http://vthinkbeyondvm.com/how-did-i-get-started-with-the-vsphere-python-sdk-pyvmomi-on-ubuntu-distro/
#Windows: http://vthinkbeyondvm.com/getting-started-with-pyvmomi-on-windows-supports-vsphere-6-7/

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-c', '--cluster',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the cluster you wish to check')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args


# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj

args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

print ("-------------------------------------")
#Check whether vCenter server is patched or not
supported_evc_mode=si.capability.supportedEVCMode
# It is not required to check "ivy-bridge" EVC mode, you can choose any EVC mode from "intel-penryn" onwords.
for evc_mode in supported_evc_mode:
    if(evc_mode.key == "intel-ivybridge"):
        ivy_masks=evc_mode.featureMask
        break

vCenter_patched=False
for capability in ivy_masks:
  if(capability.key in ["cpuid.SSBD"] and capability.value=="Val:1"):
   print ("Found::"+capability.key)
   vCenter_patched=True
if(not vCenter_patched):
  print ("No new cpubit found, hence vCenter server is NOT patched")
else:
  print ("New CPU bit is found, hence vCenter Server is patched")
print ("Current vCenter server build::"+si.content.about.fullName)

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
 print ("Cluster not found, please enter correct EVC cluster name")
 quit()

print ("Cluster Name:"+cluster.name)

# Get all the hosts available inside cluster
hosts = cluster.host

#Iterate through each host to get MaxEVC mode supported on the host
for host in hosts:
 print ("----------------------------------")
 print ("Host:"+host.name)
 feature_capabilities = host.config.featureCapability
 flag=False
 for capability in feature_capabilities:
  if(capability.key in ["cpuid.SSBD"] and capability.value=="1"):
   print ("Found::"+capability.key)
   flag=True
 if(not flag):
  print ("No new cpubit found, hence "+host.name+" is NOT patched")
 else:
  print ("New CPU bit is found, hence "+host.name+" is patched")
	
atexit.register(Disconnect, si)

Let us take a look at below output.

Output

C:\Professional\vThinkBeyondVM\Spectre posts>python hosts_patched_ssbd.py -s 10.20.30.35 -u Administrator@vsphere.local -c “New Cluster”
Enter vCenter password:
————————————-
Found::cpuid.SSBD
New CPU bit is found, hence vCenter Server is patched
Current vCenter server build::VMware vCenter Server 6.7.0 build-8833179
Cluster Name:New Cluster
———————————-
Host: 10.20.30.51
No new cpubit found, hence 10.20.30.51 is NOT patched
———————————-
Host: 10.20.30.52
Found::cpuid.SSBD
New CPU bit is found, hence 10.20.30.51 is patched

Above output shows that vCenter server is patched and one of the two ESXi hosts is patched successfully.

Further learning resources
  1. per-VM EVC tutorial
  2.  Part-1: Managing Cluster level EVC using pyVmomi
  3. Part 2: Managing Cluster level EVC using pyVmomi
  4. Tutorial on getting started pyVmomi  on linux
  5. Tutorial on getting started pyVmomi on Windows

I hope you will find this post useful, please stay tuned for my next blog post on per-VM EVC wrt to these mitigation patches.

vSphere 6.7: What is per-VM EVC and How to configure & manage it using pyVmomi?

Enhanced vMotion Compatibility (EVC) is without a doubt has been one of the famous vSphere features. As you know already,  EVC is a cluster level feature, which makes it possible to have vMotion across different generations of CPU within the cluster.  vSphere 6.7 has taken EVC to the next level.  This latest release has introduced one of cool features around EVC i.e. per-VM EVC.  Recently I got an opportunity to explore this cool feature and I thought to share my learning with you. In this blog post, I will take a through below items around per-VM EVC.

  • What is per-VM EVC?
  • Notes on per-VM EVC
  • per-VM EVC UI workflow
  • Configuring per-VM EVC using pyVmomi
  • Playing around featureMask using pyVmomi
  • Disabling per-VM EVC using pyVmomi
What is per-VM EVC?

As name indicates, per -VM EVC can be enabled on individual VM. Great thing about per VM EVC is that, it not only works on VMs inside the cluster but also VMs outside of the cluster.  Unlike cluster level EVC, this feature enables vMotion across clusters, standalone hosts, cross vCenters & hybrid clouds such as VMware cloud on AWS (from on-prem vCenter). How cool is that?

Notes on per-VM EVC
  • User needs to have vSphere 6.7
  • VM hardware version must be vmx-14
  • User must powered off the VM before configuring per-VM EVC
  • If user is enabling per -VM EVC on a VM, which is inside EVC enabled cluster, EVC mode on the VM should be equal or lower than that of EVC mode configured on cluster.
  • If  user wants to enable this feature from UI, only vSphere H5 client  supports it and not the flex based client.
  • per-VM EVC works fine with vSphere DRS
  • If user clones per-VM EVC configured VM, it will retain per-VM EVC configuration on cloned VM.
  • If underlying host does not support EVC mode configured on VM, VM can not power ON
  • User can enable per-VM EVC on a VM though VM is already part of a EVC enabled cluster.
per-VM EVC UI workflow

If you ask me, it is pretty easy to configure per-VM EVC from vSphere H5 client (flex client does not support). User just needs to click on the VM >> Configure >> VMware EVC >>Edit >> Configure EVC mode of your choice (of course EVC mode supported by underlying host/Cluster). Please take a look at below H5 client screenshots.

Per VM EVC UI configuration workflow

I configured “Intel ivy-bridge” EVC mode and below is how it looks like post configuration.

per VM EVC UI post configuration
Configuring per-VM EVC using pyVmomi

vSphere 6.7 has exposed a vSphere API to configure per-VM EVC i.e. ApplyEvcModeVM_Task(). We need to pass right “featureMask” to this API in order to configure appropriate EVC mode on the VM. Note that every EVC mode defines its own set of featureMask. Ex. intel-sandybridge EVC mode will have corresponding set of featureMask, so is for intel-ivybridge and so on..

Since we need to pass “featureMask” for particular EVC mode, first important thing user needs to get hold of is right EVC mode. Once we get hold of right EVC mode, we need to get corresponding  “featureMask”.  In our case, lets configure per-VM EVC on a VM residing  on a standalone host.  As I said, first we need to find max EVC mode supported on the host. This can be easily found using the script I discussed in my blog post here (Refer section on “Getting max EVCMode key“). It can be quickly found from vSphere web client or H5 client host summary as well. In my case, my host “max EVCMode”was “intel-haswell”. It does mean that I can enable per-VM EVC with featureMask for either “intel-haswell” EVC mode or lower EVCMode such as “intel-ivybridge” , “intel-sandybridge” etc.  I chose to enable per-VM EVC on “intel-ivybridge” EVC mode.

Now that we have finalized EVC mode i.e. intel-ivybridge, we now need to get hold of corresponding “featureMask”. Please take a look at below code snippet.

si= SmartConnect(host=args.host, user=args.user, pwd=args.password, sslContext=s)
supported_evc_mode=si.capability.supportedEVCMode
for evc_mode in supported_evc_mode:
	if(evc_mode.key == "intel-ivybridge"):
		ivy_mask=evc_mode.featureMask
		break

If you take a look above snippet, it is fairly easy to get featuremask for particular EVC Mode.

putting it together

This script is available on my github-repo here

 
# Author: Vikas Shitole
# Website: www.vThinkBeyondVM.com
# Product: vCenter server/ per-VM EVC (Enhanced Compatibility Mode)
# Description: Script to get enbale/disable per-VM EVC on VM
# Reference: 
# How to setup pyVmomi environment?:
# On linux:  http://vthinkbeyondvm.com/how-did-i-get-started-with-the-vsphere-python-sdk-pyvmomi-on-ubuntu-distro/
# On windows: http://vthinkbeyondvm.com/getting-started-with-pyvmomi-on-windows-supports-vsphere-6-7/


from pyVim.connect import SmartConnect
import ssl
from pyVmomi import vim
import atexit
import sys
import argparse
import getpass
 
 
def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-v', '--vmname',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the VM to be configured per VM EVC')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args
 
 # Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj
 
 
args = get_args() 
s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
 
si= SmartConnect(host=args.host, user=args.user, pwd=args.password, sslContext=s)
content=si.content
vm= get_obj(content, [vim.VirtualMachine],args.vmname)

if(vm and vm.capability.perVmEvcSupported):
        print ("VM available in vCenter server and it supports perVm EVC, thats good")
else:
        print ("VM either NOT found or perVMEvc is NOT supported on the VM")
        quit()

supported_evc_mode=si.capability.supportedEVCMode
for evc_mode in supported_evc_mode:
	if(evc_mode.key == "intel-ivybridge"):
		ivy_mask=evc_mode.featureMask
		break
	 
vm.ApplyEvcModeVM_Task(ivy_mask,True)
print ("ApplyEvcModeVM_Task() API is invoked, check out your H5 client")

Line #80: This confirms whether VM found or not and it also confirms whether VM supports per-VM EVC or not.
#86-90: It is about getting right featureMask corresponding to particular EVC Mode.
#92: Finally we called the API “ApplyEvcModeVM_Task()”

How to run this script
“python perVMEVCNew.py -s 10.161.81.159 -u Administrator@vsphere.local -v DRSPerVMEVC”, user needs to enter vCenter password. Below is how it looks like.

I looked into vSphere H5 client and this is how it looks like. isn’t it cool?

Playing with featureMask

In line #86, we learned how to get hold of featureMask. is it the only way we can get featureMask and pass it to this API? Answer is NO. There are multiple ways. However, I recommend to follow the way I did on line #86 above.
Sometime you may want to configure per-VM EVC with the same featureMask as that of EVC enabled cluster.  This can be handy specially for the VMs those are outside of the EVC cluster. In such case, you can simply copy the featureMask from EVC cluster and pass it to the per-VM EVC API. Let us take look at below code snippet.

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource], args.cluster)

if(cluster):
        print ("Cluster available in vCenter server, thats good")
else:
        print ("Cluster is NOT available in vCenter server, please enter correct name")
        quit()
		
evc_cluster_manager=cluster.EvcManager()

evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
        print ("Current EVC Mode::"+current_evcmode_key)
else:
        print ("EVC is NOT enabled on the cluster")
        quit()
features_masked = evc_state.featureMask

Above EVC API property I had already discussed in my post on cluster level EVC APIs here (Refer: section #4 : Exploring EVC Cluster state). Here is the complete script, where featureMask is copied from EVC cluster. In addition, you can also choose to copy the featureMask from already per-VM EVC configured VM but if you do not document it properly, it can lead to confusion in future, hence it is always better to follow one way consistently to avoid any issues in future.

Disabling per-VM EVC

If you ask me, it is pretty easy. We just need to invoke the same API but without any featureMask as follows


vm.ApplyEvcModeVM_Task(None,True)

Some useful resources on EVC

1. Part-1: Managing EVC using pyVmomi
2. Part 2: Managing EVC using pyVmomi
3. Tutorial on getting started pyVmomi  on linux
4. Tutorial on getting started pyVmomi on Windows

I hope you enjoyed this post, let me know if you have any questions/doubts.

New fling : DRS Entitlement Viewer: a feature walk-through

A cool new fling was just released i.e. DRS entitlement viewer. As stated on fling site

“DRS entitlement viewer gives the hierarchical view of vCenter DRS cluster inventory with entitled CPU and memory resources for each resource pool and VM in the cluster”.

Good thing about this fling is that it will be deployed as HTML5 client plugin, which enables you to enjoy its features from vSphere H5 client itself. As requirement we need to have “vCenter server Appliance 6.5 or 6.7”.  Let us take a look at how to deploy. I fyou ask me, it is pretty easy and quick.

  • Fling Deployment

Step 1: Unzip the plugin package (downloaded zip bundle from fling site) to directory “/usr/lib/vmware-vsphere-ui/plugin-packages/” . Below is how it should look like. Note the exact fling directory and its content.

Step 2: Setting advanced options “CompressDrmdumpFiles set to 0” and “DrmdumpResActions set to 1”. Below is how it looks.

You can also add these DRS cluster level advanced options using below PowerCLI code. You can extend this code to configure these options on all the DRS clusters in your vCenter server.


$Cluster = Get-Cluster "Your cluster Name"
$Cluster | New-AdvancedSetting -Name CompressDrmdumpFiles -Value 0 -Type ClusterDRS
$Cluster | New-AdvancedSetting -Name DrmdumpResActions -Value 1 -Type ClusterDRS

 

Step 3: Restart vSphere H5 client service using below commands.
service-control –stop vsphere-ui
service-control –start vsphere-ui

Above step can be automated using vCenter REST APIs as well (either using PowerCLI or any other programming language without SSH connection)

I think, even first step can be automated using “ExtensionManager() vSphere API” without making any SSH connection to vCenter server appliance. (more on this in future post)

  • Fling features

As soon as vSphere H5 client is up, we should be able to see  the new “DRS entitlements” section under the “Monitor” tab for each Cluster. When I logged into my local H5 client, I was able to see below DRS entitlement from CPU and Memory perspective.

You can see DRS entitlement for every VM is showcased. You may wonder how come CPU entitlement is 0 MHz?. The reason is that, all VMs are completely idle, hence that is absolutely expected. As a first step, you just need to click on “recompute” to get the current DRS entitlement across all VMs/ResourcePools. Note that when you actually do some RLS changes on clustered VMs/RPs and you would like to see updated DRS entitlement, you will have to either click “Run DRS” or wait till default DRS execution (every 5 min).

One of the most impressive features of this fling is having ability to do “What if” analysis. Below 3 super cool scenarios are supported.

#1. What if user modifies RLS settings across VMs or resource pools across clusters. (RLS : Reservation, Limit, Shares). To try out this, first select RLS radio button and you can choose to simulate the RLS settings for one or more VMs/RPs (there is settings icon for every VM/RP except root). Finally click on “submit” button to get an idea on change in DRS entitlement for those VMs/RPs for which RLS settings were modified. Please take a look at below couple of screenshots to understand this better.

I simulated one of VM’s memory reservation to “500 MB” and once I “submit” this change, below is how What if analysis looks like.

#2: What if all the VMs’ resource demand is at 100%. I did try this analysis and below is how it was looking like. How cool is that?

#3: You can have What if analysis for both RLS and 100% both at the same time

Note: As specified above, RLS changes are just simulations, it does not actually get updated on VMs/RPs. How cool it is to do this analysis without even updating actual RLS settings on cluster!  Also,  I would highly recommend you to read the user guide to understand all What if scenarios in detail and rest of the features.

Another impressive feature is to have option to get “PowerCLI” code snippet to actually update RLS settings once your analysis is done. Please take a look at one of the examples.

I am sure this fling would be very handy for you. If you have any doubts or need any help, please leave the comment.

Below are some of the very handy DRS flings

Cluster Rules Manager

DRS lens

DRS doctor

DRS dump insight

 

vSphere 6.7 : What is PMem and How vSphere DRS works with PMem configured VMs?

vSphere 6.7 supports one of the exciting technologies on computer memory i.e. Non-Volatile Memory (NVM) or persistent memory (PMem).  In this post my focus is on briefing what is PMem technology and explore how DRS works with PMem configured VMs.

Image is taken from VMware site
What is Persistent Memory (PMem)?

In brief, PMem is next generation memory technology whose data transfer speed is as good as DRAM (50-300 ns, 100 times faster than SSDs) and unlike DRAM, it can even retain the data after reboots .  If you ask me, it is going to greatly improve the performance of applications running performance sensitive workloads  ex. analytics and acceleration databases etc. The virtual machines, which require low latency, high bandwidth and at the same time persistence are right workloads to be benefited from this technology.

Below diagram highlights the 2 modes supported by vSphere.

PMem modes supported on vSphere

i) Direct-access mode (vPMem): It requires hw-version 14 and PMem-aware VMs ex. Windows server 2016 or RHEL 7.4.x

ii) Virtual Disk mode (vPMemdisk): This works with legacy/traditional VMs, no requirement of having hw version 14 and VM need not to be PMem aware.

If you want to understand more about PMem technology , I would recommend you to watch this youtube presentation by VMware’s CTO Richard B.

vSphere DRS interop with PMem VMs

First of all, I deployed one quick lab setup: 1 vCenter, 3 ESXi hosts and simulated PMem on 2 out of 3 hosts. All these 3 hosts are inside the DRS enabled cluster. Since I do not have any PMem aware VMs, I explored PMem in vPMemdisk mode i.e. Virtual Disk Mode pointed above

Note: I did simulate PMem on  2 hosts  using a cool hack provided by engineering i.e. by running this esxcli command “esxcli system settings kernel set -s fakePmemPct -v 33”. Note that simulation must be used for lab or educational purposes only. William has written a nice article on the same as well.

Below is how the PMem storage policy and PMem local datastores look like

 

Now that we have required PMem simulated hosts, lets start exploring how DRS works with PMem. Since I did not have any PMem aware guest, as I said earlier, I explored PMem with DRS in virtual disk mode described above i.e. vPMemDisk, which works fine for legacy guest OSes. Below are my observations on the same.

i) VM creation work-flow (vSphere H5 client)

I created a VM with PMem of size 512 MB on DRS cluster. Below screenshot shows how PMem can be selected as storage.

Below screenshot shows how PMem storage policy comes into picture. This policy gets selected automatically once you select “PMem” as shown above.

Now lets take a look at the VM summary.

ii) DRS initial placement
  • The VM I initially created got registered on one of the hosts with PMem storage and I observed that DRS places (once powered on) the VM on same host, where it was residing.
  • Then I created 2 PMem (same vmdk size) configured VMs and powered on. I see that DRS placed VMs on 2 separate hosts as expected.
iii) Host maintenance mode
  • I had 2 hosts with PMem and a host without PMem. I put one of the hosts with PMem into maintenance mode, where one of the PMem configured VMs was residing. I observed that DRS migrated the VM to another PMem host as expected.
  • Then I went ahead and put even 2nd host into maintenance mode. Since there is no any host with PMem by now, I observed 2nd host can not be put into maintenance mode.
iv) DRS Affinity rules
  • I played around both VM-VM and VM-host rules and I observed that DRS tries to satisfy rules but never migrates PMem configured VM to non PMem host even though rules gets violated.
v) DRS load balancing

I could not explore more on DRS load balancing with PMem VMs but what I have observed is that DRS prefers recommending non-PMem migrations more overs PMem VM migrations. If it is absolutely needed, only then DRS recommends PMem VM migration to other compatible hosts.

Notes on PMem
  • With vSphere 6.7, only one PMem local datastore is allowed per host.
  • Normal datastore operations are not available on PMem datastores and vSphere H5 client also does not list PMem datastores under regular datastore views. Only operation available is monitoring stats.
  • Monitor PMem datastore stats using ESXCLI 
  • Host-local PMem default storage policy can not be edited. Also, user can not change the PMem VM storage policy. In order to change, user either needs to migrate or clone that VM
  • PMem will be reserved no matter VM is powered on or not
  • Though PMem can be configured on standalone hosts. However, it is highly recommended to use DRS.
  • DPM (distributed power management) supports PMem as well.

Official documentation can be found  here & here

I hope you enjoyed this post. Please stay tuned for my another blog post on PMem, where I will discuss about creating PMem VMs using API.