Tag Archives: Spectre vulnerability

pyVmomi script: How to confirm ESXi hypervisor & microcode patches are applied?: Spectre vulnerability

Some time back, VMware has updated one of KBs on “Hypervisor-Assisted Guest Mitigation for branch target injection”. Please take a look at this KB here. In this KB, there is one important section on “how to confirm a host has both patched microcode and patched VMware hypervisor?” In fact, there are multiple ways to confirm it and one of the ways is specified in the KB itself i.e. scanning VM vmware.log file for new cpuids. Last time, I had automated exactly the same way using PowerCLI. If you prefer PowerCLI, you can use mine or you can also refer nice PowerCLI script written by William for the same. If there are PowerCLI scripts available for confirming these patches, you may wonder, what is that I am going to write in this post? Since there is a very handy host level vSphere API property available to confirm these patches, I thought to write a quick pyVmomi script. This will be very handy for people who prefer pyVmomi. Also as it uses native vSphere APIs, it will be faster than its PowerCLI equivalent.

This script i.e. hosts_patched.py available on my git repo as well.

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-c', '--cluster',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the cluster you wish to check')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args


# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj



args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
 print ("Cluster not found, please enter correct EVC cluster name")
 quit()

print ("Cluster Name:"+cluster.name)

# Get all the hosts available inside cluster
hosts = cluster.host

#Iterate through each host to get MaxEVC mode supported on the host
for host in hosts:
 print ("----------------------------------")
 print ("Host:"+host.name)
 feature_capabilities = host.config.featureCapability
 flag=False
 for capability in feature_capabilities:
  if(capability.key in ["cpuid.STIBP", "cpuid.IBPB","cpuid.IBRS"] and capability.value=="1"):
   print ("Found::"+capability.key)
   flag=True
 if(not flag):
  print ("No new  cpubit found, hence "+host.name+" is NOT patched")
 else:
  print ("New CPU bit is found, hence "+host.name+" is patched")
	

atexit.register(Disconnect, si)


This script takes VCIP, username, password and cluster-name (with or without EVC) as parameter. Let us take a look at below output.

Output:
vmware@localhost:~$ python hosts_patched.py -s 10.192.30.40 -u Administrator@vsphere.local -c EVCCluster
Enter vCenter password:
Cluster Name:EVCCluster
———————————-
Host: 10.20.43.35
Found ::cpuid.IBPB
Found ::cpuid.IBRS
Found ::cpuid.STIBP
New CPU bit is found, hence 10.20.43.35 is patched
———————————-
Host:10.20.43.36
No new cpubit found, hence 10.20.43.36 is NOT patched
vmware@localhost:~$

Based on above script, it is clear that there are 2 hosts inside the cluster and only one of them has exposed new cpuids, that confirms it is patched with both ESXi hypervisor & microcode.

Further learning:

1. I highly recommend you take a look at last post on EVC cluster.
2. Empty EVC cluster issue discussed here is fixed with latest vCenter Patches
3. My PowerCLI script to achieve the same
4. Tutorial on getting started pyVmomi

I hope you enjoyed this post, let me know if you have any questions/doubts.

pyVmomi script: How to confirm whether EVC cluster is patched or not ? Spectre vulnerability

As per latest update from VMware, new vCenter and ESXi patches are released. These patches primarily will help enable “Hypervisor-Assisted Guest mitigation”. With these patches, one of the important aspects is to make sure whether entire EVC cluster is patched/upgraded or not. In this blog post, we will take a look at the pyVmomi script, which will confirm whether EVC cluster is patched with both hypervisor & microcode patch or not. Before we take a look at the script, let me quote statement from VMware KB on EVC cluster behavior.

“The vCenter patches enable vMotion compatibility to be retained within an EVC cluster.
In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated. At that time, the cluster will automatically upgrade its capabilities to expose the new features.”

As per above quote, new features should be available on EVC cluster once all the hosts inside the cluster are patched. Now lets take a look at the script.

This script i.e. is_evc_cluster_patched.py is available on my git-hub repo.


from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for talking to vCenter')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='vSpehre service to connect to')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='Port to connect on')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='Username to use')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='Password to use')

    parser.add_argument('-c', '--cluster',
                        required=True,
                        action='store',
                        default=None,
                        help='Name of the cluster you wish to check')	

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter vCenter password:')

    return args




# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
 obj = None
 container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
 for c in container.view:
  if name and c.name == name:
   obj = c
   break
 container.Destroy()
 return obj


args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
 print ("Cluster not found, please enter correct EVC cluster name")
 quit()

print ("Cluster Name:"+cluster.name)
evc_cluster_manager=cluster.EvcManager()


evc_state=evc_cluster_manager.evcState
current_evcmode_key= evc_state.currentEVCModeKey

if(current_evcmode_key):
 print ("Current EVC Mode::"+current_evcmode_key)
else:
 print ("EVC is NOT enabled on the cluster, please enable it first")
 quit()

feature_capabilities = evc_state.featureCapability

flag=False
for capability in feature_capabilities:


    if(capability.key in ["cpuid.STIBP", "cpuid.IBPB","cpuid.IBRS"] and capability.value=="1"):
        print ("Found::"+capability.key)
        flag=True

if(not flag):
    print ("No new cpubit found on EVC cluster,hence cluster is NOT fully patched/upgraded")
else:
    print ("EVC cluster is patched, enjoy!, this also confirms all the hosts inside this EVC cluster are patched as well")


atexit.register(Disconnect, si)


If you take a look at line 81-85, these are exact same EVC vSphere API properties that we discussed in my tutorial here. Same properties I have used to confirm whether EVC cluster is patched/upgraded or not. This script takes VCIP, username, password and EVC cluster name as parameter. Let us take a look at the output.

Output:

From output, it is clear that EVC cluster is patched successfully. It does mean that all the hosts inside the clusters are patched with both microcode and hypervisor patch. If at-least one of the hosts inside this EVC cluster is NOT patched, EVC cluster is NOT going to expose new cpuids on cluster and will be called as NOT patched.

Please note that since EVC APIs got introduced from vSphere 6.0, above script will be applicable for vSphere 6.0 and 6.5. Also, microcode/BIOS support is available for few Intel/AMD CPU models, please check out VMware KB for more details on it.

Some useful resources on EVC

1. If you want to confirm each host inside the cluster (with or without EVC) is patched or not, please take a look at my latest blog post
2. Empty EVC cluster issue discussed here is fixed with latest vCenter Patches
3. Part-1: Managing EVC using pyVmomi
4. Part 2: Managing EVC using pyVmomi
5. Tutorial on getting started pyVmomi

I hope you enjoyed this post, let me know if you have any questions/doubts.

PowerCLI script : How to confirm a ESXi VMware hypervisor & microcode patches are applied?: Spectre vulnerability

Some time back, VMware has released one of important KBs on “Hypervisor-Assisted Guest Mitigation for branch target injection”. Please take a look at this KB here. In this KB, there is one important section on “how to confirm a host has both patched microcode and patched VMware hypervisor?” and below is what is posted on KB.

Confirmation of Correct Operation
To confirm a host has both patched microcode and patched VMware hypervisor, use the following steps:
1.Power on a Virtual Machine which is configured to use Virtual Hardware Version 9 or later.
2.Examine the vmware.log file for that VM and look for one of the following entries:
“Capability Found: cpuid.IBRS”
“Capability Found: cpuid.IBPB”
“Capability Found: cpuid.STIBP”
3. Any of the above log entries indicates that both the CPU microcode and hypervisor are properly updated.

I thought it would be really handy if we automate it using PowerCLI. Accordingly, I started looking for any existing cmdlets which can help me to retrieve the VM logs (specially vmware.log) and I could see PowerCLI guru Luc Dekens had written a article on retrieving VM logs long back. I leveraged the same and came up with one quick PowerCLI script to confirm whether a host has both patched microcode and patched VMware hypervisor or not. Here we go.

Update 03/20: As per this KB, new ESXi/vCenter patches are available, hence update (01/13) posted below is no more valid. Once you start applying new patches, please start using either PowerCLI script discussed in this blog post or my latest pyVmomi script for the same.

Update 01/13: As per latest update from VMware on KB, I have published another PowerCLI script, I recommend you to take a look at it before reading this post further.

Script execution steps:
i) It gets all the connected hosts inside the specified cluster. It can be easily modified to consider host from particular datacenter as mentioned above.
ii) It will then iterate through each host in sequence and create a dummyVM.
iii) DRS automation on created dummyVM will be disabled since we need to powerON the VM and if DRS is enabled on the cluster, as part of DRS initial placement workflow, it may place/powerON the VM on some other host inside the cluster.
iv) Now it will powerON the dummyVM
v) Download the vmware.log file at specified location
vi) Scan for the log lines specified in KB (posted above) & record the results.
vii) PowerOff the created VM (to avoid any disk space utilization, you can choose to delete this VM as well)
viii) Step iii) through viii) will repeated for every host inside the cluster.
xi) Finally log the result into one CSV file. CSV file will have HostName & Status (Patched or Un-Patched). Refer CSV sample below.

Below script is available on my git-hub repo as well

<# .SYNOPSIS PowerCLI script:For spectre vulnerability : "How to confirm whether ESXi hypervisor & CPU microcode patches are applied or not?" .NOTES Author: Vikas Shitole .NOTES Site: www.vThinkBeyondVM.com .NOTES Please add the vCenter server IP/credetails as per your environment .NOTES Relese notes: VC 6.5 U1g: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-65u1g-release-notes.html VC 6.0 U3e: https://docs.vmware.com/en/VMware-vSphere/6.0/rn/vsphere-vcenter-server-60u3e-release-notes.html VC 5.5 U3h: https://docs.vmware.com/en/VMware-vSphere/5.5/rn/vsphere-vcenter-server-55u3h-release-notes.html #>

#vCenter Connection, please modify as per your env.

Connect-VIServer -Server 10.160.20.30 -Protocol https -User administrator@vsphere.local -Password VMware!32


function Get-VMLog{
<# .SYNOPSIS Retrieve the virtual machine logs .DESCRIPTION The function retrieves the logs from one or more virtual machines and stores them in a local folder .NOTES Author: Luc Dekens .PARAMETER VM The virtual machine(s) for which you want to retrieve the logs. .PARAMETER Path The folderpath where the virtual machines logs will be stored. The function creates a folder with the name of the virtual machine in the specified path. .EXAMPLE PS> Get-VMLog -VM $vm -Path "C:\VMLogs"
.EXAMPLE
	PS> Get-VM | Get-VMLog -Path "C:\VMLogs"
#>
 
	param(
	[parameter(Mandatory=$true,ValueFromPipeline=$true)]
	[PSObject[]]$VM,
	[parameter(Mandatory=$true)]
	[string]$Path
	)
 
	process{
		foreach($obj in $VM){
			if($obj.GetType().Name -eq "string"){
				$obj = Get-VM -Name $obj
			}
		}
		$logPath = $obj.Extensiondata.Config.Files.LogDirectory
		$dsName = $logPath.Split(']')[0].Trim('[')
		$vmPath = $logPath.Split(']')[1].Trim(' ')
		$ds = Get-Datastore -Name $dsName
		$drvName = "MyDS" + (Get-Random)
		New-PSDrive -Location $ds -Name $drvName -PSProvider VimDatastore -Root '\' | Out-Null
		Copy-DatastoreItem -Item ($drvName + ":" + $vmPath + "vmware.log") -Destination ($Path + "\" + $obj.Name + "\") -Force:$true
		Remove-PSDrive -Name $drvName -Confirm:$false
	}
}
$report = @()

#Location where vmware.log file gets downloaded
$drive="C:\"
$vmname="MyVM"

# Any of below lines must be found in vmware.log file to confirm microcode & VMware hypervisor patch
$pat1='Capability Found: cpuid.IBRS'
$pat2='Capability Found: cpuid.IBPB'
$pat3='Capabliity Found: cpuid.STIBP'

$clusterName="EVCCluster" #Your cluster name, script assumes that cluster is available
$cluster= Get-Cluster -Name $clusterName

#Get connected hosts from above cluster
$esxhosts = Get-Cluster $cluster | Get-VMHost -State Connected

#Counter used to give unique name to dummyvm, you can use any number of your choice.
$i=45

#Iterating through each host for VM creation and scanning vmware.log file
Foreach ($ESXHost in ($esxhosts)){
	$vm=$vmname+$i
#Creating dummy vm with below configuration
	New-VM -Name $vm -VMHost $ESXHost -ResourcePool $cluster -DiskGB 1 -MemoryGB 1 -DrsAutomationLevel Disabled -DiskStorageFormat Thin
	Start-VM -VM $vm -RunAsync -Confirm:$false  #DRS may powerON this VM on some other host inside the cluster
	$dest=$drive+$vm
	Get-VMLog -VM $vm -Path "C:\"
	if (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat1}){
 		$row = '' | select HostName, Status
 		$row.HostName = $($ESXHost.name)
 		$row.Status="Patched"
 		$report += $row
 		Write-Host "Matched pattern:"+$pat1+":"+$vm+":"+$($ESXHost.name) 
 }
 ElseIf (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat2}){
  	$row = '' | select HostName, Status
 	$row.HostName = $($ESXHost.name)
	$row.Status ="Patched"
    $report += $row
    Write-Host "Matched pattern:"+$pat2+":"+$vm+":"+$($ESXHost.name) 
 }
  ElseIf (Get-ChildItem -Path $dest -Filter "*.log" | Where {Get-Content -Path $_.FullName | Select-String -Pattern $pat3}){
   	$row = '' | select HostName, Status
  	$row.HostName = $($ESXHost.name)
 	$row.Status ="Patched"
 	$report += $row
   	Write-Host "Matched pattern:"+$pat3+":"+$vm+":"+$($ESXHost.name) 
 }
 Else{
  	$row = '' | select HostName, Status
 	$row.HostName = $($ESXHost.name)
	$row.Status ="Un-Patched"
  	$report += $row
 	Write-Host "Nothing matched on for VM on host:"$($ESXHost.name) 
 }
 Stop-VM -VM $vm -RunAsync -Confirm:$false
 #you can delete this VM as well to avoid any disk space consumption or minimize the disk size
 $i++
}
#Log the report into this CSV file, you can provide your name
$report | Sort HostName | Export-Csv -Path "D:PatchStatus.csv"

#Disconnect the vCenter server
 Disconnect-VIServer -Confirm:$false
 

Sample CSV file result:
#TYPE Selected.System.String
“HostName”,”Status”
“10.20.30.20”,”Patched”
“10.20.30.21”,”Un-Patched”

Notes:
1. For the sake of simplicity I have hardcoded some values, please do change as per your environment.
2. If you have any comment/feedback on above script, please do provide.
3. There can be other ways to automate the same. Since I started with this approach, I continued. I haven’t yet thought about other approach (probably we may be able to just use datastore browser APIs to navigate/scan vmware.log file instead of downloading it outside).
4. I would recommend you to use this script as sample and make changes as needed. Let me know what changes you did or probably any other way you found.
5. I have tested this script on vCenter/ESXi 6.5 & it worked as expected for me. I will update this article as I update above script

Note: I would like you to understand “vMotion and EVC Information” section from KB, specially below lines. Let me know if you have any doubts I can share my understanding.

“In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated”

Some must read links:
1. VC 6.5 U1g release notes
2. VC 6.0 U3e release notes
3. VC 5.5 U3h release notes
4. Microsoft has released similar validation script for their Guest OS

I hope this will be helpful.