Tag Archives: VMware

PowerCLI script: How to hide the speculative-execution control mechanism for VMs running on affected processors ?

Here is the latest KB update from VMware on Hypervisor-Assisted Guest Mitigation for branch target injection. I would highly recommend you to read this latest KB to understand all minute details. As per KB, it is recommended NOT to apply recently released ESXi patches i.e. ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG on servers based out of affected processors. However, if you have already applied aforementioned patches on affected servers/hosts, VMware recommends to do following in order to hide the speculative-execution control mechanism for virtual machines(quoting from KB)

Update 03/20: As per this KB, new ESXi/vCenter patches are available. With new patches released as on 03/20, this post is NO more valid. Once you start applying new patches, please start using either PowerCLI script for confirming both hypervisor and microcode patches discussed on my earlier blog post or my latest pyVmomi script for the same.

Quote started

For servers using affected Intel processors (see Table 1.) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following::

On each affected ESXi host, add the following line in the /etc/vmware/config file:
cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-”
This will hide the speculative-execution control mechanism for virtual machines which are power-cycled afterwards on the ESXi host.
This line will need to be removed after applying a future fixed microcode from Intel in order to enable the full guest OS mitigations for CVE-2017-5715.
When convenient, power-cycle virtual machines on the affected ESXi hosts; rebooting of the ESXi host is not required.

Quote ended

The way I wrote a PowerCLI script for confirming patch validation, I thought it would be really handy to have another quick PowerCLI script to hide the speculative-execution control mechanism for virtual machines as suggested by VMware in KB.

What this script does?

1. Get all the connected hosts from specified vCenter server datacenter.
2. Start iterating through each ESXi host.
3. Check whether ESXi host max supported EVC mode is “intel-haswell” or “intel-broadwell”. If yes, script will then check whether new cpuids are really exposed to these hosts.
4. If new cpuids are exposed on haswell or broadwell hosts, it picks that host for adding “cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” into “/etc/vmware/config” file.
5. Before updating above file, script does check whether SSH is enabled on picked (haswell or broadwell) host, if SSH is not enabled, it does enable it.
6. If SSH on picked host is already enabled, it will create SSH session & takes the backup of the file /etc/vmware/config, which gets named as per time stamp.
7. Once file is backed up, it will go ahead and add “cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” line at the end of file “/etc/vmware/config”
8. Once file edit operation is done, script does disable SSH on the host where it was not enabled before editing. It also confirms whether SSH is really disabled.
9. Above steps will be repeated only for intel-haswell or intel-broadwell based hosts, where new cpuids are exposed.
10. It keeps logging (on console) the hosts which are edited/not-edited
11. Finally it disconnects vCenter server session.

This script is available on my on my git repo: hideNewCpuids.ps1

After executing above script, it is critical to do VM power-cycle (PowerOFF the VM & PowerON) to get this change affected. Once you do powerCycle, you may wonder how to verify whether speculative-execution control mechanism for virtual machines is actually got hidden or not. When we say “hide the speculative-execution control mechanism for virtual machines”, as earlier blog post pointed, it exposes new CPUIDs to the VM i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP. So it is meant that after script execution, once user does VM power-cycle, these CPUIDs should NO more be exposed to VMs.

How to quickly verify these cpuids are hidden from VM? It can easily done using vCenter MOB (Managed object browser) as follows:

1. To check whether host is having new cpuid capability: please use vCenter MOB (https:// your VC IP/mob): https://10.192.10.20/mob/?moid=host-10&doPath=config (replace your VC IP , host moid and check for “featureCapability” property, you will get list of new cpuids this host is capable of.

Note: Patched ESXi hosts continue to show new CPUIDs (value as 1) even after editing /etc/vmware/config file. This is expected since update to file /etc/vmware/config just hides speculative-execution control mechanism for virtual machines.

2. To check whether new cpuids are hidden from VM:
https://10.192.10.20/mob/?moid=vm-19&doPath=runtime (replace your VC IP , vm moid and check for “featureRequirement” property.

Result:
Before editing file /etc/vmware/config: “”featureRequirement” property of VM shows new CPUIDs i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP

After editing file /etc/vmware/config : “”featureRequirement” property of VM does NOT show new CPUIDs (PowerCycle needed). i.e. cpuid.IBRS, cpuid.IBPB, cpuid.STIBP

How to restore the host to old state? i.e. before editing /etc/vmware/config file
– Before doing this, do make sure that you completely understand the impact it can have on your environment. KB #52345 gives more details.
– To restore, you can either remove added line from /etc/vmware/config or you can comment out using “#” as prefix. To verify this change, you will have to do powercycle. No host reboot required as specified in KB post file update.

Notes:
1. For the sake of simplicity I have hardcoded some values, please do change as per your environment.
2. If you have any comment/feedback on above script, please do provide.
3. I have tested this script on vCenter/ESXi 6.0 & it worked as expected for me.

Additional references:
1. If you are new to MOB, refer official doc

2. I would like to acknowledge this post on Posh-SSH module. I leveraged SSH part of it.

3. If you want to verify above change using PowerCLI, you can leverage nice script written by our own William Lam here.

4. I may write a python script using pyVmomi for verification similar to what we did using MOB above.

I hope this post was useful, please share as appropriate.

My first ever VMworld 2017 experience.

This year, it was the very first time I got an opportunity to be part of VMworld Europe, Barcelona & I thought it would be good to share my experience with you. I am sure you will enjoy reading it.

1] My experience as Tech-talk Speaker

I had couple of tech-talks as part of VMworld vBrownBag. This was the very first time I got an opportunity to deliver tech-talks outside India & that too at biggest conference to unknown audience (Customers, partners etc). Personally, it was really a thrilling & exciting experience for me. More exciting was, tech-talks were arranged in open VMVillage community area (I was assuming that they would be in closed hall/room) & tech-talks were being live streamed/recorded as well. Based on the interest and feedback received, I could say both talks were really well received.

Below are the youtube links to my tech talks. Please take a look and please provide your feedback if any

i) VMware Log Tuner (aka vLog Tuner or VLT):
(Project contributors: @chiragarora, @durgakarri & @vThinkBeyondVM (myself))

Youtube link:

ii) DRS Cluster Rules Manager:
(Project contributors: @durgakarri, Gururja Hegdal & Vikas Shitole)

Youtube link :

Take a look at all vBrownBag tech-talks

2] Here are some of the VMworld highlights:

As usual, general sessions by our leaders were awesome. It was truly amazing to be part of such huge community & buzz. Apart from general sessions, as per me, below are the highlights.
i) VMware cloud on AWS (VMC or VMW on AWS): As expected, VMC had a lot of coverage across quick talks and breakout sessions. For me, the most insightful breakout session on VMC was by Solution Architect from Amazon and Technical marketing director from VMware i.e. “Business value of VMware Cloud on AWS”. In addition to these sessions, I attended some more insightful breakout sessions on VMC as well as other VMware products.

Here are some of top sessions from VMworld Europe

Here are some of top sessions from VMworld US

William Lam has compiled very nice list of all breakout sessions available on youtube

ii). Hands on Labs: After attending couple of sessions on VMC, I was very much interested in how this works for customer in action. This is where VMC on AWS HOL helped. It was truly great experience playing with very slick HTML5 based VMC interface & configuring various VMC components.

Here are more learning resources on VMware cloud on AWS

For learning VMware products, please look at Hands on lab

iii) Solution Exchange: This is the place where all of our partner ecosystem display their solutions. I visited 6-7 booths out of many others. It was heartening to see how VMware partner ecosystem has developed solutions around VMware products.

iv) Meeting with people: After communicating with many VMware & vExperts community members over email, blogs & twitter for last 3-4 years, it was exciting to meet some of VMware as well as vExperts community personally.

Selfie with Katie Bradely, vExpert Community manager

v) Things I missed: There are couple of things I missed as my tech-talks were scheduled on back-to-back days. However, as VMworld schedule is such huge, multiple tracks at the same time, we cannot catch everything as well. Having said that, I think, I missed to be part of VMworld Hackthon organized by our very own champs “William Lam”(@lamw) & “Alan Renouf”(@alanrenouf). I heard it was really cool (However, I did personally meet both of them on the way & visited VMware {code } booth to know all the programs they run). Another thing I missed is getting signed copy of this great & most sought book “VMware vSphere 6.5 Host Resources Deep Dive” by “Frank Denneman (@FrankDenneman) & Niels Hagoort (@NHagoort)”. Of course, I missed some of live breakout sessions but I can listen to recorded ones.

Here is blog post by William on VMworld Hackathon

Know more about VMware {code }

vi) Exploring Barcelona: Since BCN vs IST difference is just 3+ hrs, jet lag was not an issue & even temperature was similar what we have now in BLR. I explored BCN on the way back from VMworld Venue to hotel (5.5 KM) by walk, it was very pleasant experience. Yes, food was the issue but fortunately, I got “Indian Bollywood restaurant” walk-able distance from my hotel. In addition, being in Barcelona, I could feel the craze about football (During Barcelona vs Juventus match), how people worship players like Messi (Similar to Cricket in India). I would have liked visiting some places there but could not manage during my short trip.


vi) Parties/goodies & VMware store:
There was exclusive vExpert party arranged by our digital marketing team. Our own vSphere team has awarded all vExpert a Swag Bag containing “Power Bank, Book, Pen, bottle & some cool stickers”. In addition, many partners were giving away lot of goodies such as T-shirts, stickers, hat, bags etc. As I visited some of booths, I did get some of them & who does not like goodies? In addition, I enjoyed looking into VMware related books & official merchandise at VMware store. It was heartening to see the book that I reviewed as technical reviewer i.e. vSphere Design essentials.

vii) Other interesting booths: There were some more exciting booths/lounge such as “meet the experts” & “vmware educational services”.

Overall, I had a great trip, met many people, and learned a few things. Clearly, VMworld is a great platform to interact with VMware people, customers & vExperts

VMware Rock Stars I get motivation from since I joined VMware

Recently I was going through some of the VMworld 2015 videos and I came across this interesting new concept this year i.e. VMware Rock Stars. I was very happy to see Duncan’s and William’s interview under this new concept as VMware Rock stars. For me both are the rock stars soon after I joined VMware around 2.5+ years back. I really enjoyed both the interviews and I thought it is worth to share with you all. I am sure, you will enjoy as well.

1. Duncan Epping, Chief Technologist

2. William Lam, Staff Engineer II

For all other super cool VMware related videos : CLICK HERE