Category Archives: Vmware private AI and IaaS platform

Blog posts on VMware vSphere Kubernetes Service (VKS) and Private AI foundation

Hybrid Linked Mode (HLM) with vCenter Cloud Gateway using REST API

In my first post on HLM configuration using API, I explained how to configure HLM using API through Cloud VC. If you haven’t still read that post, I would highly recommend you do. In this post I am going to touch upon how to configure HLM using API but from vCenter cloud gateway appliance (VCGA).

Setting up HLM from vCenter cloud gateway is three step process


1. Deploying vCenter cloud gateway appliance using official documentation

2. Making sure prerequisite for linking with Cloud Gateway Appliance are met

3. Configuring HLM from UI or using API. In our case, we would do from API

As I mentioned in my last post, API itself does not change, it is just that API end-point will be vCenter cloud gateway appliance and input parameters we pass will be changed. Lets take a look.

POST: https://<VCG IP>/rest/hvc/links

VCG IP is the IP address of the deployed vCenter cloud gateway appliance

Below is the JSON payload this API takes in . Note thumprint is optional param

{
“spec”: {
“port”: “443”,
“domain_name”: “vmc.local”,
“username”: “cloudadmin@vmc.local”,
“ssl_thumbprint”: “E9:BD:29:9F:D6:15:4F:B7:C8:90:2C:19:03:61:HB:7A:AD:FB:B1:1E”,
“admin_groups”: [
” yourcloudadmingroup@yourdomain.local “
],
“password”: “VMware123!”,
“psc_hostname”: “Cloud PSC IP or FQDN”
}
}

Let us go over each parameter passed

port: 443 is default  port for communication between VCG and Cloud VC (can be custom port also)

domain_name: Cloud VC PSC default SSO domain (its vmc.local)

username, password, psc_hostname: Cloud VC PSC credentials (PSC is always embedded in case of VMware cloud on AWS deployment)

ssl_thumbprint : Note that this parameter is optional, it will work fine even if you do not pass this param or if you pass value as blank. If you want to pass, this is Cloud PSC SSL thumbprint, you can get it using one of the ways posted here . I am going to write a post on how to get vCenter/PSC thumbprint using API itself, please stay tuned.

admin_groups: here you specify cloud administrator group(s).  Before configuring HLM linking , identity source must be configured on Onprem and cloud admin group(s) must be given global permissions. Note that identity source need not be configured on Cloud VC, refer this KB

Note: Above API can be called from H5C API-explorer or Postman/any REST client or python script as specified below

I used the same python script used for HLM linking with Cloud VC but using VCG IP as end-point & passing inputs as described above. I see it was successful as expected & I was able to see both VCs together when logged in as user from cloudadmin group configured at HLM linking.


How to schedule VM power cycle when the guest OS reboots?

Couple of weeks back vSphere 6.7 U3 released vCenter release notes & ESXi release notes and once again an update release has got great content. Apart from several other changes gone into vSphere 6.7 U3, one of the utilities I found really useful. i.e. Ability to schedule the VM power cycle when the guest OS reboots. This post is focused on the same. Before we dig into, below is what ESXi release note says about it.

PR 2394247: You cannot set virtual machines to power cycle when the guest OS reboots

After a microcode update, sometimes it is necessary to re-enumerate the CPUID for virtual machines on an ESXi server. By using the configuration parameter vmx.reboot.powerCycle = TRUE you can schedule virtual machines for power-cycle when necessary.

This issue is resolved in this release.

Based on above description, it is clear that one of the motivations behind this cool utility is the recent meltdown/spectre patches. After applying each of these patches, it was required to do cold power cycle of the VMs in order to re-enumerate the CPUID(s) introduced by that particular patch. Performing cold power cycle (power off and power on) is always tricky to plan and painful activity. This utility enables an user to schedule power cycle as part of guest OS reboots itself. How cool is that!

In order to simulate a scenario where we should see CPUIDs are re-enumerated on a VM, I created a cluster with 2 hosts, where both hosts Max EVC mode supported was “Broadwell” but I enabled EVC on “ivy-bridge” EVC mode. Then I created a GOS VM, powered on it and used below powerCLI script to add ” vmx.reboot.powerCycle = TRUE ” into vmx file.

[powershell] 
 #PowerCLI script to add vmx entry "vmx.reboot.powerCycle"
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
Connect-VIServer -Server 10.1.1.1 -User Administrator@vsphere.local -Password VMware@123 #replace your server (VC or ESXi) name

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.extraConfig += New-Object VMware.Vim.OptionValue
$spec.extraConfig[0].key = "vmx.reboot.powerCycle"
$spec.extraConfig[0].value = "TRUE"

(get-view (Get-VM -Name VMW).ID).ReconfigVM_Task($spec)  #replace your vm name

Disconnect-VIServer -Server 10.1.1.1 -Confirm:$false #replace your server (VC or ESXi) name 
 [/powershell] 

You can run above script against either vCenter or ESXi as endpoint. User also can add above vmx entry using ESXi host client as shown below.

Add parameter into vmx file

After running above script against my VM, I see that vmx file has got ” vmx.reboot.powerCycle = TRUE ” entry . Before I restart the GOS, I checked (from MOB) CPUIDs available on this VM at ivy-bridge EVC mode and below is how it looks.

CPUIDs at ivy-bridge EVC mode
VM cpuids at ivy-bridge EVC mode

In continuation to simulate the scenario, I disabled EVC on the cluster,where max EVC mode supported is “Broadwell” i.e. ideally VMs should be running on Broadwell cpu set but since we had created our VM when EVC was enabled on “ivy-bridge”, it will continue to use “ivy-bridge” cpu set/features even though underlying host is “Broadwell”. Before 6.7 U3, we had to do cold power cycle (power off, power on) to have VM on “Broadwell” cpu set. Since we added vmx entry already, we need not to do power cycle but guest reboot in-turn will do power cycle as well and CPUIDs will be re-enumerated. Below are the CPUIDs I see on VM when I restarted the Guest OS as it is now using broadwell cpu instruction set.

VM cpuids re-enumerated to broadwell

Key observations.
1. I see vmx entry works only when it is added on the Powered ON VM.
2. When I added vmx entry onto PowerOff VM, it got added but as soon as I powered on, it got removed. It will persist during suspend and resume.
3. I observed that this setting is one time i.e. when user restarts the Guest OS, I do not see vmx entry i.e. vmx.reboot.powerCycle = TRUE any more. Hence next time you want to have similar functionality, you need to set it once again on powered ON VM.
4. This utility is just not only for cpuid re-enumeration. cpuid re-enumeration is one of the use-cases. I think it is great idea to schedule the power Cycle when you upgrade VM hardware version or upgrade vmware tools.

I hope enjoyed the post. Let me know if you have any comment.

How to configure Hybrid Linked Mode (HLM) using vCenter REST API

This is going to be my first 2019 post and in this post, I am excited to share couple of ways to configure Hybrid Linked Mode (HLM) using REST API. One of the ways is using UI REST client i.e. apiexplorer and another way is using python. If you are not already familiar with vCenter server REST APIs, I would suggest to first go over my earlier post i.e. Getting started with vCenter REST APIs.
Before I move further, I would like to point out that I am not explaining end to end steps required to configure HLM since Emad already has a detailed post on it here. My focus is on vCenter REST API required to link your Onprem vCenter server to VMware cloud on AWS (Cloud vCenter).

Notes:
1. As you might know, there are 2 ways from which user can configure HLM. One way is to configure HLM from Cloud vCenter,where you can see both VCs (onprem and cloud VC) from cloud vCenter vSphere client (H5C) and another way is to configure HLM using vCenter cloud gateway,which allows users to see both VCs  from vCenter cloud gateway H5C. Here is my new post on configuring HLM through vCenter cloud gateway using API

2. Just to avoid any confusion: Since this HLM linking operation is done from vCenter server or vCenter cloud gateway, REST API also comes from vCenter itself and not from CSP (Cloud services platform APIs, which are about Cloud services console operations)

3. Great news  is that API remains exactly same no matter which way you go for. Only change is: If you are configuring HLM from cloud vCenter (i.e. VMC VC), API end-point would be VMC VC IP/FQDN and if you are configuring HLM using vCenter cloud gateway, API end-point would be vCenter cloud gateway.

How to do HLM linking from H5C UI

You could see, in order to link, we need to pass PSC details and cloud administrator group(s). This is exactly we will do using REST API.

HLM APIs overview

Below are the REST APIs available around HLM. We are more interested in POST /hvc/links call as this is the key API for configuring HLM no matter its from VMC VC or vCenter cloud gateway.

Invoking from UI REST client i.e. APIExplorer

By now you might have already used apiexplorer (swagger based) vCenter REST client. To access it, just browse “https://[vCenter IP]/apiexplorer”. If you see above screen-shot, there are multiple APIs on HLM operations. The one we are interested in is “/hvc/links” POST method, which is responsible for HLM linking.

As part of above request body, we need to pass exactly the same details as we passed while configuring from UI. Let us take a look the “request_body” spec I passed.

{
“spec”: {
“port”: “443”,
“domain_name”: “vsphere.local”,
“username”: “Administrator@vsphere.local”,
“ssl_thumbprint”: “F9:1C:2B:E7:C5:A0:CC:02:D3:37:33:04:B0:2D:2F:6C:77:50:EB:9C”,
“admin_groups”: [
“yourcloudadmingroup@yourdomain.local”
],
“password”: “VMW!23,
“psc_hostname”: “10.161.2.5”
}
}

Let us go over each parameter passed.

port: 443 is default  Onprem PSC TCP port (can be custom port also)

domain_name: Onprem PSC default SSO domain

username, password, psc_hostname: Onprem PSC credentials

ssl_thumbprint :
If you need to pass, this is Onprem PSC SSL thumbprint, you can get it using one of the ways posted here . I am going to write a post on how to get vCenter/PSC thumbprint using API itself, please stay tuned.

admin_groups: here you specify cloud administrator group(s).  Before configuring HLM linking (apart from HLM standard requirements that Emad has posted here) , identity source must be configured on both Onprem and VMC first and cloud admin groups must be given global permissions on both VCs.

HLM linking using python

By this time, I am sure you know everything about the API, now lets head on to python way of doing these things.

This script is available on my github repo HERE

[python]
# Author: Vikas Shitole
# Website: www.vThinkBeyondVM.com
# Product: VMware Cloud on AWS (VMC)
# Description: Python script to configure Hybrid Linked Mode (HLM) between Onprem and VMC VC
# How to setup vCenter REST API environment?: https://vthinkbeyondvm.com/getting-started-with-vcenter-server-rest-apis-using-python/</pre>
<pre>import requests
import json
import ssl
import atexit
import sys
import argparse
import getpass

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

s=requests.Session()
s.verify=False

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for HLM linking')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='VMC VC IP or FQDN')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='VMC VC username')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='VMC VC password:')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='PSC port')

    parser.add_argument('-d', '--domainname',
                        type=str,
			default='vsphere.local',
                        action='store',
                        help='Onprem PSC domain name')

    parser.add_argument('-pu', '--pscuser',
                        required=False,
			default='Administrator@vsphere.local',
                        action='store',
                        help='Onprem PSC username')

    parser.add_argument('-pp', '--pscpass',
                        required=False,
                        action='store',
                        help='Onprem PSC password')

    parser.add_argument('-ph', '--pschost',
                        required=True,
                        action='store',
                        help='Onprem PSC host IP or FQDN')

    parser.add_argument('-a', '--admingroup',
                        required=False,
                        action='store',
                        default='yourcloudadmin@yourdomain.local',
                        help='Cloud admins group')	

    parser.add_argument('-pt', '--pscthumb',
                        required=True,
                        action='store',
                        help='Onprem PSC thumbprint')

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter VMC VC password:')

    if not args.pscpass:
        args.pscpass = getpass.getpass(
            prompt='Enter PSC password:')

    return args

args = get_args()
headers = {'content-type':'application/json'}
session_response= s.post('https://'+args.host+'/rest/com/vmware/cis/session',auth=(args.user,args.password))

if session_response.ok:
	print ("Session creation is successful")
else:
	print ("Session creation is failed, please check")
	quit()

payload = {
  "spec": {
    "port": args.port,
    "domain_name": args.domainname,
    "username": args.pscuser,
    "ssl_thumbprint": args.pscthumb,
    "admin_groups": [
      args.admingroup
    ],
    "password": args.pscpass,
    "psc_hostname": args.pschost
  }
}

json_payload = json.loads(json.dumps(payload))
json_response = s.post('https://'+args.host+'/rest/hvc/links',headers=headers,json=json_payload)
if json_response.ok:
	print ("HLM link is established")
else:
	print ("HLM link is NOT established, please check")
print (json_response)
print (json_response.text)

[/python]

Below is how you would run the script, take a look at the what parameters need to be passed. Each parameter is explained in above script as well.

I hope you enjoyed reading this post. Here is my new post on configuring HLM through vCenter cloud gateway using API.

VMworld US 2018: My VMware {code} power sessions experience

VMworld 2018 US was my first VMworld at Las Vegas and without a doubt, it was really exciting & insightful experience for me.  In this post, I am going to share my experience as VMware {code} power session speaker & will share youtube , facebook stream recording for the same. In addition, I will also brief on how was the excitement at VMworld as blogger.

1. Experience as power session speaker.

Beauty of these sessions was that all sessions were exclusively about coding across VMware, CICD, DevOps, open-source, hybrid cloud, cloud-native & block-chain etc.  I do not have any doubt that this track will become one of the famous VMworld tracks. Below were my power sessions.

You could see focus of all my sessions was around vSphere Automation (SOAP, REST & CLI ) , the area I am passionate about. Personally it was really thrilling experience since challenge was to make relatively complex topic such as vSphere Automation simplified and cover within stipulated time.  More exciting was that these sessions were being live streamed on facebook.  My objective was to make sure content I am presenting is simplified as much as possible that anyone will be able to understand with little or no knowledge.  After presenting these sessions, I felt that efforts put into preparation phase, multiple mock presentations were really worth. Based on the number of people attended and discussion on queries with customers, I could say that sessions were well received. Below is the list of my sessions with youtube and facebook live stream recording. I would highly recommend you to listen to all of these sessions. I am sure they will power you as well.  Your critical feedback is welcome.

2. VMware {code} power sessions

1] Closer look at vSphere Programming and CLI interfaces. 

Agenda was as follows:

  • Why APIs are needed?
  •  vSphere SOAP based SDKs
  • Getting started: vSphere API reference, MOB & pyVmomi
  • vSphere REST based SDKs
  • Getting started: API explorer
  • vSphere CLI interfaces
  • Getting started: PowerCLI
  • Useful resources

2] Overview on “Deploy & Manage vCenter server HA” using vSphere APIs

Agenda was as follows:

  • What is vCenter HA?
  •  vSphere API basics
  • vCenter HA managed objects
  • Getting started with pyVmomi
  • Sample example: Getting vCenter HA health
  • Useful resources

3] Be a ” vCenter REST API” rockstar using python

Agenda was as follows:

  • Key HTTP methods
  •  vCenter features supported @REST
  • What is api-explorer and how to use it?
  • Getting started with REST APIs using python
  • Sample example: Getting VMs and powering them
  • Useful resources

Good news is, I do have blog post associated with this content here. I would highly recommend you to read it. Watch  it on below facebook stream at 3:47:59.

Note: I would have liked to make these sessions as comprehensive as possible but in an attempt to make it simplified and due to time constraints, I had to cut short. Your feedback is welcome. In addition, I am working on uploading my slide deck , please stay tuned.

3. Excitement as VMworld Blogger

You can not imagine how proud I felt while wearing this VMworld badge as blogger.

It is absolute honor to be one among this VMworld bloggers list and being part of VMworld as blogger. Community team had setup nice blogger lounge for bloggers.

Inside general session theater, special seats with “blogger label” were made available as well. was not that extra super cool?

Most heartening moment

Last time at VMworld 2017 Barcelona , I was roaming around, not many people were knowing me. This time, I could see many people from vCommunity were knowing me, recognizing my face, saying “Hi”.  Beyond that, it was great feeling when VMware users came forward to meet me and thanked for the content I generated so far.

I would like to take this opportunity to thank  VMware community team (Eric , Elsa, Kripa, Julia ) for putting great show @power sessions and bloggers space.

pyVmomi script to confirm Speculative Store Bypass Disable (SSBD) mitigation on vSphere patches

Few hours back, VMware released vSphere patches to mitigate “Speculative Store Bypass Disable (SSBD)” security issue. Please take a look at this KB for more details. In this post, as I did in the past, I am going to provide you a pyVmomi script to confirm whether vCenter server, ESXi hypervisor and microcode patches are applied or not to mitigate this critical security issue. Before we look into script, one of the important points you should note that, these latest vSphere (both vCenter server and ESXi) patches  are cumulative &  if you haven’t applied earlier spectre vulnerability patches [released as on 20th March] yet, you can directly apply these patches to get earlier fixes as well.

pyVmomi script to confirm SSBD mitigation

Notes:

  • This script works for all vSphere releases i.e. 5.5, 6.0, 6.5, 6.7.
  • This script i.e. confirm_ssbd_patch.py is available on my github repo as well.
  • Since this patch is cumulative, focus in this script is only SSBD cpubit
  • This script takes VCIP, username, password and cluster-name (with or without EVC) as parameter.
  • Please take a note of line #72 on SSL/TLS protocol
  • As specified in the KB, you need to perform VM power-cycle post patch application.

[python]
# Author: Vikas Shitole
# Product: vCenter server
# Description: Script to confirm whether vCenter server, hypervisor and microcode patches are applied or not : vCenter/ESXi patches for Speculative Store Bypass Disable vulnerability.
# Reference: https://kb.vmware.com/s/article/55111
# How to setup pyVmomi environment?:
# Linux: https://vthinkbeyondvm.com/how-did-i-get-started-with-the-vsphere-python-sdk-pyvmomi-on-ubuntu-distro/
#Windows: https://vthinkbeyondvm.com/getting-started-with-pyvmomi-on-windows-supports-vsphere-6-7/

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
import atexit
import ssl
import sys
import argparse
import getpass

# Script to confirm whether EVC cluster is patched or not for Spectre vulenerability.

def get_args():
""" Get arguments from CLI """
parser = argparse.ArgumentParser(
description=’Arguments for talking to vCenter’)

parser.add_argument(‘-s’, ‘–host’,
required=True,
action=’store’,
help=’vSpehre service to connect to’)

parser.add_argument(‘-o’, ‘–port’,
type=int,
default=443,
action=’store’,
help=’Port to connect on’)

parser.add_argument(‘-u’, ‘–user’,
required=True,
action=’store’,
help=’Username to use’)

parser.add_argument(‘-p’, ‘–password’,
required=False,
action=’store’,
help=’Password to use’)

parser.add_argument(‘-c’, ‘–cluster’,
required=True,
action=’store’,
default=None,
help=’Name of the cluster you wish to check’)

args = parser.parse_args()

if not args.password:
args.password = getpass.getpass(
prompt=’Enter vCenter password:’)

return args

# Below method helps us to get MOR of the object (vim type) that we passed.
def get_obj(content, vimtype, name):
obj = None
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name and c.name == name:
obj = c
break
container.Destroy()
return obj

args = get_args()
s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
si= SmartConnect(host=args.host, user=args.user, pwd=args.password,sslContext=s)
content=si.content
cluster_name=args.cluster

print ("————————————-")
#Check whether vCenter server is patched or not
supported_evc_mode=si.capability.supportedEVCMode
# It is not required to check "ivy-bridge" EVC mode, you can choose any EVC mode from "intel-penryn" onwords.
for evc_mode in supported_evc_mode:
if(evc_mode.key == "intel-ivybridge"):
ivy_masks=evc_mode.featureMask
break

vCenter_patched=False
for capability in ivy_masks:
if(capability.key in ["cpuid.SSBD"] and capability.value=="Val:1"):
print ("Found::"+capability.key)
vCenter_patched=True
if(not vCenter_patched):
print ("No new cpubit found, hence vCenter server is NOT patched")
else:
print ("New CPU bit is found, hence vCenter Server is patched")
print ("Current vCenter server build::"+si.content.about.fullName)

#Cluster object
cluster = get_obj(content,[vim.ClusterComputeResource],cluster_name)
if(not cluster):
print ("Cluster not found, please enter correct EVC cluster name")
quit()

print ("Cluster Name:"+cluster.name)

# Get all the hosts available inside cluster
hosts = cluster.host

#Iterate through each host to get MaxEVC mode supported on the host
for host in hosts:
print ("———————————-")
print ("Host:"+host.name)
feature_capabilities = host.config.featureCapability
flag=False
for capability in feature_capabilities:
if(capability.key in ["cpuid.SSBD"] and capability.value=="1"):
print ("Found::"+capability.key)
flag=True
if(not flag):
print ("No new cpubit found, hence "+host.name+" is NOT patched")
else:
print ("New CPU bit is found, hence "+host.name+" is patched")

atexit.register(Disconnect, si)

[/python]

Let us take a look at below output.

Output

C:\Professional\vThinkBeyondVM\Spectre posts>python hosts_patched_ssbd.py -s 10.20.30.35 -u Administrator@vsphere.local -c “New Cluster”
Enter vCenter password:
————————————-
Found::cpuid.SSBD
New CPU bit is found, hence vCenter Server is patched
Current vCenter server build::VMware vCenter Server 6.7.0 build-8833179
Cluster Name:New Cluster
———————————-
Host: 10.20.30.51
No new cpubit found, hence 10.20.30.51 is NOT patched
———————————-
Host: 10.20.30.52
Found::cpuid.SSBD
New CPU bit is found, hence 10.20.30.51 is patched

Above output shows that vCenter server is patched and one of the two ESXi hosts is patched successfully.

Further learning resources
  1. per-VM EVC tutorial
  2.  Part-1: Managing Cluster level EVC using pyVmomi
  3. Part 2: Managing Cluster level EVC using pyVmomi
  4. Tutorial on getting started pyVmomi  on linux
  5. Tutorial on getting started pyVmomi on Windows

I hope you will find this post useful, please stay tuned for my next blog post on per-VM EVC wrt to these mitigation patches.