Category Archives: HOME

Introduction to managing vSphere Supervisor Cluster using DCLI

My last post was about introduction to vSphere Supervisor Cluster Kubernetes REST APIs, where I briefed about key constructs such as “vSphere Supervisor Cluster” and “Tanzu Kubernetes cluster” but major focus was to introduce REST APIs around it and how to execute these APIs using H5C built-in API Explorer. As per me, it is must to read that post before you proceed. This post is about how to manage vSphere Supervisor Cluster using DCLI (Datacenter CLI). As you might already know that DCLI is a CLI exclusively for REST APIs and it supports vSphere Automation APIs (REST APIs for vSphere), VMware cloud on AWS and even NSX as well. Beauty of the DCLI is that it comes by-default as part of VCSA installation though we can install it in other Guest OSes also.

As I mentioned in the last post , “vSphere Supervisor Cluster” REST APIs are exposed by vCenter server i.e. every DCLI command we execute would be executed against vCenter server, where vSphere Supervisor cluster is configured or going to be configured. Let’s dive into it. In my case, I am going to access DCLI that comes automatically with VCSA.

First step is to SSH VCSA 7.0 and run below command

dcli +interactive +server 10.20.30.40 +skip-server-verification

We need to pass vCenter server IP as a server. I will be running DCLI in interactive mode so that we get automatic help on available options with press of space or it gets showed as you type. This is very useful if you do not know where to start for new APIs as this.




We are now in DCLI console. As I showed in my last post, these REST APIs are identified as “namespace management” & “namespaces”. Since these are “vcenter” APIs, when we just type “vcenter” and press “space”, you will get auto-populated list of options available as shown below, is not that cool ?

As in my last post, let’s check the compatibility of the cluster to confirm whether key requirements are met or not on given cluster

dcli> vcenter namespacemanagement clustercompatibility list

You can see cluster “domain-c9” (its moid) is compatible but cluster “cluster-c19” does not meet any basic requirements.

Below are few more basic commands (mostly GET calls)

Command #1: Disabling “vSphere with K8s” on given cluster

dcli> vcenter namespacemanagement clusters disable --cluster <cluster moid>

cluster moid/identifier looks like “domain-cx”. You can get it from UI (click on cluster and just check URL in browser or REST API/SOAP/MOB can help as well)

Command #2: Know compatible VDS on given cluster

dcli> vcenter namespacemanagement distributedswitchcompatibility list --cluster domain-c9

Command #3: Get compatible NSX-T edge cluster on given cluster and VDS

dcli> vcenter namespacemanagement edgeclustercompatibility list --cluster domain-c9 --distributed-switch "50 13 e2 3b 7d 5e 59 4b-63 1c c5 d9 ab 25 cd 7e"

VDS uuid you get from command #2 above.

Command #4: Get list of namespaces available

dcli> vcenter namespaces instances list

Command #5: Get available Kubernetes versions

dcli> vcenter namespacemanagement clusteravailableversions list
or
dcli> vcenter namespacemanagement software clusters get --cluster <cluster-identifier/moid>
or
dcli> vcenter namespacemanagement software clusters list

Command #6: Know what are sizes and resources requirements (TINY, SMALL, MEDIUM, LARGE )

dcli> vcenter namespacemanagement clustersizeinfo get

I hope this article was good head start for you on how to manage vSphere Supervisor cluster using DCLI. There are more advanced operations like enable, update, create (POST calls) etc that I will have deep dive into my future posts. Please stay tuned.

Further reading:
1. REST API documentation and my new post on automating Supervisor cluster configuration or Automating supervisor cluster workflows using Java SDK
2. How to get vSphere with Kubernetes?
3. vSphere with Kubernetes 101 is here
4. Introduction to vSphere Supervisor Cluster REST APIs
5. Official documentation for vSphere with Kubernetes is here


vSphere with Kubernetes: Introduction to vSphere Supervisor cluster REST APIs

vSphere 7.0 is now generally available and without a doubt “vSphere with Kubernetes” (former project pacific) is biggest highlights of the release. I think, this will have similar or even bigger impact on IT landscape since the launch of ESXi or vMotion in the past. Coolest thing is that VMs and containers can co-exist and no tools to manage them are changing. I am personally very happy that vSphere is now just not about VMs but also containers & beyond (read “vThinkBeyondVM”) as well.

Note: As you already might know “vSphere with Kubernetes” is available through VMware Cloud Foundation(VCF). I highly recommend you to read this quick post on how to get “vSphere with Kubernetes”? Post is written by none other than Kit Colbert . Very straight and simplified post if you ask me.

Before I discuss about REST APIs, I would like to provide brief about 2 important constructs below i.e. ” Supervisor cluster” and “Tanzu Kubernetes Cluster” (aka Guest cluster). Please refer vSphere with Kubernetes 101 white paper for more details.

Supervisor cluster : It is a special kind of Kubernetes cluster, where ESXi hosts inside vSphere Cluster become worker nodes and K8s Control VMs/Master VMs will run on the top same vSphere cluster. In this cluster, user creates a namespace and triggers “vSphere Pods” to run container workloads using same “kubectl” CLI. (vSphere admin manages it)

Here is a blog post demonstrating how to automate Supervisor cluster configuration

Tanzu Kubernetes Cluster” (aka Guest cluster): This is upstream-compliant Kubernetes cluster, which runs on the top of “Supervisor cluster” under the “namespace” created in Supervisor layer. This cluster uses open source Cluster API for life-cycle management (Devloper/DevOps manages it from familiar kubectl itself)


As usual I enjoy exploring new APIs got introduced as part of new vSphere release . In this post, my focus is on REST APIs exposed to manage life-cycle of ” Supervisor cluster”. As I run through vCenter API explorer (H5C -> Menu -> Devloper center -> API Explorer), I see this release has more REST APIs compared to SOAP APIs unlike any earlier vSphere releases. You can refer REST API documentation here as well. In this post, I will introduce you to new APIs got exposed as part of Supervisor cluster. I am pleased to see that 100% of the APIs are REST based. How cool is that !

First things to note that “API-endpoint” to call these APIs is “vCenter server”. i.e. we will be calling Supervisor cluster APIs against vCenter server.

Lets take look at below screenshot for how these APIs look like.


As you can see these APIs are identified under name “namespace_management” and “namespaces”
namespace_management : APIs under this section help manage all the aspect of managing “Supervisor cluster”.
namespaces : APIs under this section help manage “namespaces” created on the top of Supervisor cluster. Tanzu Kubernetes clusters are also created under namespaces created in Supervisor layer.

Most critical APIs are under “namespace_management/clusters ” section, where key APIs like enable/disable/update vSphere supervisor cluster are available. Below is how “enable” call looks like. i.e. configuring “Supervisor cluster”.
POST call
https://{server}/api/vcenter/namespace-management/clusters/{cluster}
server : vCenter IP/FQDN and cluster:cluster MOID

Same above operations can be done from H5C UI ( Menu –> “Workload Management” )
I hope, at this stage you got basic idea on where to start exploring these APIs. In my next couple of posts, I will have deep dive into critical APIs to manage “Supervisor cluster” on given cluster i.e. “enable/disable/update” Ops. For now lets invoke one of the basic API calls to see something cool in action.

In my lab environment, I have 2 clusters in my vCenter server (in VCF term, 1 workload domain with 2 clusters), where “Supervisor cluster” is enabled on one and not enabled on second cluster. I chose to invoke an API, which gives us information on whether clusters in my vCenter inventory are compatible for enabling Supervisor cluster or not. As you know from my earlier posts this or this, this must be a GET call as follows


https://{server}/api/vcenter/namespace-management/cluster-compatibility


This is how you invoke it quickly from native API Explorer from within vSphere client (H5C)

https://vthinkbeyondvm.com/wp-content/uploads/2020/04/Cluster-compat.png

Below is what I got the response, pasted as it is

Response

[vcenter.namespace_management.cluster_compatibility.summary   {

},vcenter.namespace_management.cluster_compatibility.summary   {

  • “cluster”: “domain-c9”,
  • “compatible”: “true”,
  • “incompatibility_reasons”: [ ][]

},]

As you see, my first cluster (moid: domain-c19) is not compatible and API gives us nice info on what are the basic requirements not met such as HA is not enabled, DRS is either not enabled or not in fully automated mode and new NSX-T VDS (aka conversed VDS) is not configured as well. How cool is that! As expected second cluster (moid: domain-c19) is perfectly fine as Supervisor cluster is already enabled there.

I hope you enjoyed my first post on this amazing innovation. Here is my new post on how to manage Supervisor cluster using DCLI. Please stay tuned for my detailed post on its cool functional aspects. Also, here is the official documentation on “vSphere with K8s” for your further learning. I enjoyed providing my candid feedback on various aspect of this documentation last month, I am sure you will love reading it.

Once again I would like to remind you that “vSphere with Kubernetes” is available through VCF and once you have VCF workload domain ready, you would be re-directed to vCenter H5C to proceed for further ops on managing/enabling vSphere with Kubernetes.

Further reading:
1. REST API documentation & my new post on automating Supervisor cluster configuration or Automating supervisor cluster workflows using Java SDK
2. How to get vSphere with Kubernetes?
3. vSphere with Kubernetes 101 is here
4. Introduction to manage vSphere Supervisor Cluster
5. Official documentation for vSphere with Kubernetes is here


Hybrid Linked Mode (HLM) with vCenter Cloud Gateway using REST API

In my first post on HLM configuration using API, I explained how to configure HLM using API through Cloud VC. If you haven’t still read that post, I would highly recommend you do. In this post I am going to touch upon how to configure HLM using API but from vCenter cloud gateway appliance (VCGA).

Setting up HLM from vCenter cloud gateway is three step process


1. Deploying vCenter cloud gateway appliance using official documentation

2. Making sure prerequisite for linking with Cloud Gateway Appliance are met

3. Configuring HLM from UI or using API. In our case, we would do from API

As I mentioned in my last post, API itself does not change, it is just that API end-point will be vCenter cloud gateway appliance and input parameters we pass will be changed. Lets take a look.

POST: https://<VCG IP>/rest/hvc/links

VCG IP is the IP address of the deployed vCenter cloud gateway appliance

Below is the JSON payload this API takes in . Note thumprint is optional param

{
“spec”: {
“port”: “443”,
“domain_name”: “vmc.local”,
“username”: “cloudadmin@vmc.local”,
“ssl_thumbprint”: “E9:BD:29:9F:D6:15:4F:B7:C8:90:2C:19:03:61:HB:7A:AD:FB:B1:1E”,
“admin_groups”: [
” yourcloudadmingroup@yourdomain.local “
],
“password”: “VMware123!”,
“psc_hostname”: “Cloud PSC IP or FQDN”
}
}

Let us go over each parameter passed

port: 443 is default  port for communication between VCG and Cloud VC (can be custom port also)

domain_name: Cloud VC PSC default SSO domain (its vmc.local)

username, password, psc_hostname: Cloud VC PSC credentials (PSC is always embedded in case of VMware cloud on AWS deployment)

ssl_thumbprint : Note that this parameter is optional, it will work fine even if you do not pass this param or if you pass value as blank. If you want to pass, this is Cloud PSC SSL thumbprint, you can get it using one of the ways posted here . I am going to write a post on how to get vCenter/PSC thumbprint using API itself, please stay tuned.

admin_groups: here you specify cloud administrator group(s).  Before configuring HLM linking , identity source must be configured on Onprem and cloud admin group(s) must be given global permissions. Note that identity source need not be configured on Cloud VC, refer this KB

Note: Above API can be called from H5C API-explorer or Postman/any REST client or python script as specified below

I used the same python script used for HLM linking with Cloud VC but using VCG IP as end-point & passing inputs as described above. I see it was successful as expected & I was able to see both VCs together when logged in as user from cloudadmin group configured at HLM linking.


How to schedule VM power cycle when the guest OS reboots?

Couple of weeks back vSphere 6.7 U3 released vCenter release notes & ESXi release notes and once again an update release has got great content. Apart from several other changes gone into vSphere 6.7 U3, one of the utilities I found really useful. i.e. Ability to schedule the VM power cycle when the guest OS reboots. This post is focused on the same. Before we dig into, below is what ESXi release note says about it.

PR 2394247: You cannot set virtual machines to power cycle when the guest OS reboots

After a microcode update, sometimes it is necessary to re-enumerate the CPUID for virtual machines on an ESXi server. By using the configuration parameter vmx.reboot.powerCycle = TRUE you can schedule virtual machines for power-cycle when necessary.

This issue is resolved in this release.

Based on above description, it is clear that one of the motivations behind this cool utility is the recent meltdown/spectre patches. After applying each of these patches, it was required to do cold power cycle of the VMs in order to re-enumerate the CPUID(s) introduced by that particular patch. Performing cold power cycle (power off and power on) is always tricky to plan and painful activity. This utility enables an user to schedule power cycle as part of guest OS reboots itself. How cool is that!

In order to simulate a scenario where we should see CPUIDs are re-enumerated on a VM, I created a cluster with 2 hosts, where both hosts Max EVC mode supported was “Broadwell” but I enabled EVC on “ivy-bridge” EVC mode. Then I created a GOS VM, powered on it and used below powerCLI script to add ” vmx.reboot.powerCycle = TRUE ” into vmx file.

[powershell] 
 #PowerCLI script to add vmx entry "vmx.reboot.powerCycle"
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
Connect-VIServer -Server 10.1.1.1 -User Administrator@vsphere.local -Password VMware@123 #replace your server (VC or ESXi) name

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.extraConfig += New-Object VMware.Vim.OptionValue
$spec.extraConfig[0].key = "vmx.reboot.powerCycle"
$spec.extraConfig[0].value = "TRUE"

(get-view (Get-VM -Name VMW).ID).ReconfigVM_Task($spec)  #replace your vm name

Disconnect-VIServer -Server 10.1.1.1 -Confirm:$false #replace your server (VC or ESXi) name 
 [/powershell] 

You can run above script against either vCenter or ESXi as endpoint. User also can add above vmx entry using ESXi host client as shown below.

Add parameter into vmx file

After running above script against my VM, I see that vmx file has got ” vmx.reboot.powerCycle = TRUE ” entry . Before I restart the GOS, I checked (from MOB) CPUIDs available on this VM at ivy-bridge EVC mode and below is how it looks.

CPUIDs at ivy-bridge EVC mode
VM cpuids at ivy-bridge EVC mode

In continuation to simulate the scenario, I disabled EVC on the cluster,where max EVC mode supported is “Broadwell” i.e. ideally VMs should be running on Broadwell cpu set but since we had created our VM when EVC was enabled on “ivy-bridge”, it will continue to use “ivy-bridge” cpu set/features even though underlying host is “Broadwell”. Before 6.7 U3, we had to do cold power cycle (power off, power on) to have VM on “Broadwell” cpu set. Since we added vmx entry already, we need not to do power cycle but guest reboot in-turn will do power cycle as well and CPUIDs will be re-enumerated. Below are the CPUIDs I see on VM when I restarted the Guest OS as it is now using broadwell cpu instruction set.

VM cpuids re-enumerated to broadwell

Key observations.
1. I see vmx entry works only when it is added on the Powered ON VM.
2. When I added vmx entry onto PowerOff VM, it got added but as soon as I powered on, it got removed. It will persist during suspend and resume.
3. I observed that this setting is one time i.e. when user restarts the Guest OS, I do not see vmx entry i.e. vmx.reboot.powerCycle = TRUE any more. Hence next time you want to have similar functionality, you need to set it once again on powered ON VM.
4. This utility is just not only for cpuid re-enumeration. cpuid re-enumeration is one of the use-cases. I think it is great idea to schedule the power Cycle when you upgrade VM hardware version or upgrade vmware tools.

I hope enjoyed the post. Let me know if you have any comment.

How to configure Hybrid Linked Mode (HLM) using vCenter REST API

This is going to be my first 2019 post and in this post, I am excited to share couple of ways to configure Hybrid Linked Mode (HLM) using REST API. One of the ways is using UI REST client i.e. apiexplorer and another way is using python. If you are not already familiar with vCenter server REST APIs, I would suggest to first go over my earlier post i.e. Getting started with vCenter REST APIs.
Before I move further, I would like to point out that I am not explaining end to end steps required to configure HLM since Emad already has a detailed post on it here. My focus is on vCenter REST API required to link your Onprem vCenter server to VMware cloud on AWS (Cloud vCenter).

Notes:
1. As you might know, there are 2 ways from which user can configure HLM. One way is to configure HLM from Cloud vCenter,where you can see both VCs (onprem and cloud VC) from cloud vCenter vSphere client (H5C) and another way is to configure HLM using vCenter cloud gateway,which allows users to see both VCs  from vCenter cloud gateway H5C. Here is my new post on configuring HLM through vCenter cloud gateway using API

2. Just to avoid any confusion: Since this HLM linking operation is done from vCenter server or vCenter cloud gateway, REST API also comes from vCenter itself and not from CSP (Cloud services platform APIs, which are about Cloud services console operations)

3. Great news  is that API remains exactly same no matter which way you go for. Only change is: If you are configuring HLM from cloud vCenter (i.e. VMC VC), API end-point would be VMC VC IP/FQDN and if you are configuring HLM using vCenter cloud gateway, API end-point would be vCenter cloud gateway.

How to do HLM linking from H5C UI

You could see, in order to link, we need to pass PSC details and cloud administrator group(s). This is exactly we will do using REST API.

HLM APIs overview

Below are the REST APIs available around HLM. We are more interested in POST /hvc/links call as this is the key API for configuring HLM no matter its from VMC VC or vCenter cloud gateway.

Invoking from UI REST client i.e. APIExplorer

By now you might have already used apiexplorer (swagger based) vCenter REST client. To access it, just browse “https://[vCenter IP]/apiexplorer”. If you see above screen-shot, there are multiple APIs on HLM operations. The one we are interested in is “/hvc/links” POST method, which is responsible for HLM linking.

As part of above request body, we need to pass exactly the same details as we passed while configuring from UI. Let us take a look the “request_body” spec I passed.

{
“spec”: {
“port”: “443”,
“domain_name”: “vsphere.local”,
“username”: “Administrator@vsphere.local”,
“ssl_thumbprint”: “F9:1C:2B:E7:C5:A0:CC:02:D3:37:33:04:B0:2D:2F:6C:77:50:EB:9C”,
“admin_groups”: [
“yourcloudadmingroup@yourdomain.local”
],
“password”: “VMW!23,
“psc_hostname”: “10.161.2.5”
}
}

Let us go over each parameter passed.

port: 443 is default  Onprem PSC TCP port (can be custom port also)

domain_name: Onprem PSC default SSO domain

username, password, psc_hostname: Onprem PSC credentials

ssl_thumbprint :
If you need to pass, this is Onprem PSC SSL thumbprint, you can get it using one of the ways posted here . I am going to write a post on how to get vCenter/PSC thumbprint using API itself, please stay tuned.

admin_groups: here you specify cloud administrator group(s).  Before configuring HLM linking (apart from HLM standard requirements that Emad has posted here) , identity source must be configured on both Onprem and VMC first and cloud admin groups must be given global permissions on both VCs.

HLM linking using python

By this time, I am sure you know everything about the API, now lets head on to python way of doing these things.

This script is available on my github repo HERE

[python]
# Author: Vikas Shitole
# Website: www.vThinkBeyondVM.com
# Product: VMware Cloud on AWS (VMC)
# Description: Python script to configure Hybrid Linked Mode (HLM) between Onprem and VMC VC
# How to setup vCenter REST API environment?: https://vthinkbeyondvm.com/getting-started-with-vcenter-server-rest-apis-using-python/</pre>
<pre>import requests
import json
import ssl
import atexit
import sys
import argparse
import getpass

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

s=requests.Session()
s.verify=False

def get_args():
    """ Get arguments from CLI """
    parser = argparse.ArgumentParser(
        description='Arguments for HLM linking')

    parser.add_argument('-s', '--host',
                        required=True,
                        action='store',
                        help='VMC VC IP or FQDN')

    parser.add_argument('-u', '--user',
                        required=True,
                        action='store',
                        help='VMC VC username')

    parser.add_argument('-p', '--password',
                        required=False,
                        action='store',
                        help='VMC VC password:')

    parser.add_argument('-o', '--port',
                        type=int,
                        default=443,
                        action='store',
                        help='PSC port')

    parser.add_argument('-d', '--domainname',
                        type=str,
			default='vsphere.local',
                        action='store',
                        help='Onprem PSC domain name')

    parser.add_argument('-pu', '--pscuser',
                        required=False,
			default='Administrator@vsphere.local',
                        action='store',
                        help='Onprem PSC username')

    parser.add_argument('-pp', '--pscpass',
                        required=False,
                        action='store',
                        help='Onprem PSC password')

    parser.add_argument('-ph', '--pschost',
                        required=True,
                        action='store',
                        help='Onprem PSC host IP or FQDN')

    parser.add_argument('-a', '--admingroup',
                        required=False,
                        action='store',
                        default='yourcloudadmin@yourdomain.local',
                        help='Cloud admins group')	

    parser.add_argument('-pt', '--pscthumb',
                        required=True,
                        action='store',
                        help='Onprem PSC thumbprint')

    args = parser.parse_args()

    if not args.password:
        args.password = getpass.getpass(
            prompt='Enter VMC VC password:')

    if not args.pscpass:
        args.pscpass = getpass.getpass(
            prompt='Enter PSC password:')

    return args

args = get_args()
headers = {'content-type':'application/json'}
session_response= s.post('https://'+args.host+'/rest/com/vmware/cis/session',auth=(args.user,args.password))

if session_response.ok:
	print ("Session creation is successful")
else:
	print ("Session creation is failed, please check")
	quit()

payload = {
  "spec": {
    "port": args.port,
    "domain_name": args.domainname,
    "username": args.pscuser,
    "ssl_thumbprint": args.pscthumb,
    "admin_groups": [
      args.admingroup
    ],
    "password": args.pscpass,
    "psc_hostname": args.pschost
  }
}

json_payload = json.loads(json.dumps(payload))
json_response = s.post('https://'+args.host+'/rest/hvc/links',headers=headers,json=json_payload)
if json_response.ok:
	print ("HLM link is established")
else:
	print ("HLM link is NOT established, please check")
print (json_response)
print (json_response.text)

[/python]

Below is how you would run the script, take a look at the what parameters need to be passed. Each parameter is explained in above script as well.

I hope you enjoyed reading this post. Here is my new post on configuring HLM through vCenter cloud gateway using API.