Tag Archives: vSphere with Kubernetes

How to gracefully remove host from NSX-T based Supervisor Cluster?

HOME, Vmware private AI and IaaS platform, , ,

Recently got this question couple of times on how to gracefully remove host from existing Supervisor cluster, which is configured with NSX-T. I thought it is worth to write a quick post with detailed steps. Removal of the host from existing Supervisor cluster can be for multiple reasons, it can be for some maintenance or adding host into other supervisor cluster etc.

If you are still not aware of what is this Supervisor cluster, I would highly recommend you read this blog post.

Below steps are written assuming you have NSX-T based Supervisor cluster configured. In NSX-T based supervisor cluster, ESXi hosts are also working as kubernetes worker nodes, hence it is important they are properly removed from cluster. In new 70 U1 capability vSphere with Tanzu with vSphere (VDS) networking, ESXi hosts are no more worker nodes as there it is all about Tanzu kubernetes clusters (aka Guest clusters), hence removing host from VDS based Supervisor cluster is same as earlier.

Steps :

  1. Identify the host you would like to remove.
  2. Put the host in maintenance mode
  3. Since host is now going into maintenance mode, all the vSphere pods running on this node/host will be re-created on the other available hosts (DRS will take care of recommending proper host).
  4. VMs running on this host such as Supervisor control plane VMs or Guest Cluster VMs or any other normal VMs will be migrated to other available hosts as usual.
  5. Once host goes into maintenance mode, spherelet service on the host will be stopped but spherelet vib pushed to ESXi host will be still there. you can check spherelet vib and service status using below commands. SSH to host for running below commands.
  6. “esxcli software vib list | grep spherelet” & “/etc/init.d/spherelet status”
  7. When host goes into maintenance mode, in kubernetes term, it is called node is drained. i.e. this host is no more ready for taking any k8s workloads. When you run “kubectl get nodes”, it should be shown as “not ready”.
  8. NSX-T host transport node will be still as it is in configured state. You can confirm it from NSX-T UI or you can also check “nsx” vibs on host in maintenance mode.
  9. Now instead of removing this directly from inventory, move this host as standalone host into datacenter.
  10. As soon as you move this host as standalone host, spherelet vib on the host will be removed as well i.e. no spherelet service will be on it as well.
  11. Also, NSX-T host transport node will be un-configured as well automatically. i.e. All NSX-T vibs are removed at this stage. You can check it from NSX-T UI, it should be shown as “Not Configured”. This happens because while configuring NSX-T, we had applied host transport node profile on vSphere cluster level.
  12. “kubectl get nodes” or H5C UI (Cluster >> Monitor >> Namespaces >> Overview) should not show this host as worker node.
  13. Only reference now pending is, this host is still part of Distributed switch (DVS/VDS) you configured as part of Supervisor Cluster. You can remove it from usual DVS UI workflow.
  14. At this stage, this host is completely free to go for maintenance or add into another Supervisor cluster. You can remove it now from vCenter server inventory as needed.
  15. To add host back into same Supervisor cluster again, it is better to have host in maintenance mode, make sure you add this host back to DVS/VDS configured. Move this host into cluster, exit from maintenance mode and it will automatically install spherelet and NSX-T vibs and after few min, it will be ready for taking k8s workloads.

Further reading:
1. vSphere with Kubernetes (now vSphere with Tanzu) 101 is here
2. Official documentation for vSphere with Kubernetes is here
3. Automation around Supervisor cluster
4. Read about new 70 U1 capability vSphere with Tanzu with vSphere (VDS) networking
5. Automating supervisor cluster workflows using Java

Introduction to managing vSphere Supervisor Cluster using DCLI

HOME, Vmware private AI and IaaS platform, vSphere API,

My last post was about introduction to vSphere Supervisor Cluster Kubernetes REST APIs, where I briefed about key constructs such as “vSphere Supervisor Cluster” and “Tanzu Kubernetes cluster” but major focus was to introduce REST APIs around it and how to execute these APIs using H5C built-in API Explorer. As per me, it is must to read that post before you proceed. This post is about how to manage vSphere Supervisor Cluster using DCLI (Datacenter CLI). As you might already know that DCLI is a CLI exclusively for REST APIs and it supports vSphere Automation APIs (REST APIs for vSphere), VMware cloud on AWS and even NSX as well. Beauty of the DCLI is that it comes by-default as part of VCSA installation though we can install it in other Guest OSes also.

As I mentioned in the last post , “vSphere Supervisor Cluster” REST APIs are exposed by vCenter server i.e. every DCLI command we execute would be executed against vCenter server, where vSphere Supervisor cluster is configured or going to be configured. Let’s dive into it. In my case, I am going to access DCLI that comes automatically with VCSA.

First step is to SSH VCSA 7.0 and run below command

dcli +interactive +server 10.20.30.40 +skip-server-verification

We need to pass vCenter server IP as a server. I will be running DCLI in interactive mode so that we get automatic help on available options with press of space or it gets showed as you type. This is very useful if you do not know where to start for new APIs as this.




We are now in DCLI console. As I showed in my last post, these REST APIs are identified as “namespace management” & “namespaces”. Since these are “vcenter” APIs, when we just type “vcenter” and press “space”, you will get auto-populated list of options available as shown below, is not that cool ?

As in my last post, let’s check the compatibility of the cluster to confirm whether key requirements are met or not on given cluster

dcli> vcenter namespacemanagement clustercompatibility list

You can see cluster “domain-c9” (its moid) is compatible but cluster “cluster-c19” does not meet any basic requirements.

Below are few more basic commands (mostly GET calls)

Command #1: Disabling “vSphere with K8s” on given cluster

dcli> vcenter namespacemanagement clusters disable --cluster <cluster moid>

cluster moid/identifier looks like “domain-cx”. You can get it from UI (click on cluster and just check URL in browser or REST API/SOAP/MOB can help as well)

Command #2: Know compatible VDS on given cluster

dcli> vcenter namespacemanagement distributedswitchcompatibility list --cluster domain-c9

Command #3: Get compatible NSX-T edge cluster on given cluster and VDS

dcli> vcenter namespacemanagement edgeclustercompatibility list --cluster domain-c9 --distributed-switch "50 13 e2 3b 7d 5e 59 4b-63 1c c5 d9 ab 25 cd 7e"

VDS uuid you get from command #2 above.

Command #4: Get list of namespaces available

dcli> vcenter namespaces instances list

Command #5: Get available Kubernetes versions

dcli> vcenter namespacemanagement clusteravailableversions list
or
dcli> vcenter namespacemanagement software clusters get --cluster <cluster-identifier/moid>
or
dcli> vcenter namespacemanagement software clusters list

Command #6: Know what are sizes and resources requirements (TINY, SMALL, MEDIUM, LARGE )

dcli> vcenter namespacemanagement clustersizeinfo get

I hope this article was good head start for you on how to manage vSphere Supervisor cluster using DCLI. There are more advanced operations like enable, update, create (POST calls) etc that I will have deep dive into my future posts. Please stay tuned.

Further reading:
1. REST API documentation and my new post on automating Supervisor cluster configuration or Automating supervisor cluster workflows using Java SDK
2. How to get vSphere with Kubernetes?
3. vSphere with Kubernetes 101 is here
4. Introduction to vSphere Supervisor Cluster REST APIs
5. Official documentation for vSphere with Kubernetes is here