A customer on VCF 9.1.1, with vSphere Kubernetes Service running five guest clusters off one Supervisor: dev, QA, production and two shared-services clusters. The question arrived as an aside, in the middle of a host build on the same site: random oddball one from the team, where does this cert live, you think? An ESXi host’s Spherelet was logging a refused certificate request every thirty seconds, and somebody wanted to know which certificate that even was.
The answer to the aside was small. The answer to the question underneath it was five clusters with no API server. kubectl get cluster -A on the Supervisor read READY False for every one of them, kubectl against any of them was connection refused on port 6443, and it was not that the API servers had gone down. They had never come back up: etcd had no quorum, because every control plane node’s certificates had expired. This is where those certificates live, why they expired on a date nobody had written down, why bringing them back is not the same as renewing them, and the script that did it.
The Date Nobody Was Watching
TKG node certificates are valid for one year from the node’s creation date, not the cluster’s. A cluster built eighteen months ago whose control plane nodes were last rolled thirteen months ago has expired certificates whatever the cluster’s own metadata says, and the cluster’s metadata is what everyone looks at. The field that would have said so is Machine.status.certificatesExpiryDate, and nothing watches it by default.
kubectl get machine -A -o custom-columns=NS:.metadata.namespace,CLUSTER:.spec.clusterName,NAME:.metadata.name,CREATED:.metadata.creationTimestamp
NS CLUSTER NAME CREATED
dev tkgs-dev tkgs-dev-control-plane-9jmcc 2025-10-23T14:04:12Z
dev tkgs-dev tkgs-dev-control-plane-fhxxr 2025-10-23T14:07:32Z
dev tkgs-dev tkgs-dev-control-plane-tnjhn 2025-10-23T13:59:43Z
prod tkgs-prod tkgs-prod-control-plane-49d9c 2025-11-13T00:52:49Z
prod tkgs-prod tkgs-prod-control-plane-ctsdk 2025-11-13T00:44:55Z
prod tkgs-prod tkgs-prod-control-plane-tflnv 2025-11-13T00:49:54Z
qa tkgs-qa tkgs-qa-control-plane-k5hcr 2025-10-23T14:49:58Z
qa tkgs-qa tkgs-qa-control-plane-lvnmb 2025-10-23T14:46:40Z
qa tkgs-qa tkgs-qa-control-plane-xtqq5 2025-10-23T14:43:52Z
shared-services shared shared-75mn2-5kvm2 2025-10-23T12:51:21Z
shared-services shared shared-75mn2-bxzmq 2026-07-25T12:47:51Z
shared-services shared shared-75mn2-lp8zj 2026-07-25T12:53:29Z
shared-services shared shared-75mn2-wp5wh 2026-07-25T12:50:37Z
shared-services tkgs-shared-services tkgs-shared-services-control-plane-48tb4 2025-02-19T14:44:39Z
shared-services tkgs-shared-services tkgs-shared-services-control-plane-ctdwz 2025-02-19T14:42:23Z
shared-services tkgs-shared-services tkgs-shared-services-control-plane-dcr5g 2025-02-19T14:49:49ZThe day this was run was 23 October 2026. Read the CREATED column again. The dev and QA control planes were created on 23 October 2025 between 13:59 and 14:50, and their certificates expired that morning between 14:01 and 14:51, to the minute, one year on. The older node of the shared cluster expired at 12:53. Production was created on 13 November and had three weeks left. The second shared-services cluster had expired in February and nobody had noticed, because nothing on it was being used. Five clusters, one Supervisor, and the only thing they had in common was a rollout date.
Renewal, in the Cluster API world these clusters live in, is not renewal. Cluster API does not rotate a certificate in place: it replaces the node before the certificate expires, when told to. On a VKS cluster the switch is the certificateRotation block of the kubernetes cluster variable, enabled: true and renewalDaysBeforeExpiry: 90, which lands on the KubeadmControlPlane as rolloutBefore.certificatesExpiryDays. The current ClusterClass defaults it on; these clusters were built on an earlier one, had it unset, and had not been upgraded in a year, which is the other thing that would have rotated them.
Where the Certificates Live
Back to the aside, because the question was a good one. There are three layers of certificates in a Supervisor estate, and the symptoms of each look alike from a distance.
| Layer | Where | Rotated by |
|---|---|---|
| Supervisor control plane VMs | /etc/kubernetes/pki and /etc/vmware/wcp/tls on each Supervisor control plane VM |
certmgr certificates rotate from the vCenter (KB 322994), or the automatic renewal that starts 180 days out |
| ESXi hosts, the Spherelet | /etc/vmware/spherelet/client.crt, the bootstrap certificate, and kubelet-client-current.pem, the one in use |
the Spherelet itself, by certificate signing request to the Supervisor |
| Guest cluster control plane nodes | /etc/kubernetes/pki on each node |
a node rollout, when certificateRotation is on; otherwise kubeadm certs renew all plus a restart of the four static pods, then certmgr tkc certificates rotate for the rest (KB 323453) |
The Supervisor’s own certmgr certificates list showed a healthy first layer: every certificate on the three control plane VMs in date to 2027 or later. The Spherelet on the hosts was the aside. The bootstrap certificate on every host had expired in April, which is expected once a host has rotated onto its own; the current certificate was in date on every host; and the Unauthorized the hosts were logging was the Supervisor declining a signing request, not a host’s certificate expiring. A dozen-line loop over the hosts printing both dates, the Spherelet state, its last stop and its last refusal is in the repository as spherelet_cert_check.sh; it answered the aside in two minutes: which file, which date, and that the hosts were being refused rather than expiring. The third layer was the outage.
The Trap for Anyone Reading the Dates
On a control plane node, kubeadm certs check-expiration reports:
kubeConfig file /etc/kubernetes/scheduler.conf does not have an embedded client certificateThat looks like the problem and is not. It is normal for a kubeadm cluster using a CA-signed kubelet configuration, and anyone who stops there goes looking for a missing certificate that was never supposed to be there. The expiry that matters is in the serving certificates, and the fastest honest read is openssl x509 -noout -enddate in a loop over /etc/kubernetes/pki/*.crt. Once renewed, the same command reads cleanly to October 2027 on every line; the MISSING! super-admin.conf it prints on these clusters is a kubeadm version expecting a file that older clusters never had, and is harmless.
Recovery Is Not Renewal
The mechanics of one node are short: kubeadm certs renew all, then stop the four static pods, kube-scheduler, kube-controller-manager, kube-apiserver and etcd, with crictl, and let the kubelet bring them back on the new certificates. Three things make it more than that on a cluster that is already down.
The nodes have to be done in series, and a node that does not come back is not a reason to stop. On a healthy cluster it would be: you stop, you look, you protect the quorum you have. On a cluster whose certificates have already expired the opposite holds. There is no quorum to protect, no node’s API server can start until enough etcd members are back in date, and stopping leaves the cluster exactly as down as it was. So the script waits four minutes for a node’s four containers, logs where it got to, and goes on to the next node, which is what restores quorum; a second run afterwards settles whatever was left. Restarting every etcd at once is the one thing that turns this into a restore from backup.
kubeadm can hand you a valid certificate no client will accept. kubeadm certs renew all reads kubeadm-config from the cluster to learn the SANs the API server certificate needs. The cluster’s API is down, so it cannot, and it says so and falls back to defaults:
[renew] Error reading configuration from the Cluster. Falling back to default configurationA default SAN set that has lost the control plane endpoint is a correctly signed certificate for the wrong names, and a node restarted onto it is worse than a node with an expired one. The script reads the API server certificate’s SANs before and after the renewal, and a node whose SANs changed is not restarted: the running API server still serves the old certificate, the node is no worse than before, and the fix is a renewal with an explicit kubeadm configuration carrying the right certSANs and controlPlaneEndpoint. On these clusters the SANs came through unchanged every time, and the script said so for every node.
The state of a node is not its expiry date. A node can have a renewed file on disk and an API server still serving the old certificate, because somebody renewed by hand and never restarted. Comparing dates misses that. The script reads the serial of /etc/kubernetes/pki/apiserver.crt and the serial of what port 6443 actually serves, and places every node in one of four states before it touches anything:
| State | Meaning | What --commit does |
|---|---|---|
ready |
the file is in date and the API server serves that same certificate | nothing |
restart |
the file is in date but the API server serves something else, or nothing | only the restart |
renew |
the file is expired, or inside the window (30 days by default) | renew, then restart |
problem |
the node could not be read | nothing, and the run stops before the commit |
A plan with any problem node refuses to go further, because a partly done control plane is worse than one that is uniformly out of date. That is what the second shared-services cluster read, and it was left for another day rather than half-done.
What the Run Looked Like
Everything runs from a Supervisor control plane node: the cluster’s SSH key is its <cluster>-ssh secret in its namespace, the login is vmware-system-user, and the node addresses come from the Machine objects, or from VSphereMachine and VirtualMachine when the Machines carry none, which at this site they did not. One pass over everything first:
sh tkc_cert_renew.sh --all
################################################## dev/tkgs-dev
cluster: dev/tkgs-dev ssh key: secret tkgs-dev-ssh
renew a certificate with fewer than 30 days left
mode: plan - nothing is changed
NODE ADDRESS STATE APISERVER CERT EXPIRES
tkgs-dev-control-plane-9jmcc 10.244.0.101 restart Oct 23 16:04:27 2027 GMT
tkgs-dev-control-plane-fhxxr 10.244.0.102 renew Oct 23 14:09:01 2026 GMT
tkgs-dev-control-plane-tnjhn 10.244.0.98 renew Oct 23 14:01:12 2026 GMT
3 node(s) need work. Re-run with --commit.
...
################################################## shared-services/tkgs-shared-services
NODE ADDRESS STATE APISERVER CERT EXPIRES
tkgs-shared-services-control-plane-48tb4 10.244.0.85 problem unreadable
tkgs-shared-services-control-plane-ctdwz 10.244.0.82 problem unreadable
tkgs-shared-services-control-plane-dcr5g 10.244.0.87 problem unreadable
3 node(s) could not be read. Fix that before going further - a partly
done control plane is worse than one that is uniformly out of date.The first dev node reads restart because it had been renewed by hand earlier that afternoon, while the script was being written, and never restarted. The plan caught it. Then production, committed:
sh tkc_cert_renew.sh -n prod tkgs-prod --commit
...
==== tkgs-prod-control-plane-49d9c (10.244.0.138) renew
renewed
SANs unchanged
stopped kube-scheduler
stopped kube-controller-manager
stopped kube-apiserver
stopped etcd
all four running again
==== tkgs-prod-control-plane-ctsdk (10.244.0.136) renew
...
==== after
NODE STATE APISERVER CERT EXPIRES
tkgs-prod-control-plane-49d9c ready Oct 23 16:48:47 2027 GMT
tkgs-prod-control-plane-ctsdk ready Oct 23 16:48:52 2027 GMT
tkgs-prod-control-plane-tflnv ready Oct 23 16:49:07 2027 GMT
etcd membership:
575d9dcddef90201, started, tkgs-prod-control-plane-tflnv, https://10.244.0.137:2380, ...
98ab7ac322968fec, started, tkgs-prod-control-plane-ctsdk, https://10.244.0.136:2380, ...
c83923048b0e78fb, started, tkgs-prod-control-plane-49d9c, https://10.244.0.138:2380, ...
Any member here that is not one of the nodes above is gone and still counts
towards quorum. Remove it with etcdctl member remove <id> - by hand, after
checking the id twice.
certmgr is not on this machine, so the certificates kubeadm does not cover
have not been rotated. Install it from KB 323453 and run:
certmgr tkc certificates rotate -n prod tkgs-prodDev, QA and production were back by the end of the afternoon, every control plane certificate dated October 2027. The etcd member list is printed on purpose and acted on by hand on purpose: a member that no longer exists still counts towards quorum, and nothing automated should be removing etcd members from a cluster that was down an hour ago.
The last block is the honest part. kubeadm does not rotate everything a guest cluster has; certmgr tkc certificates rotate does the rest, and KB 323453 requires the cluster’s API and etcd to be healthy before it starts, which an expired cluster is not. So the order is this script first, certmgr second, and I had sent the customer to run certmgr on a Supervisor it was not installed on. The script now checks for the binary and prints the install step instead of assuming.
Still Open, and Said So
The Supervisor’s own vip.crt had expired in September and its pinniped.crt a year before that, both on the first layer, both with their own procedure in KB 322994, and both outside this script on purpose. The certificate rotation variable is still unset on every cluster; turning it on is a cluster spec change and a rollout, not an incident-day change. And the check that costs nothing and would have caught all of this a month out:
kubectl get machine -A -o custom-columns=NS:.metadata.namespace,CLUSTER:.spec.clusterName,NAME:.metadata.name,EXPIRES:.status.certificatesExpiryDateThree Bugs Worth Recording, Because They Are Generic
ssheats stdin in awhile readloop. A loop over hosts runs once and exits.ssh -n, and</dev/nullon the loop for good measure. This one costs an hour every time somebody meets it, and it cost one here.- Nested heredoc delimiter collision. A payload heredoc inside a heredoc with the same terminator. Different terminators, always.
- A loop variable reused across scopes.
$was both a column width and a container name turned aprintfwidth into scientific notation.%-etcdsis a memorable thing to see in output.
Lessons Learned
- The expiry is a year from the node, not the cluster. Watch
Machine.status.certificatesExpiryDate, or turncertificateRotationon and let the control plane roll itself. - Read the state, not the date. A renewed file and a served certificate are two different facts, and the serial is how you compare them.
- No quorum changes the rules. In series, carry on past a node that is slow, and never restart every etcd at once.
- Check the SANs across a renewal. kubeadm falling back to defaults is a line in the output, not an error, and the certificate it produces is valid.
- Name the next command, and check the tool exists before naming it.
Getting It
git clone https://github.com/noahfarshad/tkc-cert-renew.git
cd tkc-cert-renew
sh tkc_cert_renew.sh selftest # the decision logic, no cluster needed
scp tkc_cert_renew.sh root@<supervisor-control-plane>:
sh tkc_cert_renew.sh --all # on the Supervisor: plan, every cluster
sh tkc_cert_renew.sh -n prod tkgs-prod --commit
certmgr tkc certificates rotate -n prod tkgs-prod # afterwards, once it is installed--days widens the window, --nodes name=ip,... covers Machines that carry no address, and spherelet_cert_check.sh is the loop for the hosts. Nothing in the files is a site value.
Conclusion
Five clusters went down on the anniversary of a rollout nobody remembered, and the fix was not the renewal command. It was knowing which of three layers the certificate lived in, reading what each node was actually serving, doing the nodes in an order that rebuilds quorum instead of waiting for it, and refusing to restart onto a certificate kubeadm had quietly built from defaults. Then the one block of YAML that means it does not happen again.
References
- Replace VKS Cluster (Previously known as vSphere with Tanzu Guest Cluster) Certificates, Broadcom KB 323453:
certmgr tkc certificates listandrotate, run from a Supervisor control plane node or a jump box, and only against a cluster whose API and etcd are healthy. - Replace vSphere Supervisor Certificates, KB 322994: the Supervisor’s own certificates, and where
wcp_cert_manager.zipcomes from. - Master vSphere Supervisor Certificate Guide, KB 323421: which certificates exist, who signs them, and their lifetimes.
- Automatically rotating certificates using Kubeadm Control Plane provider:
rolloutBefore.certificatesExpiryDaysandMachine.status.certificatesExpiryDate. - VKS ClusterClass Variable Reference: the
kubernetesvariable’scertificateRotationblock.
Repository: github.com/noahfarshad/tkc-cert-renew (release 2.3.0)
Related Stories:
- Everything That Cost Us a Day Returned Success — the host build this interrupted, on the same site and the same day
- When the NSX Manager Certificate Is Already Expired — the same shape one layer down the stack: a certificate past its date and an API that will not answer until it is replaced
- When Regional Harbor Sticks, VKS Stops — the Supervisor services the guest clusters depend on
- Air-Gapped VKS on VCF 9: Make the Inside Boring — the registry the clusters pull from