Regional Harbor is not a convenience registry. In VCF 9.1 it is the registry Supervisor services, VCF CLI plugins, and VKS add-ons pull from. When that registry is healthy, a cluster create is an ordinary operation. When it is stuck mid-removal, image pulls fail closed. The Kubernetes service on that Supervisor goes down with it. A hand-installed Harbor on one Supervisor does not stand in for it. That install has no connectivity to the software depot, and it cannot cover the same uses.
Why a Teardown Takes It
The change that exposed this was ordinary. Onboard a shared vCenter into the VM Apps organization, and put the All Apps side back the way it was. The region teardown did what a teardown does. It removed the Supervisor services that region owned. Regional Harbor was one of them.
Harbor did not finish leaving. The depot entry was still there. The service was not healthy. VKS on that Supervisor was down with it. The original request, VM Apps operational, was now blocked on a registry the UI still listed as something you could uninstall again.
Why the Retry Fails
A stuck uninstall is a worse state than a missing service. A second uninstall has nothing clean to remove. A fresh install is refused because the name is still claimed by the half-deleted object. The UI offers the operation that has already failed.
The object holding the name is not a namespaced leftover. It is a cluster-scoped registry entry, a ClusterDomainResolutionEntry for the depot, in the container registry API. A role binding scoped to kube-system does not reach it. That is why a cleanup that looks authorized from the namespace still cannot finish the delete, and why the finalizer keeps the name reserved.
The Recovery, in Order
Diagnosis starts on the Supervisor, not with another click in Service Management. Confirm the depot resolution entry is still present, and confirm the account that will remove it can actually delete that cluster-scoped object. If it cannot, the elevation is temporary, scoped to the registry API and the namespace objects the finalizer is holding, and bound to the platform administrators group. It comes out when the cleanup is done. It does not become a standing grant.
Clear the finalizer on the stuck depot entry, and confirm the resolution list is empty. The name has to be free before Harbor will accept an install. Then install through the documented path: Service Management, Available, Harbor, Install. Do not invent a second registry beside it.
The install has its own authorization gap. Creating the depot service and endpoints in kube-system can fail for the same administrator the UI is using, until create rights exist for that reconcile. Those rights are temporary too. In the recovery we ran, Harbor reconciled in about 15 minutes. The depot service, endpoint, and DNS came back on a valid address. The TKG package recovered with the depot. VKS came back. VM Apps was operational at the end of the same change, which was the request that started it.
Then delete the temporary role, the binding, and the cluster role. A cleanup that leaves delete rights on system resources is a second incident waiting for a quieter day.
Before the Next Teardown
A region teardown is not a reversible button once Supervisor services have finalizers. Before you tear a region down to attach a vCenter somewhere else, write down which Supervisor services that region owns, and which of them are shared with a Kubernetes service you intend to keep. Harbor is the one that hurts, because the platform does not degrade. It stops pulling images.
If Harbor is already stuck, do not loop the uninstall. Free the depot name, confirm the resolution entry is gone, and install. Keep the extra access in files you can delete.
On a disconnected 9.1.1 site that does not have VCF Automation, the air-gap procedure is the other half of this. Download and upload the Harbor Supervisor service image, and use the package YAML whose name does not contain legacy. That path is how a site without the automation product still gets a registry the Supervisor can pull from. It is not a substitute for regional Harbor where VCF Automation is installed. The air-gap kit story is where those older arrival paths are separated from the 9.1.1 depot path.
The Results
The depot name was freed, Harbor was installed through Service Management, and the reconcile recreated the depot service, endpoint, and DNS. The TKG package recovered with the depot. VKS came back. VM Apps was operational at the end of the same change. The temporary grants were removed.
Lessons Learned
A stuck uninstall is worse than a missing service. The UI offers the operation that already failed. The object holding the name is cluster-scoped, so a kube-system binding never reaches it. The install has its own authorization gap. Creating the depot service can fail for the same administrator the UI is using until create rights exist for that reconcile. Those rights are temporary.
Getting Started
Run it beside this, from a shell that already reaches the Supervisor. Confirm the object before you clear the finalizer. Delete the grants when the reconcile finishes.
kubectl get clusterdomainresolutionentries -A
kubectl auth can-i delete clusterdomainresolutionentries.containerregistry.vmware.com
kubectl apply -f temp-elevated-cleanup.yaml
kubectl patch clusterdomainresolutionentry depot.kube-system.svc --type=json \
-p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl delete clusterrolebinding temp-elevated-binding
kubectl delete clusterrole temp-elevatedClone vcf-harbor-recovery and follow the order in the README. Confirm depot.kube-system.svc is still present. Apply the cleanup grant only if the delete is denied. Clear the finalizer on that one object. Install from Service Management. Apply the install grant only if the reconcile cannot create the depot service. Delete both grants when the TKG package is back. Do not leave delete rights on the registry API.
Conclusion
Regional Harbor is not optional. When it sticks, image pulls fail closed and VKS stops. Free the name, install through the documented path, and take the extra access back out.
References
Installing and Configuring Harbor is the product line this story sits on. Regional Harbor, installed as a VCF service across regions in VCF Automation, is a mandatory registry for Supervisor services, VCF CLI plugins, and VKS add-ons. Harbor installed by hand as a Supervisor service on one Supervisor does not have connectivity with the software depot and cannot cover those uses. The disconnected procedure, including the non-legacy Harbor package, is Deploy VKS in Air-Gapped Environments with VCF 9.1.1 and Later.
The temporary grants, the finalizer clear, and the rollback are in the repository. Delete the grants when the reconcile finishes.
Repository: github.com/noahfarshad/vcf-harbor-recovery
Related Stories:
- Air-Gapped VKS on VCF 9 — the 9.1.1 depot path, and the Harbor package whose name does not contain legacy