A cluster refused to convert to a vLCM image. The eligibility check named a transport node profile, which made no sense on hosts stood up the week before. They had NSX 4.2.3 on them anyway, installed months earlier by a manager that no longer managed them. Nobody had put it there on purpose, and nothing in the UI said it was there. This is the middle post of three about importing ESXi 8 clusters into VCF 9.1: the one about getting stale NSX off hosts without taking their networking with it.
Three Layers
The NSX was in three layers, and the third one is the one that bites:
| Layer | How many | Visible to |
|---|---|---|
Host platform VIBs, nsx-* |
24 | esxcli software vib list |
vsipfwlib |
1 | vib list, but not named nsx-* |
| esxio platform VIBs | 23 | esxcli software profile get only |
esxcli software vib list does not show esxio-platform VIBs at all, so a host can look clean and have 23 of them. Those cost most of an afternoon at the second site, discovered one at a time, before the sweep below existed.
Two Things Worth Knowing First
nsxcli -c del nsx removes a transport node. On a host that was never a transport node it succeeds and removes nothing. It also asks three confirmations, each worded differently, the last with [Y/N] rather than (Y/N), and it needs a PTY. The tool matches all three against the tail of what the host has said, so the order does not matter; the bracket form is how the third one got missed once.
esxcli software vib remove is the wrong tool when an N-VDS exists. Broadcom calls it an incorrect uninstall method (KB 337062): it strands the switch and the host loses networking entirely. It is safe only once you have proven there is no N-VDS, no VTEP and nothing in net-dvs, which is exactly what the tool checks before it will go anywhere near it.
What the Tool Does
nsx_cleanup.py surveys and removes stale, unconfigured NSX from ESXi hosts, a cluster at a time, through vCenter and SSH. Per host, in order: survey, enter maintenance mode, del nsx, reboot, wait for the host to come back, verify no NSX VIBs remain, leave maintenance mode. One host at a time within a cluster, because vSAN. It stops at the first failure.
What it refuses to touch. A host with real NSX configuration. VTEP vmkernels, NSX properties on the distributed switch or an N-VDS mean the host was genuinely prepared, and unwinding that is the careful path in KB 451192, not a loop. The tool checks first and stops rather than guessing.
By version, not by name. Not every NSX VIB is called nsx-something; vsipfwlib is one, and nsx-proto2-libs depends on it, so a removal that skips it fails on a dependency error. What they do share is a version string. The tool finds the dominant version among the nsx-* VIBs and then claims everything at that version, including the esxio side, which it reads from esxcli software profile get rather than from vib list.
A self-correcting loop. Removal retries until the set is gone: a DependencyError names a VIB that has to come off too, so it is added; a NoMatchError names one the profile lists but the host no longer has, so it is dropped. The loop runs again with the corrected list.
The cheapest moment is before the host joins anything. --direct goes straight to the hosts over SSH, with no vCenter, for hosts that have not joined a cluster yet. --vib-remove is for the case del nsx leaves the VIBs in place, which it does on a host that has no transport node, and it is only ever reached on a host the tool has already confirmed has no N-VDS, no VTEP and nothing in net-dvs. --all-at-once takes every host in a cluster together instead of one at a time, and is refused if any host has a powered-on VM, because the whole cluster goes down.
Nothing is written to disk but the run log, and no password is ever logged.
Tests
Two suites drive the tool’s decisions against stub hosts, with no vCenter and no SSH. One covers the survey, the refusals and the per-host order; the other covers the esxio sweep, the self-correcting removal loop and direct mode, and every fixture in it is real output captured from four hosts on the day the esxio layer was found. 38 checks between them.
Lessons Learned
vib listis not the whole host. The esxio platform has its own VIBs, and only the software profile shows them.- Select by version. The one VIB that is not named like the others is the one the dependency chain needs removed first.
- Never
vib removepast an N-VDS. Prove the host has no transport node first; then, and only then, is the direct removal safe. - Clear stale NSX before the host joins anything. After that it is maintenance mode, a reboot and a vSAN evacuation per host.
- Read the prompts. Three confirmations, two bracket styles, one PTY. Automate it once, properly, and never type it again.
Getting It
git clone https://github.com/noahfarshad/nsx-host-cleanup.git
cd nsx-host-cleanup/tests && python3 test_nsx_cleanup.py && python3 test_esxio.py && cd ..
python3 nsx_cleanup.py --vcenter vc01.example.com --cluster cluster-a # survey: changes nothing
python3 nsx_cleanup.py --vcenter vc01.example.com --cluster cluster-a --commit # one host at a time
python3 nsx_cleanup.py --direct --host esx05.example.com --vib-remove # a host that has joined nothing yetIt needs pyvmomi and paramiko; if the Python you typed lacks them, the tool looks for one that has them and tells you the one command that fixes it otherwise. Every name in the examples is a placeholder.
Conclusion
A host can carry 48 NSX VIBs and show you 25. Find them by version, prove there is nothing configured on top of them, and take them off with the command Broadcom supports, one host at a time. Then the image converts, and the import can begin.
References
- KB 337062, host networking lost after NSX VIBs were removed manually: why
esxcli software vib removeis the wrong tool past an N-VDS - KB 451192, the checklist for stale component removal when
del nsxdoes not finish the job on a genuinely prepared host
Repository: github.com/noahfarshad/nsx-host-cleanup (release 1.0.0)
Related Stories:
- Importing ESXi 8 Into VCF 9.1 When ESX 9 Won’t Run on the Hardware — the import these clusters were being built for
- Prepared Is Not Working: The Overlay a VCF Import Leaves Out — the NSX that was supposed to be there, after the import
- Sixteen Hosts, Seven Requirements, One Command: Getting ESXi Hosts Ready for VCF Commissioning — the host prep that runs first, and the moment
--directis for