Four imported clusters all showed Prepared, NSX Configuration: Success, green hosts and zero alarms. And TEP IP Address: Not Set on every host. No overlay segment reached any of them. This is the last of three posts about importing ESXi 8 clusters into VCF 9.1, and it is the one worth writing down, because nothing anywhere said anything was wrong.
What an Import Leaves
A brownfield import prepares the hosts into the VLAN transport zone and nothing else. That is documented and intentional (KB 444225), and the remedy it gives is to edit the transport node profile by hand, per cluster, in the UI.
Except on this estate the overlay zone was attached. What was missing was everything underneath it: transport VLAN 0, untagged, on all four clusters, where every working cluster in the same NSX carried a real TEP VLAN; and ip_assignment_spec: NoIpv4, no pool, so nothing to allocate a TEP from. Both are defaults the import leaves behind. Neither is an error anywhere.
NSX Installs Through vLCM, and vLCM Has Opinions
Before the overlay could be fixed, host preparation itself failed on a freshly built cluster:
Solution(s) apply failed because the vSphere Lifecycle image contains either a
new ESXi version or a new addons version or new components.That string is not from NSX. It is vLCM refusing, relayed by NSX. On an image-managed cluster NSX installs its VIBs as a vLCM solution, and the solution apply only adds VIBs; it will not update base ESXi, addons or components. So any pending image drift makes it refuse. The compliance screen said the rest:
Solution components of disabled Solutions VMware NSX-T 4.0.1.1.0 will be removed from the hosts during remediation. The cluster needs remediation to finish enabling these Solutions: VMware NSX-T 9.1.1.0.
One remediation did both: stripped the old, installed the new. The fix was never “fix NSX”; it was “settle the image, then let NSX in.”
A second cluster then failed differently:
Live Patch cannot be performed together with solution(s) VMware-HBR-Agent that
will be enabled or disabled live.Live patching cannot add or remove a solution component. Turning off Enforce Live Patch in the cluster’s remediation settings clears it. But the HBR agent being removed is vSphere Replication, so that one deserved a question to the customer before anyone clicked.
The Object Model
NSX MANAGER
+----------------------------------------------------------------------+
| |
| TRANSPORT ZONE UPLINK PROFILE IP POOL |
| +--------------+ +----------------+ +---------------+ |
| | overlay-tz | | transport VLAN | <--> | cidr | |
| | vlan-tz | | teaming | (1) | gateway | |
| +------+-------+ | MTU | | range | |
| | +-------+--------+ +-------+-------+ |
| | (2) | (3) | (4) |
| +---------------+--------+-----------------------+ |
| v |
| TRANSPORT NODE PROFILE |
| +------------------------------+ |
| | host_switch = the VDS | |
| | transport_zone_endpoints | <-- (2) the zone |
| | host_switch_profile_ids | <-- (3) the VLAN |
| | ip_assignment_spec | <-- (4) the pool |
| +--------------+---------------+ |
| | |
| TRANSPORT NODE COLLECTION |
| (binds the profile to one cluster) |
+-----------------------------+----------------------------------------+
| (5) host preparation runs here
v
vSPHERE CLUSTER
+---------------+---------------+
v v
ESXi HOST ESXi HOST
vmk10 / vmk11 vmk10 / vmk11
(vxlan TCP/IP stack) (vxlan TCP/IP stack)
| |
+------ Geneve tunnels ---------+
|
v
EDGE NODES
TEP on a *different* VLAN, so every
host-to-edge tunnel is ROUTED -- see (1)Five places it breaks. Only (5) shows up as an error. The other four present as a quiet absence of TEPs, which is why this is worth a diagram.
- Pool has no gateway, edges on another tunnel subnet: nothing reaches an edge. Seen on this estate once already, during the VM migration: the edges’ pool had no gateway, so no workload host could reach an edge, and every screen looked fine.
- No overlay zone: VLAN-backed only. The import’s default.
- Transport VLAN 0, or disagreeing with the pool’s subnet: addresses allocate and carry nothing. The most expensive failure, because every screen says success.
- No IP assignment:
TEP IP Address: Not Set. There is nothing to build a tunnel endpoint from. - Host preparation skipped or failed, usually on vLCM image compliance:
Install SkippedorPreparation Failed. The configuration above can be perfect and no host is ever touched.
The diagnostic that collapses most of this: compare against a cluster whose overlay already works. Its transport zone, uplink profile and pool are a correct answer sitting in the same NSX Manager. Copy, don’t design.
What the Tool Does
nsx_prep.py puts working overlay onto brownfield-imported clusters. It takes its overlay transport zone and uplink profile from a reference cluster that is already healthy, and nothing is invented: if the reference has no overlay zone, it says so and stops rather than guessing at one. Six clusters is six chances to mistype a teaming policy on a live fabric, so the end state is declared once:
nsx: nsx01.example.com
reference: cluster-ref
pools:
tep-222-leaf03-04:
cidr: 10.0.22.0/24
range: 10.0.22.201-10.0.22.250
gateway: 10.0.22.254 # the address the leaves hold. Never guessed
clusters:
cluster-a: { vlan: 221, pool: tep-221-leaf01-02 }
cluster-b: { vlan: 222, pool: tep-222-leaf03-04 }python3 nsx_prep.py build --config nsx.yaml # plan: changes nothing
python3 nsx_prep.py build --config nsx.yaml --commit # create missing pools, set the VLANs, point the clusters at their pools, verifyEach step is skipped where it is already right, so a run that failed halfway is finished by running it again. There is no state file: every phase reads the live system and plans against it. survey lists every cluster NSX knows about and where each one stands, with every uplink profile’s transport VLAN and every pool’s subnet, gateway and free count, which is where the TEP VLANs and subnets actually live. verify is the part that matters after a write: TEP addresses, tunnel state from host to edge, and the pool gateway behind them. An applied overlay zone is not a working overlay. vlan refuses to change an uplink profile shared with clusters you did not name; pool checks the range and gateway sit inside the subnet, that the gateway is not inside the range, and warns when another pool already serves that subnet, because two managers handing out one /24 is how duplicate TEPs happen. gateway puts a gateway on a pool that has none. diag exists purely so the next surprise is answered with evidence rather than a fourth guess.
Two more rules from the shared set matter most here. Read the result back: NSX accepts a profile and then does not realize it, and “the write returned but the profile still reports no overlay zone” is a line worth having. Say when a check could not run: if the pool size cannot be read, the output says TEP pool capacity NOT checked with the reason, not nothing.
Tests
python3 nsx_prep.py selftest runs 89 checks on the planners. The end-to-end suite, 79 checks, drives survey and the apply orchestration against a stub that answers like the real NSX, and it is the one that found real bugs: the unit tests passed while the fetching, the matching of clusters to profiles and the write-back had never executed. Writing the stub surfaced four defects in an hour, two of which would have failed on first contact with a customer’s manager.
Still Open
The gateway on a TEP pool is the single input none of this can derive. It is the leaves’ address in that subnet, and it has to be asked for. And matching host transport nodes back to their cluster is version-sensitive: the manager API and the policy API disagree about which fields exist, which is what diag is for.
Lessons Learned
- “Prepared” is a statement about VIBs, not about tunnels. Look at the TEP column, not the status column.
- An import leaves VLAN 0 and no pool. Neither is an error. Both stop the overlay.
- The vLCM message is vLCM’s. Settle the image, then let NSX in; and ask before you let a remediation remove a solution you did not know was there.
- Copy the working cluster. The right transport zone, uplink profile and pool already exist on the same manager.
- Verify from the host up. TEP address, tunnel to the edge, gateway behind the pool. The UI’s green is earlier than any of those.
- The end-to-end stub is the test that matters. Planners can be perfect while the orchestration has never run.
Getting It
git clone https://github.com/noahfarshad/nsx-overlay-prep.git
cd nsx-overlay-prep
python3 nsx_prep.py selftest
python3 nsx_prep.py example > nsx.yaml # then set the manager, the reference cluster, the pools and the clusters
python3 nsx_prep.py survey --nsx nsx01.example.com # where every cluster stands; changes nothing
python3 nsx_prep.py build --config nsx.yaml # plan
python3 nsx_prep.py build --config nsx.yaml --commit # do it, then verifyStandard library only; PyYAML is used when it is installed. topology.txt in the repository is the diagram above. Every name, address and VLAN in the example is a placeholder.
Conclusion
Four green clusters and not one tunnel. The import did exactly what it documents, the UI reported exactly what it measures, and the overlay was two defaults away from working. Copy the cluster that works, declare the end state once, and verify from the host up.
References
- KB 444225, TEPs not assigned to ESXi hosts after a brownfield deployment: the documented behaviour, and the by-hand remedy this tool replaces
Repository: github.com/noahfarshad/nsx-overlay-prep (release 1.0.0)
Related Stories:
- Importing ESXi 8 Into VCF 9.1 When ESX 9 Won’t Run on the Hardware — how the clusters got there
- The NSX That Nobody Installed: Clearing Stale VIBs Before a VCF Import — the NSX that had to come off first
- Bridging Blue to Green: Moving Live Workloads to a New NSX Federation, One Subnet at a Time — the same TEP-gateway lesson, found during the VM migration