Importing ESXi 8 Into VCF 9.1 When ESX 9 Won’t Run on the Hardware

By /

A customer with twelve ESXi 8.0U3 hosts on 2016-era Supermicro, and VCF 9.1.1. Two things were true at once. ESX 9 will not run on the hardware: the CPUs are Broadwell, VCF 9’s CPU list starts at Skylake-SP, and the hosts boot legacy BIOS. And VCF 9.1.1 will not commission an ESXi 8 host:

Host ESXi version - 8.0.3-25595708 is not supported. VCF supported ESXi versions - [9.1.1.0.25714478]

So the clusters get built by hand on vSphere 8 and imported afterwards. That is a supported path, and it is a different animal from a greenfield domain in ways that only show up later. This post is the import half: the decision that deleted most of a work package, what has to be true before VCF will take a cluster, and the tool that builds it. The two surprises that followed, NSX nobody had installed and an overlay that wasn’t there, have their own posts.

The Collision That Wasn’t

The first of the brownfield differences: a shared NSX Manager “must be the same or later than the management domain”, and NSX 9.1 does not support vCenter 8.0.x. At vCenter 8.0.3 those collide, and the apparent way out is a dedicated three-node NSX 4.2.x cluster per domain. Six appliances, 144 GB of RAM, eight DNS records.

Upgrading the vCenters to 9.1.1 removes the collision entirely. The hosts stay on 8.0U3; only the appliances move. That one decision deleted most of a work package, and it is the kind of thing a greenfield build never makes you think about.

What Has to Be True Before VCF Will Import a Cluster

The tool, deploy_vc.py, builds a vSphere 8 cluster to the point VCF will import it as a workload domain, from one YAML file with a block per site. It runs in steps, each one read first and written only with --commit:

Step What it does
dns Forward and reverse records for the vCenter and every host
esx Each host reachable, its build read without logging in
vcsa Writes the vcsa-deploy spec and runs it when an installer path is set
upgrade Upgrades the appliance to the configured target, when one is set
objects Datacenter and cluster: HA, DRS, EVC, born on a vLCM image seeded from a host
hosts Each ESX host added to the cluster
ntp Time on every host, right after it joins
vds Distributed switch, uplinks, port groups
vmk vMotion and vSAN vmkernel adapters from the configured ranges
mgmt vmk0 and the uplink carrying it, moved onto the distributed switch
vsan vSAN enabled on the cluster
image The cluster managed by a vLCM image, taken from a host in it when it has none
ssh SSH on the appliance, which the import needs
ready The import prerequisites

Three of those earned their position the hard way. ntp sits right after hosts because it is the first thing that can be set once a host is in the vCenter, vSAN wants the clocks together before it forms a cluster, and the inventory sync that follows the import stalls on a host whose time is unmanaged; late is how it got missed twice. mgmt moves vmk0 and the uplink carrying it in one host transaction, because ESXi silently rolls back a network change that costs it the vCenter. And image exists because the import refuses a baseline-managed cluster outright, and a cluster that carries stale NSX will not convert to an image at all, which is the next post.

A fourth joined them on the day the first ESXi 8 cluster went in behind a 9.1.1 vCenter. vds now creates the switch at network.vds_version, 8.0.3 unless set. Left to choose, a 9.1.1 vCenter builds a version 9 switch, and an ESXi 8 host cannot join it: the appliance deployed, the cluster formed, the hosts joined, NTP was set, and the host attach to the switch failed with “the operation is not supported on the object”, which explains nothing. A switch that already exists at another version now blocks the step, naming the version it found and the one the hosts need, because a distributed switch is upgraded one way only. It is deleted in the vSphere Client and the step run again, and everything before it reports OK rather than being rebuilt. The same run left a blank vMotion range to fill in; --set does that from the command line, one value by its full name, with a backup of the file written first, so nobody edits a live config by hand.

A fifth came from the same vCenter. The cluster had been created before its hosts joined, so it had no vLCM image: nothing was in the inventory to seed it from, and the import refuses a baseline-managed cluster. Converting one afterwards needs a desired image first, which is the part the vSphere Client’s Setup Image wizard does when you choose to import the image from a host. image now does the same over the documented calls: the depot import from a host already in the cluster, the specification it returns imported as a draft, the draft committed, then the enablement check and the enable. The desired image is exactly what the hosts run, so nothing is non-compliant and nothing is remediated. On a fresh vCenter the objects step now seeds the first cluster from the site’s first host, which joins the cluster as it is created, so the next cluster is born on an image instead of converted to one. One way, as before: nothing happens without --commit, an image the tool cannot read is not mistaken for one that is missing, and a draft someone left open on the cluster blocks the step rather than being overwritten.

Not done here: licensing, and the import itself. The hosts themselves, build, hostname, networking, disks and certificate, are esx_ready.sh‘s job, and it runs first.

Releasing the Hosts First

The day-2 host adds on this engagement were not new hosts. They were hosts coming out of a live cluster on the old vCenter, with vSAN data on their disks and NSX on their switches, going to a new cluster on a new vCenter. Pulling one out by hand is a dozen steps in three UIs, each with a way to lose data, so host_release.py joined the repository to do it in the same style as the build: seven phases, each read from live inventory, each resumable, nothing written without --commit.

Phase What it does Refuses when
check the host is where the config says it is in a different cluster, or disconnected while still clustered or holding vSAN disks
evacuate reports what maintenance mode will relocate powered-on VMs and DRS is not fully automated
drain maintenance mode with full data migration, the multi-hour one the source vSAN would be left under 25% free, or its capacity cannot be read; checked on a plan as well as a commit
disks the vSAN disk groups out, with a noAction decommission spec so vSAN does not try to relocate what drain already moved the host is not in maintenance mode
uncluster out of the cluster to the datacenter level, which is what makes NSX uninstall from a host it manages through the cluster vSAN disks still claimed
nsx waits for NSX to finish, then force-deletes a transport node that lingers the host is a transport node and no source NSX is configured
release disconnected and out of inventory: standalone, ready for esx_ready.sh NSX still lists it, or the host still carries an N-VDS or TEP vmkernels

Two of those earned their position the same way the build steps did. uncluster sits before nsx because a host NSX manages through a cluster is not deleted from NSX directly: it leaves the compute collection and NSX uninstalls, and removing it from vCenter first leaves a stale transport node that needs a force delete. And release refuses a host still carrying NSX residue because esx_ready.sh has no NSX handling: it would remove the TEP vmkernels as extras and leave the VIBs and the N-VDS, which then fail the destination’s vLCM solution apply. The refusal prints the nsx_cleanup.py command instead. When every host in a cluster is on the release list there is nowhere to evacuate to; the plan says the cluster is being dissolved, not shrunk, names how much data goes with it, and waits for --dissolve to be said out loud. The source: block lives under the site in the same env file as the destination, so one file describes both ends of a move.

Where Each Tool Sits

Every new cluster, and every day-2 host addition, runs in this order. Each step blocks on the one before it, on purpose.

  1. host_release.py, this repository, when the hosts are coming out of another vCenter: evacuate, drain, disk groups out, out of the cluster, out of NSX, out of inventory. The host it leaves behind is standalone, which is the only kind step 1 touches.
  2. esx_ready.sh: build, hostname, vmkernels, uplinks, portgroup, disks, VMFS-6, certificate, maintenance mode (Sixteen Hosts, Seven Requirements, One Command). Stale NSX from a previous life comes off here too, before the host joins anything.
  3. deploy_vc.py, this post: the cluster, VDS, vmkernels, NTP, vSAN and the vLCM image. Its esx step refuses to build a cluster whose hosts are on mixed builds; a cluster wants a uniform image, and finding out otherwise later costs a rebuild.
  4. The VCF import, then the inventory sync in VCF Operations. The sync stalls on a host whose time is unmanaged, which is why ntp sits where it does above.
  5. nsx_prep.py: transport VLAN, TEP pool, transport node profile and host preparation (Prepared Is Not Working). A brownfield import prepares the hosts into the VLAN transport zone only, by design.

Patterns That Earned Their Place

The import tool, the NSX cleanup and the overlay tool are three standalone scripts with nothing beyond the standard library, plus pyvmomi for the vSphere one. urllib, not requests, because the jump box is whatever the jump box is. They share a set of rules, and every one of them was learned mid-engagement.

Plan, then commit. Every tool runs read-only by default and prints what it would change. --commit does it. Nothing has a -y.

Declare the end state in YAML, not in flags. The overlay tool started flag-driven, and each cluster took four commands with long argument lists, each one a chance to mistype a VLAN on a live fabric. It now takes a config, and so does this one: the sites, the hosts, the switch, the ranges, in one file.

Idempotent by construction, not by bookkeeping. There is no state file. Each step reads the live system and plans against it, and every planner returns “nothing to do” when the end state is already reached. A run that dies halfway, a run against a half-configured estate, and a second run after a clean one all behave identically: they do what is left. A state file would be one more thing to go stale.

Read the result back. A 200 means the API accepted the request. After every write these tools re-read the object and report what it says now. ESXi silently rolls back a network change that costs it the vCenter; NSX accepts a profile and then does not realize it. “The write returned but the profile still reports no overlay zone” is a line worth having.

Copy from what works; never invent. This tool seeds a new cluster’s vLCM image from a host, and when a cluster was created before any host joined, takes the image from one of them afterwards. The overlay tool takes its transport zone and uplink profile from a reference cluster that is already healthy. The one input no tool will guess is a TEP pool gateway.

Say when a check could not run. Silence reads as a pass. If something cannot be read, the output says so, with the reason, instead of nothing.

Distinguish absent from non-compliant. Two bugs in one engagement came from the same shape: a host not in the vCenter counted as compliant, and a cluster that did not exist reported as “baseline-managed” with a remedy that could not work. If you cannot read it, say so.

Tests

python3 deploy_vc.py selftest runs 289 checks without touching an environment: every step’s planner against stub hosts and a stub vCenter, the config reader, the exit codes and the “next” line a run ends with. python3 host_release.py selftest is another 158, every one of them a gate that stops the tool destroying something: the capacity arithmetic, the dissolve rule, the order of the phases, a host that has already left, a task that never finishes. It needs pyvmomi, and the tool finds a Python that has it or, with bootstrap, makes a virtualenv once per machine.

Lessons Learned

  1. Check the version rules before designing around them. The NSX-versus-vCenter collision was real at 8.0.3 and gone at 9.1.1. Upgrading the appliances cost an afternoon; the six-appliance design would have cost weeks.
  2. NTP goes in as soon as a host joins. Everything after it, vSAN and the import included, assumes the clocks already agree.
  3. Move the management vmkernel and its uplink together. Separately, the host rolls the change back and you find out from the vCenter losing it.
  4. Image-managed, or the import will not look at it. Seed the image from a host that is already right, and clear anything that stops the conversion first.
  5. If you cannot read it, say so. A missing host is not a compliant host.

Getting It

git clone https://github.com/noahfarshad/vcf-import-prep.git
cd vcf-import-prep
python3 deploy_vc.py selftest
cp deploy_vc.example.yaml env_vc8.yaml                 # then edit the sites, hosts and ranges
python3 deploy_vc.py build --config env_vc8.yaml       # plan: reads everything, writes nothing
python3 deploy_vc.py build --config env_vc8.yaml --commit

python3 host_release.py survey --config env_vc8.yaml --site site-a   # hosts coming out of another vCenter: where they are, and whether vSAN can lose them
python3 host_release.py apply  --config env_vc8.yaml --site site-a --commit

--only STEP runs one step, --precheck checks the appliance deploy or upgrade without touching anything, --set NAME=VALUE changes one config value by its full name, and --detailed-exitcode gives 0 for nothing pending, 2 for pending and 1 for blocked. Passwords come from the config or are asked for by name, and never reach the run log. Every name and address in the example is a placeholder.

Conclusion

Importing is supported, and it is not the greenfield path with a different button. The appliances move, the hosts stay, and the cluster has to arrive image-managed, time-synced and with its management network already on the distributed switch. Build that from one file, plan it first, and read every write back.

Repository: github.com/noahfarshad/vcf-import-prep (release 1.2.0)

Related Stories:

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top