Everything That Cost Us a Day Returned Success: Nine Versions of esx_ready.sh on a Live Fleet

By /

The same customer and the same site as Sixteen Hosts, Seven Requirements, One Command, one week later. VCF 9.1.1, and two batches of hosts moving on the same day: four hosts going to ESX 9.1.1 and a new cluster by the commissioning path, and four 2016-era hosts on ESXi 8 going to a new cluster by the import path, with a fresh vCenter deploying for them at close of business. Nine versions of esx_ready.sh shipped between morning and evening, 1.12 to 1.20, every one of them onto a live fleet, with a third fire in the middle that had nothing to do with hosts at all.

Here is the through-line, because the day had one: everything that cost time returned success while not having done the work. Not one crash. An upgrade that reported success and left the build where it was. A cluster leave that released no disks. A depot staged onto the one disk the cluster needed clean, followed by ready. A retried task that re-read nothing. Silent no-ops, stale caches, and ownership that survives the command that was supposed to release it. This post is those, in the order they were found, with the fix each one became and the test that now proves it.

The Build the Portal Gives You Is Not the Build VCF Wants

The customer asked for the current ESX 9.1.1 from the Broadcom portal. That is 9.1.1.0100, build 25782009. Four hosts were patched to it, prepped, and presented for commissioning:

Host ESXi version - 9.1.1.0100.25782009 is not supported.
VCF supported ESXi versions - [9.1.1.0.25714478]

VCF 9.1.1’s bill of materials wants 9.1.1.0, build 25714478, and rejects anything newer. The download portal is not the source of truth for a build that has to join a VCF instance; the BOM is. That is the first half. The second half is how you get four hosts back down a build:

esxcli software profile update     # only ever moves a VIB forward
esxcli software profile install    # the only way down

update with an older profile does not error. It reports success, the build does not move, and the next commissioning attempt fails with the identical message. That became update_mode: install in the fleet file: opt-in, because install also removes anything the profile does not carry, and the run banner now says PROFILE INSTALL - writes the image exactly, including downward so nobody is surprised. The self-test checks that no verb is hardcoded anywhere in the host-side script and that the configured one lands.

My error, recorded because it is the instructive part: the tool used to read one host’s build before accepting a target, and I removed that gate when the customer stated a build preference. The gate existed precisely to catch a stated preference that the destination will not accept.

vSAN Disk Ownership Is a Chain, Not a Flag

The best technical content of the day. --wipe-disks failed on two disks per host, on three of four hosts, with:

Error: Read-only file system during write on /dev/disks/naa.50000000000000b2

The chain, learned one wrong guess at a time:

  1. Cluster membership. While a host is a vSAN cluster member, vSAN owns its disks and partedUtil is refused on every one it has claimed. esxcli vsan cluster leave, and the script was doing this after the disk loop, not before. Fixed, with a settle after it, because leaving returns before the disks are released.
  2. Disk groups survive the leave. This is the one that mattered. Leaving a cluster drops membership; the disk groups stay. A host that was never in a cluster can still carry them, which is how the first fix got disproved: the host that failed had printed no leave the cluster line at all, and I had reordered the leave anyway. A cache disk cannot be removed on its own; the group comes apart by its cache device, esxcli vsan storage remove -s <cache disk> -m noAction. -m noAction is not optional. Out of a cluster the default evacuation policy, ensureObjectAccessibility, cannot be satisfied, so the removal is refused before it starts, with a message about data accessibility that reads like a safety check rather than an impossibility.
  3. Partitions. Only now does partedUtil mklabel work.
  4. And it still needs a retry. esxcli returns before a volume has finished detaching. partedUtil refuses once and succeeds a moment later. Three attempts; real runs needed the third.
  unmounted esx70_ds1
  unmounted OSDATA-67ed8baf-b1687e45-130a-001122334455
      waited 20s for the volumes to detach
      still listed after 20s: esx70_ds1 OSDATA-67ed8baf-b1687e45-130a-001122334455
      cleared on attempt 3
  CHANGE remove esx70_ds1 OSDATA-67ed8baf-b1687e45-130a-001122334455, clear eui.00000000000000c3 - now eligible for vSAN

Two adjacent findings from the same loop. Clear per device, not per datastore. Two volumes on one disk, a datastore and a stale OSDATA from an earlier install, mean two unmounts and one mklabel. Per-datastore clearing issues the mklabel after the first unmount while the second volume is still mounted, and fails every time. A stale core dump blocked the clear. The guard checked whether a dump file existed on the datastore. The right check is whether it reads Active: true or Configured: true. A leftover file from a previous install is not a reason to keep a disk, and treating it as one blocked four hosts for an afternoon.

And the one that was embarrassing. datastore: auto meant “the largest local datastore with room”. On a vSAN host that is the cache disk. So the tool staged a 2 GB depot onto the exact disk the cluster most needed clean, upgraded the host, and reported ready. Devices are now ranked: the boot device first, then disks vSAN would not claim, then disks it would, last and with a note on the line.

The Feature That Got Deleted

Requirement 4 of host commissioning, as every runbook states it: the management portgroup is named Management Network. Four freshly upgraded ESXi 8 hosts, every other check green, and this on each of them:

PROBLEM rename portgroup 'VMkernel 2' -> 'Management Network' failed: Error: Invalid option -n
Usage: esxcli network vswitch standard portgroup set [cmd options]
  -p|--portgroup-name=<str>
  -v|--vlan-id=<long>

esxcli has no rename for a standard portgroup. set takes a VLAN id and nothing else. The only way to change the name is to add the new portgroup and move vmk0 onto it, which drops the connection the prep run is travelling over, on hosts that had already been stranded once on this engagement. The first fix made the rename opt-in behind a flag. That was worse: a documented flag that issues an invalid command. The second fix deleted the feature, and the help text says why. Nothing in the vSphere 8 build, the VCF import or the cluster build reads the name, and the four hosts went on to their cluster under VMkernel 2. The correct fix for a feature that cannot work safely is to remove it, and say so.

Testing a Script That Only Runs Somewhere Else

The most reusable thing built that day, and probably the actual subject of the post. esx_ready.sh is one file that generates a payload and runs it over ssh on an ESXi host, under busybox. That is normally untestable without a host, and a host is the one thing you do not want to test on at four in the afternoon with a fleet half prepped. It now has:

sh esx_ready.sh selftest      # 104 checks, no network, no dependencies

How it works: the host-side payload is extracted from the script itself (it lives between : <<'REMOTE_EOF' markers, so the outer script never executes it); the template placeholders are filled with scenario values; esxcli, partedUtil, vdq and the rest are replaced by stubs on PATH in a temp directory, each one logging its full argument list to a call log and answering from fixture files; and the payload runs under sh with that PATH. Assertions come in four shapes, and the last one is the valuable one:

Helper Asserts
ck a parser returns an exact value
has the output says something
did / hasnt a command was or was not run
order command A appears in the call log before command B

order is what catches ordering bugs: leave the cluster before wiping the disks, break up the disk group before mklabel, release the uplink before attaching it. Every one of the day’s failures was an ordering or state bug, and none of them would have been caught by asserting on output. The count went 83 → 89 → 96 → 104 across the day. Every production failure became a check before the fix shipped. That is the discipline worth writing about; the stub harness is just the mechanism. Two fixture details that took a try to get right: a stub that mutates state (vsan storage remove empties the storage list) has to be reset before each scenario, and a failure-injection file holding a countdown is how you test “refused twice, then succeeds” without a real race.

Where the Buttons Moved in VCF 9.1.1

Independently useful, and thinly documented.

  • Host commissioning and add-to-cluster live in the vSphere Client: Global Inventory Lists, Hosts, Unassigned Hosts, Commission Hosts. Broadcom’s own Commission ESX Hosts page says it in those words, requires the management domain vCenter or a VCF instance with VCF SSO and vCenter linking, and adds: “As an alternative, you can perform this task using the SDDC Manager UI.”
  • VCF Operations has no host inventory actions: no commission, no decommission, no add-to-cluster. It is where ESX patching lives in 9.1, under Fleet Lifecycle Management, the ESX Configure Upgrade workflow, with a custom selection that goes down to individual hosts inside a cluster. The division of labour is the thing nobody states plainly: VCF Operations owns host lifecycle; the vSphere Client owns host inventory.
  • Both inventory actions are refused from a managed workload-domain vCenter, Authorization Error, for an account with every role, because that vCenter is neither of the two supported entry points. Do it from the management domain vCenter or SDDC Manager.
  • A retried SDDC Manager task reuses cached inventory. After changing anything on a host, start a new task. Retrying the old one re-reads nothing.
  • found zero SSD devices for SSD cache tier in the Add Hosts wizard means no untagged SSD candidate. capacityFlash=1 marks a disk as capacity, and an untagged SSD is what vSAN wants for cache; a host reporting “5 SSD, 0 HDD” can still fail this.

ESXi Quirks, Cumulative

The ones from the first post still stand: busybox has no tr, no working command -v, a long ssh command line gives /bin/sh: File too large so the payload goes over stdin, and 32-bit arithmetic silently overflows on byte counts. New since:

  • esxcli returns before a volume finishes detaching
  • esxcli software profile update only moves VIBs forward
  • esxcli network vswitch standard portgroup set has no rename
  • a vSAN disk group survives leaving the cluster
  • vusb0 presents as a network adapter and is counted as an uplink (known, not yet excluded)
  • the SSH service policy has to be set, not just the state, or SSH is off again after a reboot, and a run that reboots a host then strands there. VCF disables SSH after commissioning takes a host’s key, so every later batch needs it turned back on first.

The Honest List

Mistakes made during the day, kept because a post that only contains the findings is not useful to anyone doing this work.

  • I sent the customer to run a tool, certmgr, that was not installed, on a Supervisor I had just told them to log into.
  • I removed the build-verification gate because the customer stated a build preference, which directly caused the commissioning rejection above.
  • I shipped auto depot staging that picked the vSAN cache disk, then reported ready.
  • I “fixed” the disk failures by reordering the vSAN cluster leave, on a host that was never in a vSAN cluster. The output said so and I did not read it.
  • I suggested retrying an SDDC Manager task that uses cached inventory.

The pattern in four of the five: acting on a model of the system instead of on what the system had just printed.

What Shipped

Tool Start of day End of day Where
esx_ready.sh 1.12, no self-test 1.20, 104 checks esxi-vcf-host-prep 1.2.0
tkc_cert_renew.sh — 2.3, 10 checks, written during the incident tkc-cert-renew, and its own story
host_release.py in use since the week before 1.0, 158 checks, now public vcf-import-prep 1.1.0
deploy_vc.py 1.0 1.0, used, not changed the same release

esx_ready.sh 1.12 to 1.20, in order: selftest; update_mode: install; --clear-datastores; min_uplinks:; boot-device-preferred depot staging; per-device datastore clearing with wait-for-detach and retry; --restore-vswitch for a distributed switch a vCenter left behind; the vSAN cluster leave moved before the disk loop; the portgroup rename removed; vSAN disk-group teardown by cache device. host_release.py is the step before all of it when the hosts are coming out of another vCenter: evacuate, drain, disk groups out, out of the cluster, out of NSX, out of inventory, with a capacity gate on the source vSAN that refuses an evacuation it cannot verify.

Lessons Learned

  1. Success is not evidence. update returned 0 and moved nothing; the cluster leave returned 0 and released nothing; the retried task returned green on stale inventory. Read the state back after every write, and compare it to what you asked for.
  2. The destination decides the build. The BOM, not the portal, not the customer’s preference, and the tool should refuse a target the destination will not accept, which is the gate I removed.
  3. Ownership is layered. Membership, disk groups, partitions. Each layer has its own command, and the order is the whole fix.
  4. Test the order, not the output. A call log and an order assertion catch the class of bug that output assertions never see.
  5. Delete what cannot work safely. A flag that issues an invalid command is worse than no feature. Say why in the help text.

Getting It

git clone https://github.com/noahfarshad/esxi-vcf-host-prep.git
cd esxi-vcf-host-prep
sh esx_ready.sh selftest                # 104 checks; connects to nothing
cp hosts.example.yaml hosts.yaml        # the build the BOM wants, the bundle, update_mode if going down
read -rs ESXPW && export ESXPW
sh esx_ready.sh plan
sh esx_ready.sh apply --wipe-disks      # leaves the vSAN cluster, breaks up the disk groups, then clears
unset ESXPW

The full key list, the states, the refusals and the --restore-vswitch flow are in the README. Every name, address and device id in the example and the self-test is a placeholder.

Conclusion

None of it crashed. That is the point. The day’s failures were a build that did not move, disks that were not released, a depot on the wrong disk and a task reading last hour’s inventory, and every one of them reported success. The tool now reads the state back, orders the commands the way the system actually works, refuses the target the destination will refuse, and proves all of it against a stubbed host before it touches a real one. One check per outage, before the fix ships.

References

Repository: github.com/noahfarshad/esxi-vcf-host-prep (release 1.2.0), and github.com/noahfarshad/vcf-import-prep (release 1.1.0) for host_release.py

Related Stories:

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top