The management domains had finished their upgrade to VCF 9.1.1. The workload domains had not, and the product said they had. In VCF Operations, each workload domain’s upgrade view reported that the domain was already running the target versions. Edit Plan and Cancel Plan were both greyed out. No upgrade sequence was offered. Component Versions listed NSX, vCenter, and ESXi as on target.
The Challenge
Greyed plan controls usually mean an in-flight task or a permissions problem. Neither was true. The domains were idle, and the same account could plan other upgrades. The UI was describing a state the version header contradicted: the domain header still read 9.1.0 while every listed component claimed to be on target.
Reading the release from the SDDC Manager API explained the contradiction. The domain had a customized bill of materials. The host and vCenter targets were pinned to the 9.1.0 builds they were already running. A customized BOM does not show up as a banner. It shows up as a domain that refuses to plan because, under the pin, there is nothing to plan.
The VCF version on a domain header is the floor of the component versions, not a separate number someone typed in. That is why NSX could already be at 9.1.1. The NSX manager was shared with the management domain, which had completed its upgrade, while the workload domain’s own host and vCenter targets were still pinned. The header stayed at the floor. The component list, read carelessly, looked finished.
The Fix
The UI could not clear it. The plan buttons were the thing that was broken. The path was the SDDC Manager API: identify the domain, read the current release target, and clear the customized BOM with the update-release call for that domain. After the pin was gone, the plan controls came back and the domain could be sequenced like any other 9.1.0 workload domain moving to 9.1.1.
The two sites were not on the same patch to begin with. One site’s vCenter pin was not the other site’s vCenter pin. Clearing “the” customized BOM meant clearing each domain’s own target, not assuming the management-domain upgrade had written a single fleet-wide version.
What to Watch For Next Time
A domain that reports every component on target, while its header shows an older VCF version, is not healthy. Check isCustomizedBom before you chase locks, tasks, or permissions. The field is not in the upgrade screen. One API read would have saved the guessing.
It will happen again if the next upgrade plan is created with custom target versions selected. The pin is not a residue of a failed upgrade. It is what the product stores when someone, or something, sets explicit component targets. Leave those targets unset unless you mean to freeze the domain.
The same class of “the installer cannot see the thing you know is there” shows up in custom ESXi image work. Different layer, same habit: believe the API record of the target, not the screen that is summarizing it.
The Results
After the pin was cleared, the plan controls came back. Each workload domain could be sequenced as a 9.1.0 domain moving to 9.1.1. The two sites did not share a vCenter pin. Each domain’s own target had to be read and cleared. The management domains were already unpinned and were left alone.
Lessons Learned
A greyed plan button is not a lock and it is not a permissions problem until the release record says so. isCustomizedBom is the field. The domain header is the floor of the component versions, so NSX can be on 9.1.1 while the header still reads 9.1.0 because the host and vCenter targets are pinned. Believe that record, not the component list read in a hurry.
Getting Started
Run it beside this, against your SDDC Manager. clear prints the payload and does not write until --commit.
export SDDC_URL=https://sddc.essential.coach
export SDDC_TOKEN='...'
python3 domain_release.py list
python3 domain_release.py show DOMAIN_ID
python3 domain_release.py clear DOMAIN_ID --target 9.1.1.0
python3 domain_release.py clear DOMAIN_ID --target 9.1.1.0 --commitClone vcf-domain-release. Export SDDC_URL and a bearer token you already have. Run list, then show on the domain whose header and component list disagree. clear prints the payload and does not write unless you pass --commit. Hard-refresh the Upgrades tab after the write. Lifecycle of the components themselves is still done in VCF Operations.
Conclusion
The screen summarized. The release record did not match. One read of the domain target would have ended the guessing, and one write cleared the pin the buttons could not reach.
References
The field name is in the product API, not only in the script. Get Release By Domain returns isCustomizedBom and describes it as the identifier for a VCF release versus a customized BOM. The same page states that a domain VCF version accounts for the floor of all the component versions. Update Release By Domain ID is the PATCH that writes the target version and the component patches. The interoperable set those patches should point at is the VCF 9.1.1 Bill of Materials. Lifecycle of the components themselves is done in VCF Operations, as described in Lifecycle Management of VCF Components.
The read, the dry-run clear, and the commit that writes the target version are in the repository. It does not collect a token. You bring one.
Repository: github.com/noahfarshad/vcf-domain-release
Related Stories:
- VCF on VxRail: What to Check First — depot and lifecycle before you assume the buttons are the bug
- Getting a Standalone ESXi Host Ready for VCF Commissioning — a different gate, after the domain will actually plan