A production Avi Load Balancer estate had to move onto a new VMware Cloud Foundation 9.1.1 build with NSX Federation across two datacenters, and gain failover between them. The old estate was one Avi 31.1.2 controller cluster serving 67 virtual services. In about two weeks we deployed a 32.1.3 cluster at each site, found out the hard way that Avi under Federation lives on the Local Manager, rebuilt both sites on that design, built GSLB, and moved every service. This post is the platform half: the controllers, the licensing, the supported design and what broke on the way to it. GSLB and the migration tool have their own posts.
Where We Started
Blue. One 31.1.2 controller cluster at the first site, running production: 7 Service Engines and 67 virtual services on 66 VIPs. The UI counted 61 early on; the migration tool’s full export found 67. Every service was single-arm on a stretched NSX Federation Tier-1, using a Global Manager segment. Some applications’ DNS names still pointed at an F5 in front of them. Blue had no GSLB, and whether it could fail over to the second site at all had never been confirmed.
Green. A new VCF 9.1.1 build with NSX Federation between the two sites. The distributed firewall had already been migrated by API, 281 rules, 13 policies, 246 groups and 19 services, all imported disabled. That pattern became the template for the Avi tooling.
What we asked of ourselves. All of it: virtual services, VIPs, pools, certificates, health monitors, every component, at both sites. GSLB for four DNS zones. Then a cutover, once everything was in place, because nothing trickles data until DNS flips. And the way it had to be built: idempotent, deployable from YAML, customer-agnostic code with the customer’s environment in one file, one migration tool that does everything, and nothing created by hand.
The Decisions That Held
- One controller cluster per site. This is forced, not chosen. Broadcom’s documentation says the controller is always present in the same domain as the NSX Manager it is associated with. Green has two NSX Local Managers, so it needs two Avi clusters. A stretched controller is not an option.
- Avi before the Supervisor. A Supervisor on NSX networking binds to whichever load balancer exists when it is activated. Treat that choice as fixed.
- Avi 32.1.3 and VCF 9.1.1 are a matched pair. The VCF Operations installer refused 32.1.3 against NSX 9.1.0 (“image is not compatible with NSX_T_MANAGER 9.1.0.0200”). NSX went to 9.1.1 across all domains first. Nobody had planned for that prerequisite.
- A rebuild, not an upgrade, cut over by DNS. Every virtual service gets a new VIP on green. A DNS change moves each application, and reverting the DNS change moves it back. Blue keeps serving until then.
- Site-local NSX objects only. This came later, after KB 428430. Each site has its own Tier-1, linked to that site’s Tier-0, and local segments for engine management and data. The controller writes each VIP as a /32 static route on the Tier-1. Route advertisement, plus a permit in the Tier-0’s deny-by-default prefix list, carries the VIP north.
- Cross-site failover is GSLB, active/standby by priority. The first site is preferred; the second answers only while the first site’s member is down.
- Mirror blue’s addressing. Per site, one /24 holds the VIPs (.10 to .240) and the engines’ data addresses (.241 to .254), single-arm, as blue was.
- Licences, not configuration, are the limit. All the Service Engines at both sites draw from one 14-unit licence pool. Capacity decisions were made against that pool.
Two caveats were written down for the customer before any of it was built. A VIP is worthless if the servers behind it are down. And GSLB is the traffic-steering half of a DR design, not the DR design.
Deploying the First Site
VCF 9.1 deploys the controllers through VCF Operations rather than from an OVA: the workload domain, Manage Components, Avi Load Balancer, Install. We chose 32.1.3, Small, 3 nodes. The controllers land in the management domain even when you install against a workload domain. The wizard rejected two sets of node addresses (“Ips are not in management subnet”) before the third set, on the management network, went through.
Small is 6 vCPU, 32 GB and 512 GB per node. The disks are thick, lazy-zeroed, with no choice offered: about 3 TB raw on vSAN at FTT=1 for the cluster. The anti-affinity rule has to be added by hand.
Two things stood between the controllers and a cloud. The NSX cloud failed with “Valid Vcenter config doesn’t exist in cloud” until a local content library existed on the management vCenter. And the NSX cloud VCF Operations creates comes up in VPC mode, which can only be chosen when a cloud is created, so the application services went on a classic NSX cloud we built ourselves.
The licensing block
Service Engine creation failed with “Insufficient licensable resources available to create Service Engine.” From 32.1.1 the controller accepts only V2 licences, so the old key loaded, showed 14 units, and still licensed nothing. A new controller defaults to the ENTERPRISE_WITH_CLOUD_SERVICES tier, and in that tier Service Engines are licensed only from an Avi Cloud Console pool. The pool was empty: the capacity was sitting unallocated in the console the whole time. The documented 30-day evaluation was never present on this controller. The fix was to allocate the V2 licence to the Default pool. The controller showed licensed and connected the next day.
Avi is licensed on data plane capacity. The control plane consumes nothing. Avi Enterprise is a separate entitlement from VCF core licensing and doesn’t appear in VCF Operations’ licence view, and KB 443599 is worth noting for later: an Avi version mismatch can block an SDDC Manager upgrade plan.
The First VIP, and the GM: Prefix
Service Engines deployed, registered and reported healthy, and never attached a data interface. The test virtual service wouldn’t place. We lost about a day to wrong turns: DHCP versus static addressing, a “static IP” checkbox and the IPAM profile, and VPC mode, the lead suspect for a day until a rebuild disproved it. Blue was sitting right there as a working reference the whole time. Comparing green’s settings with blue’s killed each theory in minutes. Diffing the two beats reasoning from first principles.
The real error came from the controller CLI, not from any screen: Network 'GM:net-…' matching not found in Vcenter. Avi 32.1.3 names network objects discovered from the NSX Global Manager with a GM: prefix, then looks up the vCenter port group by that name. The port group has no prefix, so the lookup never matches. Renaming the object through the API returned 200, and discovery reverted it within minutes. The NSX team had flagged the prefix first.
The workaround was to build the data segment on the NSX Local Manager instead. The first green virtual service came up the next day. We opened a Broadcom case with a control-case table: blue 31.1.2 against green 32.1.3, both referencing Global Manager objects, with only green failing.
A platform tool came out of this phase, kept apart from the migration tool on purpose. It has trust, diag, preflight, snapshot, repair, teardown, build, verify and cycle subcommands; its preflight checks that the data network’s name matches a vCenter port group before any object is created, and it pins certificates after a full-chain check. It can delete things, and keeping that out of the migration tool is what lets the migration tool promise it never deletes.
The Supported Design
Broadcom’s answer cited KB 428430: Avi communicates and integrates directly with the NSX Local Manager API, has no visibility into or management of the Global Manager, and cannot discover, reference or orchestrate lifecycle operations such as provisioning Service Engines or attaching vNICs on Global segments or Global Tier-1 gateways. The workaround is now the design: local segments for management and data, and a local Tier-1 per site linked to a Tier-0. The behaviour had changed between versions, but per Broadcom the support status never had. So the slides say unsupported, not regression.
That put production on the wrong side of the line. Blue’s services run on Global Manager objects: working, but unsupported. Two consequences followed. Don’t upgrade blue’s controller in place. And the migration is now remediation, not just a platform move.
Moving the first site’s engine management
Existing engines can’t be edited onto a new management network (“Cannot change immutable field: network_ref”); they get replaced. The surprise was that the first site’s Default-Group carried the live vSphere Supervisor’s load balancer services: the Kubernetes API, Harbor and an image proxy. A delete warning stopped the cleanup, and that warning is the only reason the Supervisor kept its load balancer. The services moved make-before-break onto new engines. Then the controller reported “Controller over license limit”: every engine holds units until it’s deleted, including the dead ones.
The second site
Three Small nodes on 32.1.3, a classic NSX cloud with VPC mode off, local VIP and management networks, DHCP relay for engine management, and one default route in the Avi VRF: one route per Tier-1, not per VIP. The test service was up the same day, with no GM: anywhere. Build both segments on the Local Manager, never the Global Manager. That’s the whole lesson of the case.
Admin login
This went TACACS+ (blue’s model), then LDAP, then an OIDC attempt, then back to LDAP. Two findings in 32.1.3 went to Broadcom. LDAP Group Match never matched AD groups, so users authenticated but got no privileges; the workaround is to set user_attributes memberOf from the CLI, which the UI has no field for, then map with an anchored Attribute Match regex. And Controller OAuth (VCF SSO) saved as a profile, but the login page never offered it, and LDAP alongside it was refused: two remote profiles are only allowed when the primary one is SAML.
The firewall
Rules were written against controller node addresses, never the cluster VIP, because traffic leaves from the node address. No rule is needed from controller to engine: the engine always initiates.
| Flow | Ports |
|---|---|
| Controller node to controller node | TCP 22, 443, 8443 |
| Engine to controller nodes | TCP 22, 8443; UDP 123 |
| Engine to engine | IP protocols 75 and 97; UDP 1550; TCP 4001 |
| Controllers to vCenter, hosts and NSX | TCP 443 |
| GSLB, site to site between controllers | TCP 443 |
| Corporate DNS to the GSLB DNS VIPs | UDP and TCP 53 |
The Supervisor Owns Its Service Engine Group
After the build, the customer wanted to consolidate engine groups to free licence units, starting with moving the Kubernetes services out of Default-Group. Disabling that group’s engines didn’t push the services anywhere: Avi built a new engine inside Default-Group instead, because a service only moves within its own group. kubectl on the Supervisor showed why. Every Supervisor namespace, each an NSX VPC, has an AviInfraSetting pinned to Default-Group; the AKO config map names no group at all, and the Supervisor generates it. The Avi for VCF 9.1 release notes say it plainly: AKO in the Supervisor does not create a cluster-specific Service Engine Group and uses the default group for each tenant.
So consolidation runs the other way: the migrated applications join Default-Group and their own group retires. The migration tool supports this with two settings; it had blocked Default-Group on purpose, and the block has a switch. Avi’s own object model allows the move in place, with the warning that moving to a new SE group is disruptive to existing connections. Before any DNS flip there are no connections to disrupt. Two limits to watch: an engine serves at most nine networks, and each VKS namespace is its own VPC.
Product Issues, KBs and Defects
| ID | What | Outcome |
|---|---|---|
| KB 428430 | Avi integrates with the NSX Local Manager only; Global segments and Tier-1s are unsupported | Design built to it at both sites |
GM: prefix (32.1.3) |
Discovered Global Manager networks carry a prefix that breaks the vCenter port group lookup; renames are reverted | Covered by KB 428430; asked Broadcom to reject Global Manager objects at configuration time |
| LDAP Group Match (32.1.3) | AD group membership never matches; users log in with no privileges | Workaround: user_attributes memberOf from the CLI plus an Attribute Match regex; reported |
| Controller OAuth (32.1.3) | The VCF SSO profile saves, but login offers no SSO | With Broadcom; LDAP used per site |
| AV-269617 | Override Management Network on an SE group has no effect in an NSX cloud | Left empty |
| AV-290059 | Deleting a virtual service can leave a stale /32 (“Next hops already exist”) | In the runbook’s troubleshooting table |
| AV-291715 | VCF Operations’ users-and-groups rule reads Compliant before it is configured | Noted |
| KB 386170 | Registering with an account that doesn’t hold the licences leaves the pool empty | Register with the licence-holding account |
| KB 443599 | An Avi version mismatch can block an SDDC Manager upgrade plan | Noted for lifecycle |
| AV-264200, AV-268512 | NSX DFW service not created for a virtual service with more than 15 ports (fixed in 32.1.3); automated certificate renewal fetches only the leaf | Noted in the design |
| Supervisor SE group (VCF 9.1) | AKO in the Supervisor uses each tenant’s default SE group | VKS stays on Default-Group; consolidate by moving applications in |
| Documentation | GSLB is in the Avi 32.1 guide but not in the Avi for VCF 9.1 guide | Written support position requested |
Lessons Learned
- Compare with a working reference before forming theories. The old controller answered each wrong theory in minutes, and reasoning from first principles cost a day.
- Capture evidence before changing anything, and ask the engine, not the UI. Every configuration screen had been checked twice before the controller CLI named the real error.
- Prove a VIP from outside the fabric. It’s the step most often skipped. When it fails, the cause is almost always the Tier-0 prefix list or redistribution. Our version was a missing route at the second site, which a jump host had hidden.
- Under NSX Federation, Avi lives on the Local Manager. One controller cluster per site, local segments and Tier-1s, and GSLB for cross-site. Something that works on global objects in an older release may still be unsupported.
- Licences are the long pole. From 32.1.1, check which tier the controller runs, whether a V2 licence is allocated to the Cloud Console pool, and how many units every engine at both sites will hold, idle and dead ones included.
- Under VCF 9.1, the Supervisor owns its Service Engine group. To consolidate, move the applications into the Supervisor’s group, not the other way round.
- Firewall rules use node addresses, never the cluster VIP. And no rule is needed from controller to engine.
What Went Wrong Along the Way
We kept a list of our own mistakes, and they’re part of the story. We assumed the controller ran on an evaluation licence out of the box; it didn’t, and the capacity was waiting in the Cloud Console pool. Two rounds of wrong node addresses went by before we read that the controllers deploy into the management domain. A day went on DHCP, a checkbox and VPC mode before we diffed against blue and asked the controller CLI. We called the GM: behaviour a regression, and Broadcom’s answer was that it had never been supported. A wrong subnet propagated into the deck, the runbook and the IP-space record, and one runbook step would have deleted a live segment that shared a name with a Global one; it was caught and rewritten. And “green has no production on it” was wrong: the Supervisor’s load balancer services were there, and a delete warning saved them.
References
- KB 428430, the Broadcom article behind the site-local design: Avi integrates with the NSX Local Manager, not the Global Manager
- KB 443599, an Avi version mismatch blocking an SDDC Manager upgrade plan
- KB 386170, the licence pool and the account the controller is registered with
Repository: github.com/noahfarshad/avi-blue-green, the migration tool that moved the services onto this design
Related Stories:
- Avi GSLB Across Two Sites: Prove Failover Before the Production Names Move — the GSLB build on these controllers, step by step
- Moving Avi from Blue to Green Without Ever Writing Back to Blue — the converge that moved all 67 services
- Bridging Blue to Green: Moving Live Workloads to a New NSX Federation, One Subnet at a Time — the VMs of the same Blue-to-Green move
- Moving NSX DFW Between Global Managers Without Turning the Rules On — the firewall migration whose pattern the Avi tooling copied