Bridging Blue to Green: Moving Live Workloads to a New NSX Federation, One Subnet at a Time

By /

We recently moved production workloads off a two-site NSX 4.2 Federation (“Blue”) and onto a new VMware Cloud Foundation 9.1 environment with its own NSX Federation (“Green”). The applications kept their IP addresses, the VMs were moved live, and every step had a tested way back. This post covers how we did it: the network design, the tooling we wrote, and what production taught us along the way.

The tooling is public as fleetparity, release 1.24.1, with the mock managers and the end-to-end flows it was tested against.

The Problem

The customer’s requirements were short and strict:

  1. Move VMs in waves the business chooses. No re-IP, no rebuilds, a few VMs at a time, after hours.
  2. Move the routing per subnet, only once every VM of that subnet has moved. The default gateway stays on Blue until then.
  3. Take the temporary plumbing away afterwards.
  4. A way back from every step.

There were constraints on top. Blue is production, so nothing on it may change outside the subnet being worked on. The people running the steps at night are the network team, not the people who wrote the tooling. And nobody wanted a runbook of twenty manual NSX clicks per subnet.

The Approach: Bridge, Move, Flip, Unbridge

For each Blue segment, Green gets a segment with the same ID, subnet and gateway address, but with its gateway switched off. A Layer 2 bridge joins the two segments over a dedicated transit VLAN on the leaf switches, so a VM on either side sits on the same broadcast domain and still reaches Blue’s gateway.

          BLUE (NSX 4.2 Federation)                         GREEN (VCF 9.1, NSX Federation)

  VMs - segment 192.0.2.0/24                         segment 192.0.2.0/24 - moved VMs
        gateway ON (until cutover)                    same ID, gateway OFF (until cutover)
            |                                                   |
     Blue bridge edge pair                              Green bridge edge pair
     (active / standby)                                 (active / standby)
            |                                                   |
            +-------- transit VLAN on the leaf switches --------+
                      (one VLAN per bridged segment)

Each subnet then goes through three steps:

Step What happens The way back
1. Migrate VMs move to Green by cross-vCenter vMotion, onto the bridged segment. Their gateway is still Blue’s, reached over the bridge. Each VM goes back to the host, datastore and folder it left
2. Cut over The subnet is added to the site’s outbound prefix list. Blue’s gateway goes off, then Green’s goes on. The tool waits until the leaves learn the route from Green. Green off, then Blue on, and the prefix-list entry is removed
3. Unbridge The bridge comes off both segments. Both bridges go back on, and Green stays the gateway

Retiring Blue’s segment is a separate, deliberate step. It’s the only one with no way back, and it waits for the customer’s sign-off.

The Toolset

fleetparity migpath: one command per step

The core is a single Python file of about 14,600 lines. It needs nothing beyond the Python standard library and stock OpenSSH for the switches. It uses PyYAML if it’s installed and its own reader if not. It runs from one jump host that can reach both Global Managers, the Local Managers, the vCenters and the leaf switches. Everything it needs to know lives in one YAML config with a block per site, and each password is named for the system it signs in to.

Every step has the same shape:

fp --move --vm wave1.csv                 # plan: reads everything, writes nothing, prints a digest
fp --move --vm wave1.csv --commit        # does it
fp --move --vm wave1.csv                 # run it again: "No changes"

What every run does

These rules applied to every run, and they turned out to matter more than any single feature:

  • Plan by default. Without --commit, nothing is written. A plan prints every object it would add, update or delete, plus a digest of exactly those changes. --commit --expect <digest> writes only that reviewed plan. If anything changed since the plan was read, nothing is written.
  • Re-read the side you must not break. After every commit, the tool re-reads Blue and fails the run if anything outside the run’s scope changed.
  • Probes across every change. Production addresses are pinged continuously during a commit, and a single lost ping fails the run. A few VMs on the segment are pinged too, so a cutover reports its actual gap.
  • Print the way back. Every commit ends with UNDO: the one command that reverses it. It’s also saved in the run’s evidence folder and in a journal.
  • Print the way forward. A clean run ends with NEXT: the one command to run next for each subnet it touched.
  • Idempotent and restartable. A state file records what the tool built, and a lock allows one run at a time. A run that stops part way finishes when you run the same command again.
  • Evidence. Each run leaves a folder with the plan, the checks, what was written and the probe results. Passwords never appear in it.

Building the bridge

--edges builds the bridge infrastructure on both sides:

  • a pair of bridge edges, cloned from an existing edge in the management subnet but without its Tier-0 uplinks
  • an edge cluster
  • a dedicated VLAN transport zone
  • a trunk port group carrying the whole transit-VLAN pool, with MAC learning
  • a DRS anti-affinity rule so the two edges never share a host

A build run then creates the Green segments and the bridge profiles. Transit VLANs come from a per-site pool. A VLAN in use anywhere (a VLAN-backed segment, infrastructure, another bridge) is never handed out, and a segment keeps its VLAN for as long as it’s bridged.

The switches. The team gives the tool the leaf ports that face the hosts where bridge edges can run. The tool reads each leaf’s live configuration and writes exact NVUE lines. Each line carries the full VLAN list that port should end up with, so nothing is pruned whether NVUE merges or replaces. It never writes a port line for a port that inherits the bridge’s VLANs. Every later run checks that each port and the peer link still carry their transit VLANs, and that no VLAN a port carried has disappeared since.

The overlay. The first rehearsal found that the Green edges’ tunnel pool had no gateway. Moved VMs reached their own gateway but nothing past the edges. The tool now checks the tunnels from every landing host to the edges on every run, and plans the missing gateway as part of the build. The second site had the same gap, and the first build there set the missing gateway before any VM moved.

Moving VMs in waves

Waves are plain CSV files: a VM column and, optionally, the subnet the VM must be on. Exports from vCenter or from the tool’s own reports work as they are.

vm,subnet,notes
app-web-01,192.0.2.0/24,
app-web-02,192.0.2.0/24,

Before any VM moves, it is checked:

  • it is on the subnet the wave file names
  • its segment is bridged on one transit VLAN, the same on both sides
  • both bridges were found active in this same run
  • every leaf port carries that VLAN

A VM that fails a check is blocked with the reason, and the rest of the wave goes ahead. VMs move one at a time, and the first failure stops the wave. Each VM’s NSX tags are captured on Blue and applied on Green, so firewall rules keep matching. Each VM also lands in the same folder path on Green as on Blue, with missing folders created first; the customer’s folders map to business services, so this mattered. A return puts every VM back on the exact host, datastore, resource pool and folder it left from.

Moving the routing and removing the bridge

A cutover refuses while any VM of the subnet is still on Blue, at either site. It also refuses if the Tier-0’s outbound filter would not advertise the subnet, or if the other site would advertise it too.

It reads the site’s route map and adds the subnet to the prefix list behind it. It turns Blue’s gateway off, then Green’s on. It then reads the leaves every five seconds, for up to 90, printing where each stands every 15, until each has learned the route through Green’s AS. A rollback removes only the prefix-list entry the cutover added. On the rehearsal segment, the measured gap while the gateway moved was about 3.4 seconds.

An unbridge refuses until Green is the gateway and no VM is left on Blue. Removing a bridge earlier would strand every VM still using Blue’s gateway.

Guard rails for the night shift

  • --test limits a run to the site’s test segment, whatever the production scope says. On Blue, it may write only that segment, its bridge and the bridge edges.
  • blue_writes per site decides whether a run may touch Blue’s bridges at all. The setting is test while a site rehearses, and true from the change window that opens production.
  • HELD. Outside --test, a run that would change NSX itself (segments, bridges, tunnel pools) does so only with --expect and the digest of a reviewed plan. Otherwise those rows are HELD, and the VMs of a wave wait. A night of waves never builds or takes apart anything by accident.
  • --set writes one config value by its full name. Passwords are asked for twice and never shown. The new file must read back with only that value changed, and the old file is kept beside it.

When the manager’s own certificate has expired

At both sites, the first attempt to deploy Blue’s bridge edges failed with OVF certificate validation failed: certificate has expired. The cause was not the edges. Blue’s NSX Manager downloads the edge image from itself over HTTPS, and its own API and cluster certificates had expired (Broadcom KB 408930). nsx-cert-fix replaced them one node at a time and deleted the failed deployments, and the edge build ran again. That tool, and the one extra step a vSphere Supervisor needs afterwards, are covered in When the NSX Manager Certificate Is Already Expired.

Testing without touching production

The tool is tested against stand-ins. Mock Global and Local Managers, mock vCenters (REST and SOAP) and fake SSH targets for the switches together reproduce how the real systems answer and fail. Five end-to-end flows step forward through every step and back again, including waves, returns, sign-ins per site, and held changes. Whenever production found a gap, it became a test first and a fix second.

How It Went

First site.

  • Rehearsal: the cycle on a test segment with three test VMs: build, move, cutover and every way back.
  • Pass 1: every Green segment of the scope was created with Blue untouched.
  • Pass 2: in a change window, Blue’s bridges were added to all 25 segments.
  • Waves: waves of 5 to 13 VMs followed, up to a 5.2 TB VM in one move. The first five waves moved 50 VMs, about 15 TB.
  • Held back: four file servers of more than 20 TB each are waiting for a wave of their own.
  • Folders: the folder feature was released the same morning the customer asked for it, in the middle of the waves.

Second site. On the first day, the team:

  • configured the transit VLANs on the leaves
  • built the bridge edges, finding and fixing the expired-certificate problem on the way
  • bridged the test segment and moved the test VMs

On the second morning they ran the cutover, unbridge and full rollback themselves, from a one-page command sheet.

Lessons Learned

Most of the guard rails above exist because something nearly went wrong first.

  1. Make the safe path the default. Plans by default, digest-pinned commits, and HELD for production builds. During one rehearsal the Blue-write switch was left on all day, and a plain commit would have bridged the whole production scope. Every commit that day happened to be a --test run. Now it can’t happen by accident.
  2. Tell the operator the one command forward and the one command back. Once, a rollback was run where an unbridge was meant, and a silent 90-second route wait looked like a hang. NEXT, UNDO and a progress line every 15 seconds ended both problems.
  3. Prefer one-shot commands to config toggles. Returning VMs used to mean listing them in the config. Someone forgot to clear the list, and the next morning’s move was held. --return --vm wave.csv now does it in one command and leaves the config alone.
  4. Check the underlay before blaming the overlay. A tunnel pool with no gateway, router MACs that must differ between Blue and Green, and expired manager certificates all looked like migration problems. Each is now a pre-flight check, or a script.
  5. Choose canaries that matter. Two production probes turned out to be idle standby domain controllers that were themselves being moved. Their own vMotion blip would have failed the run for nothing.
  6. Give very large VMs their own wave. Allow a long per-VM wait, and run the commit in tmux so a dropped SSH session can’t cut it short.
  7. Keep one state per site, in one folder. Running a wave from a copied folder split the record of where VMs came from, and returns rely on that record.
  8. Never try a password on a system it wasn’t set for. Sign-ins are per site and per system, and a prompt names the exact setting to save. A shared password once cost a failed login on the other site’s manager on every run, and NSX locks an account after five.
  9. Respect the customer’s structure. Folders, tags and placement carry meaning. Recreating them on Green was worth a release.

Getting It

git clone https://github.com/noahfarshad/fleetparity.git
cd fleetparity
bash test/run_tests.sh                                   # eight mocks on 127.0.0.1, five flows, about four minutes
cp examples/env_example.yaml env.yaml && chmod 600 env.yaml

The test run is the first safe thing to do: it needs no NSX, no vCenter and no switch. examples/env_example.yaml is a two-site config with every setting explained, and the README walks the rehearsal on a test segment before anything touches production. Every name, address and AS number in the examples is a placeholder. Nothing is written without --commit.

Closing

This approach is a lot of tooling for “move some VMs”. What it bought was a migration where every night looked the same: a plan, a commit, NEXT and UNDO, and nothing on Blue changed outside the subnet in hand. The network team could run it after hours without the people who wrote it on the call, which was the goal from day one.

References

Repository: github.com/noahfarshad/fleetparity (release 1.24.1)

Related Stories:

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top