The third vote
A container inside a network switch supplied the third vote for a two-node cluster. It took a week to build and ran for about sixteen hours. Then a third computer arrived and made it unnecessary.
Why two is the wrong number
The cluster software keeps acting only while it holds a majority of the votes. With two members that is fragile: when one goes down, the survivor holds one vote of two, which is not a majority, and it stops rather than risk acting alone. The fix is a third vote that is not a third computer — a small program somewhere else that answers “you are the majority” to whichever member can still reach it. For a year that program ran in a container on the NAS.
Off the NAS
On 8 September the vote was ruled off the NAS. The reason was concentration, not availability: the NAS already carried every storage tier, every backup lane and the identity provider behind the cluster login, and the vote was the cheapest of those to take away. Availability was not on offer either way — the NAS’s leg and the new seat both bond into the core switch, so losing that chassis takes the members’ sight of each other with it, whichever box holds the vote.
Into the switch
The access switch is a small ARM box, and its operating system can run containers. Getting one vote-sized program to run there took a list, and every line on it had to be true before the next one could be tried:
- An image built for the switch’s processor family — on a build machine of another kind and without an emulator, so the image is assembled from layers and nothing is executed during the build. The build fails if the result is not the switch’s architecture.
- A stick in the switch’s side, formatted from the pipeline with the stick’s serial number named in the command, so a stale confirmation cannot wipe a different medium.
- Container support enabled in device mode, confirmed with a physical press of the switch’s reset button.
- A virtual cable into the management VLAN, with a fixed address and a set of filter rules that let the two members’ address lookups and the one vote connection reach the container and nothing else. Measured on the way: frames leaving the container are already VLAN-wrapped when the filter sees them, so that direction can be pinned to the VLAN but not to the protocol.
- The certificate exchange. Normally the cluster logs into the vote host over SSH to swap certificates; a container on a switch has no SSH. So the request and the signed answer crossed through a folder on the stick, copied over the switch’s own file transfer against a pinned host key.
- The clock. The switch’s clock ran ahead of the nodes, and a certificate minted in the future is “not yet valid” on the members. The clock is read before the exchange, and the difference printed.
Every step that changes the switch runs as a dry run first and prints what it would do; one confirmation variable makes it real.
It voted
In the small hours of 12 September both members connected to the container and the cluster’s vote count read three, with the third answering from the switch. The NAS’s old seat was retired the same night. For the rest of that day, the tie-breaker of a two-node cluster lived on the access switch, and the cluster was quorate through it.
And then it was over
That afternoon the third computer joined the cluster. Three members are three votes; a tie can no longer happen, and a seat that only votes has no reason left. The join removed the vote from the cluster’s configuration in the same window — the cluster software keeps a phantom row with zero votes until it restarts on a member, which is harmless. The container had voted for about sixteen hours.
The container, its virtual cable, its filter rules, its name in the registry and its firewall rule are removed. The stick and the switch’s container package stay, unused.