Backups: three copies, one restore test missing
A backup stopped for thirteen days before anyone noticed. The fleet now checks the age of its copies as well as whether they can be read. Encrypted archives sit on the third node, with another copy on the NAS; a full-fleet recovery test and an archive key outside the fleet are still missing.
The rule everyone quotes is 3-2-1: three copies of the data, on two kinds of media, one of them somewhere else. Each number answers a different way of losing the data — the copy that corrupts, the shelf that fails as a batch, the building that burns.
What is backed up
Almost every guest, nightly. Since September the archives no longer go straight onto a network share. They go to a dedicated backup server running beside the rest of the fleet, which deduplicates them, keeps the last five per guest, and re-reads and checksums what it holds on a schedule of its own — so “the copy exists” and “the copy is readable” stopped being the same claim. Since 14 September the bytes land on the third node’s drive vault, an encrypted five-drive pool, and no longer on the NAS. The NAS pulls a second copy from the backup server every morning. The archives are encrypted on the node before they leave it, so the NAS holds archives it cannot read.
The backup server is the one guest that cannot use it: an archive of the backup server, stored inside the backup server, is not a backup. It writes its own nightly copy to the share directly, and that one archive is the declared exception rather than an oversight.
One machine is deliberately left out of the nightly run altogether. It stays out until its secret handling matches the rest of the fleet, because the rule here is that no backup store holds a secret in the clear.
The nightly run is five jobs, not one. They start one after another in the small hours, so two dumps never write to the same share at once. Each slot is as wide as the slowest run measured over a recent stretch of nights, plus margin — not as wide as a good night. Every job runs in snapshot mode, on storage that supports snapshots — which is the fix for something learned the hard way: a guest the cluster’s HA layer had taken over silently stopped backing up the day it joined, while its neighbour in the same job kept working. Two of the five are pinned to a node and three are not, and that is also a lesson: an unpinned job fires on every node at once, and the concurrency is exactly what the staggered slots exist to avoid.
The backup jobs live in git. A backup schedule that exists only in the Proxmox GUI is a schedule that a rebuilt node comes up without — and nothing fails, the backups just stop. So the job list is declared in the hub repository and converged on every deploy, adopting the GUI-generated job IDs verbatim so that no guest is ever backed up twice. If the cluster carries a job that git does not know about, the deploy fails loudly rather than deleting it: an undeclared job might be the only backup of something.
Home Assistant backs itself up to a dedicated share on the NAS, and the container mounts that share so a restore is a native operation rather than an upload. Before the hub was allowed to move to another machine, one of those off-box backups was decrypted and booted in isolation. It is also the half of a rebuild that git cannot supply: the hub’s own deploy path carries the configuration from git, and this backup carries everything that path never sees.
The secrets vault takes a nightly snapshot to the NAS, riding the always-armed schedule that also runs the drift check. Its restore job has actually run too, most recently on 2026-08-20.
Two more streams run on the NAS itself. Hyper Backup replicates from the NAS to a target in another building; Active Backup for Business backs up the household workstation to the NAS. Neither is declared in the repository or converged by a deploy — both are set up in the NAS’s own interface, by hand, which puts them in the same class as a backup schedule that lives only in a GUI and carries the same exposure. What the off-site job reaches is not established here: the third copy counts as a third copy, not yet as cover for any particular thing.
The firewalls and switches are code, so their backup is the repository plus the generated network model.
What watches it
Between 2026-07-13 and 2026-07-26 the nightly backup did not run, and nothing said so. Proxmox mails on failure, but a job that never starts does not fail. A healthy fleet and a fleet with no backups looked identical on every dashboard.
So every node now publishes the age of the newest archive per guest as a metric, read from the Proxmox API against the same shared storage. A guest with no archive at all is published as infinitely old — “never backed up” alerts exactly like “stale”, which means a new guest alerts until the next night’s run. That is deliberate; an un-backed-up guest is the condition. The collector refuses to write a partial file: if the API errors, the last file stays as it is, and after an hour without an update an alert fires instead of a guest quietly disappearing from the list.
The smart-home hub applies the same rule to itself: its own nightly copy is judged by the entities it keeps about the outcome, never by the call having returned.
Copies and restore tests
| 3-2-1 asks for | What exists |
|---|---|
| Three copies | Three, as far as the off-site job reaches: the guest’s own disk, the nightly archive on the third node’s vault with its second copy on the NAS, and the copy Hyper Backup writes to another building |
| Two kinds of media | Yes: node SSDs on one side, the hard drives in the third node on the other |
| One copy off-site | Yes: Hyper Backup replicates from the NAS to a target in another building — set up on the NAS by hand rather than declared in the repository, and no restore from it is on record |
| A tested restore | Partly — the vault, the smart-home hub, and one guest pulled back out of the encrypted store; the full-fleet recovery test is still outstanding |
What is planned
- Custody for the archive key, off the fleet and on paper. Losing the NAS and the cluster together, with the key only in the vault, would leave every archive unreadable.
- The remaining cleartext. The guest archives are encrypted; other backup stores still need encryption.
- The restore drill itself. Snapshot, export, wipe, restore, then count and value checks, driven from a path that does not depend on the NAS — because the drill has to work on the day the NAS is the casualty.
- An off-site copy of the vault export with a defined refresh point. The export job runs, but whether the NAS off-site job includes the export has not been checked.
Schedules, retention and monitoring were checked against the configuration, Proxmox API and backup-age metric on 8 September 2026. The NAS off-site and workstation jobs were described by the operator on 9 September; their full coverage was not checked. The storage move of 14 September was added later and is not covered by the earlier check date.