Backups: three copies, one missing proof

2026-08-22 · depth: deep dive · 8 min · storage and decision

The apartment datacenter measured against 3-2-1: what is backed up, where it lands, what watches it, and the gap — a full restore drill that has not happened yet. The backup that silently stopped for thirteen days is the reason the monitoring exists. Since September the archives go to a dedicated backup server that deduplicates them, re-reads them to prove they are still readable, and takes them encrypted, so the storage box underneath holds copies it cannot open — which moves the hardest open question from ciphertext to who holds the key. As of 2026-09-08.

The rule everyone quotes is 3-2-1: three copies of the data, on two kinds of media, one of them somewhere else. Each number answers a different way of losing the data — the copy that corrupts, the shelf that fails as a batch, the building that burns. What the rule counts is copies, not proof: whether any of the three has ever been opened again is the question this note ends on.

What is backed up

Almost every guest, nightly. Since September the archives no longer go straight onto a network share. They go to a dedicated backup server running beside the rest of the fleet, which deduplicates them, keeps the last five per guest, and re-reads and checksums what it holds on a schedule of its own — so “the copy exists” and “the copy is readable” stopped being the same claim. The bytes still land on the storage box underneath — the NAS, which is where the space is. What changed is that they are encrypted on the node before they leave it, so the NAS holds archives it cannot read.

The backup server is the one guest that cannot use it: an archive of the backup server, stored inside the backup server, is not a backup. It writes its own nightly copy to the share directly, and that one archive is the declared exception rather than an oversight.

One machine is deliberately left out of the nightly run altogether. It stays out until its secret handling matches the rest of the fleet, because the rule here is that no backup store holds a secret in the clear.

The nightly run is five jobs, not one. They start one after another in the small hours, so two dumps never write to the same share at once. Each slot is as wide as the slowest run measured over a recent stretch of nights, plus margin — not as wide as a good night. Every job runs in snapshot mode, on storage that supports snapshots — which is the fix for something learned the hard way: a guest the cluster’s HA layer had taken over silently stopped backing up the day it joined, while its neighbour in the same job kept working. Two of the five are pinned to a node and three are not, and that is also a lesson: an unpinned job fires on every node at once, and the concurrency is exactly what the staggered slots exist to avoid.

The backup jobs live in git. A backup schedule that exists only in the Proxmox GUI is a schedule that a rebuilt node comes up without — and nothing fails, the backups just stop. So the job list is declared in the hub repository and converged on every deploy, adopting the GUI-generated job IDs verbatim so that no guest is ever backed up twice. If the cluster carries a job that git does not know about, the deploy fails loudly rather than deleting it: an undeclared job might be the only backup of something.

Home Assistant backs itself up to a dedicated share on the NAS, and the container mounts that share so a restore is a native operation rather than an upload. Before the hub was allowed to move to another machine, one of those off-box backups was decrypted and booted in isolation, so this is a restore that has actually been performed. It is also the half of a rebuild that git cannot supply: the hub’s own deploy path carries the reviewed configuration, and this backup carries everything that path never sees.

The secrets vault takes a nightly snapshot to the NAS, riding the always-armed schedule that also runs the drift check. Its restore job has actually run too, most recently on 2026-08-20.

Two more streams run on the NAS itself. Hyper Backup replicates from the NAS to a target in another building; Active Backup for Business backs up the household workstation to the NAS. Neither is declared in the repository or converged by a deploy — both are set up in the NAS’s own interface, by hand, which puts them in the same class as a backup schedule that lives only in a GUI and carries the same exposure. What the off-site job reaches is not established here: the third copy counts as a third copy, not yet as cover for any particular thing.

The firewalls and switches are code, so their backup is the repository plus the generated network model.

What watches it

The failure that shaped this section: between 2026-07-13 and 2026-07-26 the nightly backup did not run, and nothing said so. Proxmox mails on failure, but a job that never starts does not fail. A healthy fleet and a fleet with no backups looked identical on every dashboard.

So every node now publishes the age of the newest archive per guest as a metric, read from the Proxmox API against the same shared storage. A guest with no archive at all is published as infinitely old — “never backed up” alerts exactly like “stale”, which means a new guest alerts until the next night’s run. That is deliberate; an un-backed-up guest is the condition. The collector refuses to write a partial file: if the API errors, the previous file goes stale and the freshness alert fires instead of a guest quietly disappearing from the list.

The smart-home hub applies the same rule to itself: its own nightly copy is judged by the entities it keeps about the outcome, never by the call having returned.

Counting, honestly

That last row is the one that governs the plan. A backup that has never been restored is a hopeful copy, so the whole-fleet restore proof stays an open gate. One guest has now been pulled back out of the encrypted store, which proves the encryption and the tooling; it does not prove the fleet.

The off-site row changed recently and the restore row did not. A copy in another building raises the count the day it is written; it raises the confidence only on the day something is pulled back out of it.

The table asks about copies, media and distance and says nothing at all about encryption — which is the row this setup would now score best on, and the row where the remaining work is not ciphertext but custody. The key that opens those archives is held in the fleet’s own secrets vault. A copy on paper, off the fleet, is owed: a key kept only inside the thing it unlocks is not yet a key you have.

What is planned

• Custody for the archive key, off the fleet and on paper. Losing the NAS and the cluster together, with the key only in the vault, would leave every archive unreadable — the failure the encryption was supposed to make survivable, arriving from the other direction.
• The remaining cleartext. Getting every backup store to hold ciphertext only is still the largest open backup workstream; the guest archives are done, other stores are not.
• The restore drill itself. Snapshot, export, wipe, restore, then count and value checks, driven from a path that does not depend on the NAS — because the drill has to work on the day the NAS is the casualty.
• An off-site copy of the vault export with a defined refresh point. The export job runs; stashing it off-site is a human act, and a human act without a recorded date counts as unproven — and whether the NAS’s own off-site job already carries it is one of the things not established.

None of it waits on hardware: what the list is short of is acts performed and dates recorded.

Job declarations, monitoring and plan checked against the hub repository, the Proxmox API and the backup-age metric on 2026-09-08; the archive store, its retention and its verification state read from the cluster the same day. The two backup jobs that run on the NAS itself — the off-site replication and the workstation backup — are as stated by the operator on 2026-09-09; they are configured in the NAS’s own interface and were not read from the repository, and what the off-site job reaches is not established. Planned items are marked as planned.

Context: the architecture note →