The apartment datacenter

2026-08-21 · depth: the architecture · 17 min · architecture and decision

How one person runs a private datacenter: git records intent, CI gates every infrastructure change, AI agents do the typing — and a human keeps merge and apply authority. What is live, what is target, and why the design ended up this way. Checked against the running system on 2026-09-08; what is planned is called planned.

TycStation keeps the smart home, media, websites, secrets and much of the network of one Cologne apartment under local control. The unusual part is not the hardware; it is the operating model. Live verification, not a merged change, decides whether something actually worked, and the household keeps functioning when the internet does not. This note answers three questions: what runs where, who is allowed to change it, and how anyone knows a change worked.

What it runs

• Smart home — Home Assistant and the household automations run as containers on a dedicated VM; the devices reach it over Zigbee, Thread, Matter and MQTT. The hub’s own voice assistant runs locally; the Amazon Echo speakers still in use are cloud devices by construction.
• Media — Plex serves the household library; bulk media stays on the NAS.
• Sites and ingress — Traefik serves websites (including this one) and selected public services. Public ingress is intentional and IPv4 only — a decision taken in September, not a stage still waiting on address-plan work.
• Identity and secrets — OpenBao issues short-lived, per-repository CI credentials through GitLab OIDC: the job proves which repository it runs for and is handed a credential for that alone, instead of carrying a stored password.
• Storage and recovery — the Synology NAS supplies shared storage, and a dedicated backup server now takes the nightly archives and encrypts them before they leave the node; a fleet-wide restore-proven drill is still a gate on the road ahead.
• Observability — Grafana, Loki and Prometheus are live. Fleet coverage is expanding and not yet complete.

The physical footprint is one vertical mini rack’s worth of ordinary gear — the hardware section below lists it.

The hardware

The constraint that shaped this list is the noise floor. This build grew up in the gap between a desk and a hi-fi tower, and a machine that whines is disqualified before its spec sheet is read. That rules out rackmount chassis and their 40 mm fans whatever they cost per core, and it pushes the whole build toward small, low-wattage machines — heat is what forces fans.

The room the rack shares: the gear sits in the gap between the desk and the hi-fi tower.

October 2025, before the rebuild: the original mini rack in that same gap.

July 2026: the three MikroTik switches arrive and get their first cabling on the desk — the core switch on top, the PoE and access switches beneath.

The switches arrived carrying the same problem in miniature: 40 mm fans, the exact part the noise budget had already ruled out at chassis scale. They were opened on the bench the day they came out of their boxes and refitted with Noctua NF-A4x20 PWM fans — a 40 × 20 mm four-pin part — before any of them was cabled in.

It is not a clean drop-in everywhere, which is the part worth knowing. The stock fans are not all the same depth: the deeper ones are 40 × 28 mm, and against those the Noctua is eight millimetres shallower. It lands on the same frame and the same screw pattern, but it does not fill the housing, so that one is a partial fit rather than a straight swap.

July 2026: both switches open on the day they arrived. The fans sit at the edge of the board, next to the heatsink they are there to cool.

23 August 2026, the night this page was revised: the fabric is in, mounted vertically in the same gap as before — and not yet fully installed. Cabling and the final mount are still to come.

Two things this list does not have. First, no noise figure — that has never been measured; power draw is metered continuously, but per-device idle figures are not broken out here. Second, there are no purchase links.

Why it ended up this way

Each layer answers a failure, limitation or recovery gap observed in the earlier system.

• One failed gateway took the apartment offline. → One fixed OPNsense VM per physical node. CARP and pfsync protect the service everything else depends on: the two firewalls answer to one shared address, and each carries a copy of the other’s open connections. The single DSL line still redials.
• The first segmentation model became fragile and leaked at layer 2. → Eight routed zones with policy at the edge: fewer risk classes, one VLAN each, explicit flows, and generated cross-plane invariants.
• Consumer-switch control fought the “git owns intent” rule and hit a PoE ceiling. → A RouterOS fabric; keep only the UniFi access point. MikroTik is fully code-driven and supplies the required PoE++; Wi-Fi stays with the stronger radio platform.
• Long-lived automation credentials create silent, fleet-wide risk. → OpenBao plus per-repository GitLab OIDC. Jobs receive short-lived, narrowly scoped tokens; plan and apply material stay separate.
• Home Assistant OS add-ons could not be pinned and rebuilt like ordinary services. → Home Assistant and its companions moved to containers, versioned as code; the instance validates and pulls main itself.
• No restore had ever been proven. → Backup verification and a full recovery proof are gates, not afterthoughts.

Complexity was also rejected. No Kubernetes without a workload trigger, no firewall config authority outside git, no physical node reinstall, and no pretend seamless WAN failover.

View 01 — system context.

The operating principles

Privacy — local processing by default. Resilience — household services survive an ISP outage. Ownership — data and configuration stay recoverable. Craft — professional platform practice at apartment scale.

None of these patterns is unusual in a cloud platform: jobs that prove who they are instead of carrying a password; a floor plan of public, private and restricted zones; a firewall pair that shares its state so either half can take over; rules that live in git rather than in a console; a recovery that has actually been performed; and telemetry from everything. Four of the six are in place; full-stack observability is live but not yet fleet-wide, and a restore-proven recovery is still owed. Putting the set behind one apartment desk is the unusual bit.

The dates say when something was last checked.

What depends on what

View 02 — the stack. Layers show dependency, not rollout status; planned elements say so.

Layers

The tools sort into four lanes — which tool owns which concern. This map answers “which technologies?”; it deliberately does not imply placement — the deployment view does that.

Today versus the plan

Repository declarations prove intent, not rollout. As of the 2026-09-08 evidence snapshot:

What runs today

• Two physical Proxmox nodes carry the fleet, and the third cluster vote is a tie-breaker service on the NAS; the small helper member that used to hold it is retired.
• The two OPNsense edge VMs run one per physical node and own the WAN legs. They are a CARP + pfsync pair: both hold every address the network sends traffic to, firewall state syncs between them, and DHCP runs hot-standby — the second copy is loaded and current, not started when the first fails. On 2026-08-23 a real swap and swap-back moved the master role and its dial-up session with it.
• The eight-zone switch fabric is applied — the renumbered VLANs, the zone gateways and the inter-zone policy are what the network runs today.
• Guests are moving onto their zone addresses and storage tiers one at a time; each replacement coexists with its predecessor until a separate, reviewed retirement.
• High availability is armed, with twelve guests under it. Both drills passed on 2026-09-01. One node was rolled through a planned reboot — its guests drained to the peer beforehand — and came back with its own guests; the other was crashed outright, and every enrolled guest was running on the surviving node about two minutes later. Replication mirrors every repair-path guest between the nodes, the nightly drift checks run, and staged targets are expected differences, not defects.
• The nightly archives go to a dedicated backup server, which deduplicates them, verifies what it holds by re-reading it, and takes them encrypted on the node before they are written. The storage box still holds the bytes and can no longer read them. One guest has been restored out of that encrypted store; the fleet-wide drill has not been run.

What is planned

• The edge pair’s hard failure drills: WAN cable of the master pulled, the node holding master powered off, a trunk member down — each with a recorded time to failover. The graceful swap is proven; the hard reads are owed.
• The full restore proof, fleet-wide, and custody for the key that opens the archives: it is held in the fleet’s own vault, and a copy off the fleet is owed. A key kept only inside the thing it unlocks is not yet a key you have.
• The cluster’s third vote moves off the storage box and onto the access switch. The reason is concentration, not availability: quorum, every storage tier and every backup lane are the one appliance today, and the vote is the cheapest of those to move. Nothing answers at the new seat yet, and the old one serves until it does.
• The existing nodes converge in place: no reinstall, no rename.

View 03 — deployment; the edge pair, the fabric and the tie-breaker vote are live. Cluster names remain TYCPVE and Tyc2pve; the lowercase names are DNS role aliases.

WAN never crosses a switch: the modem’s two LAN legs cable directly to the two nodes, one edge VM inside each. The node bonds carry the LAN trunks; storage and the tie-breaker vote ride separate NAS legs.

A cloud-style landing zone, inside one apartment

View 04 — network and landing zone; guest moves in progress.

A landing zone is a network floor plan settled before anything moves in: which zones exist, what may cross between them, and who may change that. This is the same public/private/restricted pattern the cloud ones use, scaled down to one physical fabric. Every inter-zone path crosses the edge and is denied unless an explicit flow allows it. The published front door lives in Z-DMZ (Traefik) — the one zone the internet may reach, held apart so that a break-in there lands nowhere else. A few documented NAS resilience ports intentionally bypass that zone. Plex sits in the smart-home and AV zone with the devices it streams to. The private zones separate the household’s trusted hosts, smart-home devices, guests, and one “far side not ours” segment that gets internet and nothing lateral. The restricted zones carry the secrets vault and quorum, switch, AP and KVM management, and a camera zone. Stable internal identity rides a private ULA /48 — an IPv6 range generated for this network alone and never routed on the internet. The ISP prefix rotates; names and policy do not.

Honest limits: intra-zone traffic is not filtered, the trusted zone remains broad, and per-workload microsegmentation — a rule per workload rather than per zone — is a deliberately separate hardening phase after the migration completes.

How a change reaches reality

AI changes the typing capacity, not the authority. Agents discover, propose and review. A human approves merge and gated infrastructure mutation. Verification observes the resulting system.

View 05 — change delivery.

Change delivery

Infrastructure mutation is CI-only except for documented break-glass recovery: an emergency path written down in advance, for the cases where the pipeline cannot reach what is broken. A clean plan proves intent; acceptance against the live system proves completion. Home Assistant is the deliberate exception in transport: its container pulls and validates main itself, fail-closed — there is no CI push into the smart home.

View 06 — automation. Five OpenTofu states, one lock each.

Delivery and automation

An OpenTofu state is the record of what the tool believes it has built. Five of them — the virtualisation hub, the network fabric, the edge pair, DNS, and the Wi-Fi controller — are separate lock domains: independent changes run in parallel while each state stays serialised. The per-node edge states were retired on 2026-08-23, and the pipeline now refuses to run against them. Merge-request pipelines are read-only — plan and check only. A plan reads the shared state and the plan-side secrets; the credentials that can change anything are bound to the protected branch and reach only the apply job. Apply is a human pressing a button on an on-LAN runner, with short-lived OpenBao credentials scoped to exactly that job. Shared CI templates provide the defaults, but review and protected branches remain part of the control — a consumer pipeline can still weaken an inherited rule, so the guard is the branch rule, not the template.

Two guards run fail-closed. One scans the project’s entire git history for credentials, not just the change in front of it, and it runs on every pipeline in every repository; the other reads every infrastructure declaration for misconfiguration, in the repositories that carry one. Neither is allowed to warn instead of failing.

Verification is its own graded pass, not a look at the logs. Eleven invariants are read back off the switches themselves and graded one by one; a twelfth is checked against the source text before anything runs. The run then prints its own blind spots beside the results. A verifier that reaches the switches only through the plane it audits cannot see a break in that plane. The graders themselves run against a synthetic network with a fault seeded for each invariant, so a grader that has stopped noticing gets caught.

Drift is checked in two directions. One comparison holds the running world against the record and catches a change made around the pipeline; the other holds the record against itself and catches a change that was merged and then never applied, which is a different failure and needs its own alarm. Neither is allowed to soft-fail — a drift check that warns instead of failing is a false green.

The pipelines are shaped differently from repository to repository, and none of them carries every stage — a repository runs the ones its content needs. The kinds of check, and the jobs that carry them:

The migration: build the target without reinstalling the nodes

The sequence is deliberately staged and reversible until final retirement, in four stages: establish the new substrate, converge the existing nodes in place, prove recovery and failover, then seal.

View 07 — the migration; the substrate stage is done and guest moves are in progress, high availability is armed and both cluster drills passed, the restore drill is still owed. No physical node reinstall and no node rename.

Migration plan

As of 2026-09-08 the substrate stage is done: console access and backups were proven before anything mutated, the fabric renumber is applied, both replacement firewalls are live with the WAN recabled directly to the nodes, and the shared storage plane exists. Guests are moving in recovery-tier order. Then each node is drained and converged behind its working firewall peer — the helper member is already retired and the tie-breaker vote is live. High availability is armed and both drills have passed; the whole-fleet restore drill is still owed. Rollback assets are retired only after a stability soak, and that step is the only irreversible one.

The decisions that define the platform

• Git records intent; observation decides truth. Plans and documentation never substitute for live acceptance evidence.
• The edge is a fixed pair, not a floating guest. One OPNsense VM per node; CARP and pfsync move the service, not the VM. One firewall is the configuration authority and pushes every synced section to its partner over the vendor’s own sync channel; git is the only origin of that configuration, and nothing is ever authored on the standby.
• Storage follows recovery tier. Repair-path guests stay NAS-independent on replicated local ZFS; suitable VMs use shared NFS.
• ULA carries identity and policy. The ISP prefix rotates; stable internal names and rules use a private ULA /48.
• The quorum vote is a vote, not a compute node. Two physical members plus a dedicated tie-breaker service — on the storage box today, ruled in September to move onto the access switch, which will carry it in a container of its own.
• Eight zones now; workload isolation later. Risk-class boundaries are fixed without pretending intra-zone microsegmentation already exists.

Accepted residuals

• One DSL line means firewall failover redials; it is node redundancy, not link redundancy.
• Simultaneous NAS and node loss removes quorum and needs confirmed-peer-down break-glass.
• Loss of the core switch chassis isolates switched traffic, but not the pair itself: two of the cluster’s three internal links and the firewalls' shared state ride cables that run directly between the nodes.
• Intra-zone traffic is not filtered and the trusted zone remains broad.
• Per-workload default-deny is a separate hardening phase after the migration completes.

Three incidents that changed the design

• Sole-edge outage — promoted the firewall pair from “later” to the plan of record.
• Docker on NFS — established explicit local-ZFS exceptions and storage tiers.
• Layer-2 leak — replaced fragile segmentation with eight routed risk zones and derived invariants.

Audited from the outside

In September the whole estate was audited against an enterprise rubric, from the outside in, and scored 45 of 132. The method, the cost and the places where the audit was wrong about itself are in 45 of 132; the findings themselves are not published.

Everything above was checked on 2026-09-08. What runs was read from the cluster’s own API and from the smart-home hub; the switch and firewall planes were read from the declarations CI applies to them, not from the devices themselves on that date. Planned items are marked as planned.

More field notes →