TycStation — field notes

The house decides, and no clock does

Tue, 08 Sep 2026 00:00:00 +0200

The smart home in this flat has no “leave” button and no bedtime on a wall clock. It derives five house states from a door, a phone and the presence sensors, and every other behaviour hangs off that one variable: the welcome light when someone comes home after dark, the lock, the air conditioning and the cameras when nobody is left, a vacuum that cleans only while the flat is empty, a bathroom fan that knows why it is running, a party that stays sound-reactive when the hub is down. This note goes through the rules and the reasons — a clock that cut a real daytime sleep short, a reminder that lit the bedroom seven times around a sleeping body, a fan stuck on boost for a day because a timer never fired — and is plain about what is not yet proven: the lock and the air conditioner have never been caught actually reacting to a real departure, and a lighting bridge still runs its own behaviours beside the hub in two rooms. As of 2026-09-08.

The goal is written down in one sentence: the flat is in the right state without being operated. Bright when someone comes home after dark, dark and quiet while someone sleeps, everything off and locked when nobody is left. The house derives all of that from sensors and the phone. Buttons exist only for intent that cannot be sensed — going to sleep now, cameras on or off. Nothing runs on a wall clock except the two things that genuinely are about the time of day: sunset, and how warm a welcome light should be.

Two facts, not a clock

In everyday termsThe house does not guess from the time of day whether you are out. It waits for the front door, then for your phone to walk away, then checks that nobody else moved for ten minutes — the way a colleague would notice you had actually left, not just gone to the kitchen.

The derivation. Four inputs, two facts, three outcomes — and no clock anywhere in it.

Two facts are derived independently, and neither is a timer.

I left is the phone leaving the home zone, or dropping out of Bluetooth range of the flat, while the front door has been closed for two minutes. Someone is still here is any room-presence sensor firing in the last ten minutes. The two combine into a table:

I left	Someone still here	The house
yes	no	Away: lights off, air conditioning off, door locked, cameras armed
yes	yes	Home without me: nothing global changes
no	—	Day, Evening or Sleep

Neither input alone would do. The door also opens for a guest or a parcel, and the phone also leaves the zone on a two-hundred-metre walk; both together, in order, describe a person leaving. And presence is checked over ten minutes rather than instantly because the room sensors let go of a person at very different speeds: the infrared ones after about ten seconds, the radars after a few minutes. A person sitting still in the living room can be invisible to all of them for a while.

One radar is deliberately not on that list. Its moving-target channel flapped tens of thousands of times a day in an empty flat, which kept “last movement” permanently under ten minutes and made Away unreachable. It was dropped from the presence inputs rather than thresholded, because its still channel cannot be thresholded either.

The calendar feeds the same decision. An all-day vacation entry is one more departure signal; a running home-office event refuses Away altogether, so a quiet afternoon at the desk does not lock the door and kill the air conditioning behind someone who is still there.

The five states and every event that moves between them. Party is the one with no way out, which is the open decision at the end of this note.

Arriving

The door opening on an empty flat, or the phone’s zone flipping to home, triggers the welcome — but only if it is dark, by sun or by a light sensor. The welcome is time-tiered: warm until nine in the evening, dimmed until eleven, a night light after that. If the bedroom is already asleep, only the hallway night scene runs. A restore-safe thirty-minute timer stops a second door cycle from firing it again.

The first cut of this lit the whole flat at full brightness at one in the morning. The cause was a Bluetooth “not home” reading that turned out to be false — the phone had never left — and the fix was not a better Bluetooth threshold but a different question: was any presence sensor quiet for the last ten minutes? Bluetooth is one leg of arrival and departure; it may not decide anything on its own.

The air conditioner joins on the way in: when the phone enters the home zone while the house is Away and the living room is above a threshold, cooling starts before the door opens. That one has been seen working on a real homecoming.

Sleep has a way out

In everyday termsA bedtime switch is easy. The hard part is who switches it back — because if nobody does, the bedroom’s motion light stays dead all day.

Sleep is the one state a sensor cannot derive, so it has buttons: a long press on the button by the bed puts the bedroom to sleep while the living room stays awake. A button on the wall pad puts the whole house to sleep. The bedroom has its own override — asleep, awake, or following the house — so one person can sleep while another is up.

What the mode actually does is deliberately narrow. Exactly one automation derives a single flag from it, and that flag is the only night guard on the bedroom’s motion light: by day, motion lights the room; at night it does not. The rule written beside it is that the flag must always have exactly one writer — with none, the feature is silently dead.

The exit is where the design shows. Sleep ends when the bedroom stands empty for forty-five minutes, or five minutes after the phone’s alarm has rung. There is no clock exit, because a noon clear-out cut a real daytime sleep short on 2026-08-25. And the alarm only arms the exit; it does not perform it. An alarm ringing means an alarm rang, not that anyone got up. A recurring late-evening reminder on the phone cleared the flag every night. On 2026-09-06 the radar then lit the whole bedroom seven times between roughly half past five and a quarter to nine, around a body that had never left the bed. Since then a body the radar loses for a moment keeps the room asleep; only an empty bedroom decides.

What Away means

In everyday termsLocking up is the easy half. The hard half is being sure the flat is really empty before it happens — and knowing, afterwards, whether the lock actually turned or was simply shut already.

Away turns the lights off, the air conditioning off, locks the door and arms the cameras. Two of those legs have never been caught in the act: on the first real departure the lock was already locked and the air conditioner already off, so the automation had nothing to do. The derivation is proven from the recorder — a real departure produced Away correctly and no false Away has followed — but the actuators are, as of this writing, untested on a real evening.

What Away actually drives. The two dashed legs are the ones that have never been caught in the act.

The vacuum cleans on Away, once a day, in daylight hours, and only if it is docked and both of its own error sensors read clean — a fault is never a reason to send it out again. Anyone coming home, or the house going to Sleep, sends it back to the dock from wherever it is. Its do-not-disturb window is the house’s, too: on Sleep the automation saves the device’s own begin and end times, moves the window to now, and restores the saved window on exit. The vendor’s switch only arms a schedule — it does not silence the dock emptying itself in the night. Faults reach the phone once per class, silent while the house is Away and said once on the way back in if they outlasted the trip.

The cameras are the split described in the tour note: the recorder on the storage box keeps recording, an object detector in its own guest watches the same streams, and the hub decides what a detection means. That deciding half is small and has three rules.

A camera whose smart plug is about to open is disabled in the detector first, and re-enabled two minutes after the plug closes, so a power cycle never reads as a lost stream. Which plug powers which camera was measured, not read off the names — one plug labelled “kitchen” in the interface sits under a dining-table camera. One camera has no plug and cannot be gated.
A person detected while the house is Away raises one push and one mail with the recorder’s frame attached and, where the detector recognised a face, the name leads the text. The alert fires off the detector’s raw event stream rather than its reviewed alerts, because reviewed alerts exist only for cameras the detector records, and it records nothing here. One push per tracked object falls out of the payload itself — the zone list an object has entered only ever grows, so the condition has exactly one edge.
The balcony camera alerts only for a person inside a drawn zone along the parapet; the street beyond it is masked. Before the re-aim that camera raised three quarters of all detections in the fleet, most of them a fingernail of frame for eight seconds. After it: a handful a day, and a walk onto the balcony raises exactly one push.

The recorder’s own Home Mode is left alone. It arms nothing on this installation, and the writer the hub was racing turned out to be the vendor app’s geofence on the same phone, which won every transition by minutes. Mirroring the house state onto it bought nothing, so the mirror was dropped rather than defended.

The robot vacuum, for the record, is never detected as a person. Across an hour of cleaning the cameras raised seventy-odd events and every one of them was taller than wide — people. A disc on the floor is not.

Air

In everyday termsThe bathroom fan used to be two light switches that a phone app happened to control. Now it is a fan that knows why it is running and how long it has left.

The bathroom’s two-stage exhaust fan is two relays. The radio stack had created them as lights, because the relay advertises itself as one. They are now overridden to switches and wrapped into a single fan entity with two levels — base ventilation, which is meant to run always, and boost. Boost has named causes: the dryer drawing power, the bathroom light on, or the humidity sensor above a threshold. A thirty-minute timer releases it, re-armed every five minutes while a cause holds, and the timer survives a restart of the hub, so a shower run that spans one resumes its countdown instead of losing it.

That last property is why the package exists. On 2026-09-06 boost was found to have been on since the previous night’s restart, for no reason at all. The interface-built automation that released it used a “condition true for thirty minutes” trigger, which never fires when the condition is already true at start — so every restart that landed mid-quiet left the boost on for good. The hub restarts about twice a day when it pulls its own configuration; every timer in the house has to survive that, or it is not a timer.

Two more relays that were “lights” went the same way: an amplifier’s mains relay and a momentary relay that pulses a PC’s power button, which is now a button rather than a light that switches itself off after a second.

The rest of the air story is measurement before action. Every room’s carbon dioxide, particulate and humidity sensor sits on a floor-plan dashboard. Two air purifiers run firmware written here after the vendor’s servers went dark; one stands on the balcony and blows through the wall, so it is the bedroom’s supply air and the only mechanical air exchange the flat has. The humidity channel on the built sensor nodes is not consumed by any rule: a paired comparison put it well off a reference in the same room, and until that is understood, humidity logic reads the other instruments. Demand control of the supply fan on the bedroom’s carbon dioxide is the next step, and it is not built.

The purifier whose manufacturer went dark, running firmware built in the flat. The readings on its face are the ones it publishes to the house broker.

Alarms that may not be missed

In everyday termsA sensor that has gone quiet and a sensor with nothing new to say look identical from the outside. The flat tracks when each one last spoke, not only what it last said, so a dead battery cannot pass for a plant that is fine.

Two leak sensors sit where a leak costs the most: under the kitchen sink and under a water-cooled PC. A leak speaks on the phone at alarm volume through silent mode, leaves a sticky notification, announces on the voice satellite and mails as the record. It repeats every two minutes until the sensor is dry, for about an hour of repeats at most, then clears its own notification. A sensor that stops reporting is its own alert. The PC’s power is deliberately not cut: a forced power-off of a running machine is a person’s call.

Two soil probes guard two plants. A dry reading has to hold for an hour before it alerts — one low sample after watering must not undo a fresh reading — and then it reminds once a day. The probes sleep and report only on a meaningful change, so a value can sit unchanged for days. The staleness alert therefore keys on when the probe last reported, not when the value last changed — a forced resend proved those are two different things. One probe needed a fix to the device library that had not shipped upstream yet; it is carried locally.

Entity health is counted the same way: unavailable entities only, never “unknown”, per integration rather than as one flat number, because a flat count is deaf to a mid-sized integration going dark. And the nightly off-box backup reports its outcome from the two entities the hub maintains about it, never from “the call returned” — the backup note has the thirteen silent days that rule comes from.

A party that works without the hub

In everyday termsNobody is running the disco from a laptop. Each lamp listens to the room the way a person taps a foot to the music, so nothing stops when the machine in the cupboard does.

The fifth house state had a button and no behaviour for weeks. What it does now rests on one fact read from the devices: every lamp in the room that can react to sound has its own microphone. The dodecahedron lamps and the moon lamp run a sound-reactive firmware effect on their own; the curtain of lights has seven microphone modes of its own; the two sculptural lamps sit in a mode where music sync is forced on. Arming a party sets each device to its own sound-reactive mode and leaves it alone. No audio is streamed anywhere, nothing analyses the room centrally, and a party keeps going if the hub falls over mid-song.

The one device class that cannot hear is the lighting bridge, and the hub drives it — carefully. The bridge holds the room; it does not keep the beat: a deep dim wash that never changes across the living room, and colour moving with crossfades across six accent fixtures only. The first cut switched every ceiling light to a new saturated colour every second and a half and looked like cheap blinking RGB; the beat belongs to the devices that can actually hear it. The bridge’s own “dynamic” scenes were measured live and do not move, so they are not used.

Four things run around that. Every touched light is snapshotted on arm and restored on disarm, per lamp rather than per group. The door-open alert and the device-offline notifiers go quiet, because guests are in and out. The television’s light-sync box switches off: the bridge streams one session at a time, and the party wins. And the fog machine gets power on a hard two-hour cap — the hub can switch it on, it cannot make fog.

A keep-alive re-enters the colour loop every half hour because the hub aborts a loop at ten thousand iterations, which at party pace is under three hours — an ordinary evening.

Party has no automatic exit, and that is recorded as a decision the house still owes rather than a bug. Every other mode here has a deterministic way out, and this house’s own rule is that a mode needs one. Whether the exit is “flat empty for half an hour” or “sunrise” is a decision about the house, not about the code.

Buttons that say when they are wrong

The dual-mode wireless buttons and knobs in the flat can each run in two firmware modes, and a hidden triple-click flips them. In the wrong mode a knob sends one event every second and a half instead of a command in a quarter of one. Command mode is the standard. An announcer watches for event-mode traffic from any of the six units, spares included, and says so out loud — so a knob that was flipped in a drawer announces itself rather than feeling broken for a week.

The desk knob is the volume of the room’s audio processor: a scaled step per detent into a target, an immediate first write and a quarter-second follow cadence, so a fast twist does not collapse into one step. Press is play/pause, double-press is mute.

What is not proven, and what is planned

The actuator leg of Away — the lock and the air conditioner reacting to a real departure. Both were already in their target state on the departures logged so far.
A late arrival lighting the flat before the door closes on a real evening; the one logged arrival found the lights already on.
The Bluetooth walk-through that would qualify room-level presence as more than a corroborating input.
The lighting bridge’s own behaviours still run on a wall clock in the hallway and the bathroom while the hub drives the same lamps on arrival — the one arrangement this house rules out. Retiring them is a step inside the vendor’s app, tied to one pass over every button, dial and pad.
A power cycle of each camera plug with a person watching, to confirm the measured plug-to-camera binding against the interlock.
Party’s exit, above.
The hub’s own sleep signal from the watch, which needs a permission granted on the phone.

What those items have in common is the distance between a rule and a sighting. The derivation itself is proven on a real departure. Most of what hangs off it has so far only been reasoned about — and a house that is right in the recorder is not yet a house that has been seen being right.

Every automation named here was read from the running hub’s automation list on 2026-09-08 and its rules from the smart-home configuration repository on the same date; the incidents carry the dates recorded beside the rules they changed. Planned items are marked as planned.

45 of 132, and what the score was worth

Tue, 08 Sep 2026 00:00:00 +0200

In September this fleet was audited against an enterprise rubric: every repository, every pipeline, the live state of both nodes, the firewalls, the storage box, the smart-home hub and every published site. It scored 45 points out of 132, with a median criterion score of 1 and five of nine critical rows failing. This note is about the method, the cost — seventy-two and a half million tokens measured across 358 agent runs, plus two passes whose accounting is missing entirely — and about the part worth more than the number: what the audit got wrong about itself. One in eight findings the first pass wrote down did not survive being re-read. Of 86 critical findings re-checked against live state, one was simply false. And the headline verdict was an artefact of the rubric, which asks for a second reviewer and an on-call rota that a one-person household fails by construction. As of 2026-09-08.

Everything else on this site is written from the inside. This one was written by something that was told to assume none of it was true.

What was graded. Everything in this note is about the machines in this photograph and the repositories that describe them.

What was read

In everyday termsNot a tour and not a questionnaire. Every file in every repository, the history behind those files, and then the machines themselves — asked what they were actually doing rather than what the files said they should be.

Seven repositories, plus the plugin marketplace and the workstation’s own configuration. The git history of each. Every continuous-integration file. All sixty-five alert rules. Every published site.

Then the running estate, read-only: both Proxmox nodes, the firewall pair, the storage box, the smart-home hub, the metrics and log stores, and the forge. Where a claim in a file could be settled against a live API, it was settled against the live API — the audit’s standing instruction was that a document is a claim, not evidence.

How it ran

Each stage is a separate run of agents, with what it cost underneath. Two later passes are in the cost table and not drawn here.

Before any auditing, two inputs were built. The vendor’s own documentation per technology area was collected into 14 research checklists, so that “best practice” meant a fetched page rather than a memory. And 264 of this household’s own prompts — every time the operator had corrected something — were mined into 63 review lenses, then into 14 conduct rules and 57 standing rulings, and finally into a reviewer agent. The estate was graded partly against its own owner’s past objections.

The rubric itself scores 33 criteria from 0 to 4, so the ceiling is 132.

The audit proper produced 109 verified reports: 38 on components, 36 on documents, 17 through the lenses, 8 on git history, 4 on plan drift and 6 on the guard and lint rules. A further 11 domain consolidations sit on top of those. Underneath sits a register of 2,415 records, 2,154 of them canonical after duplicates were folded together.

What did not survive

In everyday termsThe interesting number is not how much the audit found. It is how much of what it found was wrong, and whether anyone went back to check.

Every one of the 109 reports was re-read by a different pass, and the result was tallied by one rule: inside each section, one heading or one table row is one item.

2,269 items went into verification. 275 came out as not real.

Then the 86 findings the register held at critical severity were re-checked individually against code and live state. Sixty-seven read confirmed, twelve confirmed but weaker than first written, and six belonged at a different severity altogether. One was false: the audit had recorded the vault restore as never having run, and it had succeeded on 2026-08-20 — a date already the backups note. The job number went into the record in place of the finding.

The audit also filed findings against its own inputs. A pattern search written with an unescaped alternation character, which had carried one recommendation up to high severity. A report graded against a repository head already two commits stale. Checks that were handed back as “run this” instead of being run. And a low-severity bucket appearing in at least 14 of the 36 documentation inputs, in defiance of the standing rule — mined from the operator’s own corrections, and adopted by the audit — that nothing is filed as minor.

The last check was aimed at the report’s own language. A search for the vocabulary of deferral — minor, nit, later, non-blocking, out of scope, consider, might — was run against the finished text. It returned 76 hits on 42 lines, and every one of them is accounted for in the report by location: a quoted line of source code that itself defers, a semantic-version term, a measured interval, or the record of the search itself. A softening check is only evidence if its result is read rather than reported.

And what could not be verified is a list rather than a silence. Console readings the read-only tooling cannot reach. Values the safety classifier refuses to hand over. Operator-only surfaces: who holds which key, which account has two-factor turned on, who performed a particular action on a particular day. And the mutations that would settle a claim outright but would change the estate in order to do it. Twenty-one questions in the critical set stay unanswerable from that vantage point, and each is recorded with the command that would settle it and whose job it is to run it.

The score, and why it was the least useful thing

The number on the left is real. It is also close to the only number that rubric could have produced here.

45 of 132. Median 1. Five of nine critical checklist rows failing. Not enterprise grade, and the report says so in its first line.

The rubric’s own passing line is that no criterion sits below 2, the median reaches 3, and no critical row fails. The first of those three is unreachable with one operator, and the report admits it in the same section that applies it. So the verdict was decided before the first file was read. Worse, it inflates the count: a bus factor of one, no second reviewer and no on-call rota are all scored as defects, and none of them is a defect. They are a description of a household.

One figure made that plain. The audit counted merge requests merged by the account that opened them and read the ratio as an absence of review. The count is accurate and useless: there is one account here, and it cannot distinguish the person from the agent working on their behalf. What it measures is how many people live in the flat.

A separate review followed, over the report and the live estate both, and named the bar that should have been used. Two questions. Can an outsider get in without a credential? If a piece dies, can it be rebuilt from git, the vault and an off-site copy? Measured that way the design is sound and the holes are countable — the review counts four, which is a different sentence from 45 of 132, and not a comfortable one either. The findings underneath the framing mostly held as well: token scopes, runner mounts, log contents and firewall rules were re-checked one by one and stood up in every case the reviewer went back to. What the audit got wrong was grading, ownership and framing. Not facts.

What it cost

Measured from the runtime’s own per-run accounting.

Pass	Tokens	Tool calls	Agent runs
Component, lens, document, history and rules	36,290,808	11,200	98
Verification of all 109 reports	27,460,793	8,106	218
Findings register and inventory	3,872,917	388	13
Re-check of the 86 critical findings	3,326,489	1,164	23
Independent triple double-check	1,369,950	471	5
Editorial pass	230,887	51	1
Measured total	72,551,844	21,380	358

Two further runs report no figure and are not in the total: the foundation pass that built the checklists, the lenses and the rubric, and one verification pass stopped part-way through to add a lighter mode. The total is a floor.

What is deliberately not here

Twelve findings are canonical criticals. Three have since been closed; nine have not. What they are, which machine they sit on and what closes them are not on this page, and will not be while they are open — the same goes for the ranked list of ways in. This paragraph exists so that the omission is on the record rather than hidden behind a cheerful summary.

What it changed, and what it did not

As of 2026-09-08, twenty-seven findings are closed. Sixty-one have a change written and waiting. 2,327 have nothing written at all.

The distance between those three numbers is the honest result. A merged fix is not an applied one, an applied fix is not a verified one, and none of the three is a closed finding. Clearing out the stale merge requests on 2026-09-08 returned 77 records to open in a single pass, because the change had been written and never landed.

Which is the same lesson the rest of this site keeps arriving at from the other direction. A thing is not done because it was decided, or written, or merged. It is done when it has been seen working.

Method, coverage, cost and the re-verification results read from the audit’s own generated figures and report on 2026-09-08; the closed and open counts from the findings register on the same date. The findings themselves are not published.

The config that deploys itself

Thu, 03 Sep 2026 00:00:00 +0200

One service in this apartment datacenter is not deployed by CI. The smart-home hub fetches the reviewed branch itself, merges it, runs Home Assistant’s own configuration check against the merged tree, and resets back to its last-good commit if the check fails — or if the check cannot run at all. In exchange, the running machine may commit its interface edits straight to that protected branch, unreviewed. This note is about that trade and the controls that make it survivable: a branch nobody can rewrite, a gate that fails closed, and a commit path that refuses a secret-shaped file before the commit exists. The last one exists because ignoring files was not enough. It is also honest about what the pipeline cannot prove: no automation is tested, no firmware is compiled, and much of the running instance lives in storage git never sees. As of 2026-09-08.

Everything else in this fleet changes the same way: a merge request is reviewed, a pipeline runs, and a person presses the button that applies the new state. The smart home runs the other way round.

A pull, not a push

In everyday termsEverywhere else in this building a change is delivered to the machine, the way a parcel is. Here the machine collects it, opens it on the doorstep, and hands it back if the contents are wrong.

The hub’s container carries a forked deploy integration that watches the reviewed branch, fetches it every five minutes, and merges it into its own configuration directory. There is no deploy job and no login seam: nothing in CI reaches the running instance.

Dashboards, scenes and scripts are edited in the interface, by hand, and a one-way push from CI would overwrite that work on every deploy. So the sync is two-way, and everything that follows exists to make that safe.

Merge, check, roll back

In everyday termsA delivery is checked at the door before it is accepted, and refused whole if anything about it is wrong. If the person who does the checking is off sick, nothing is accepted at all — and that is the deliberate part.

The integration records the commit it is standing on, merges the incoming branch, and only then runs Home Assistant’s own configuration check against the merged tree. If the check reports errors it resets hard to the recorded commit. Nothing reloads; the box goes on serving the tree that worked.

The other branch matters more. If the validator itself raises — a helper whose signature moved, an import that failed — it does not wave the tree through. It blocks, and the comment in the code is one line: a gate that cannot run must block.

A pull that passes goes the other way: an event fires and a git-tracked automation applies it — a reload, or a full restart when the core configuration or an integration’s code changed. A merge to the reviewed branch can restart the house, on a tree the check has already validated.

The machine writes back

In everyday termsThe house is allowed to write in the logbook that governs it. So the logbook cannot be erased afterwards, and the pen refuses the page when what is about to be written looks like a key.

Interface edits are auto-committed on the box and pushed straight to the protected branch. No merge request, no review — and this is the only repository in the fleet where that is allowed. CI still runs on those commits, but only after they are on the branch, which is why the fail-closed guards below, not the pipeline, are the gates on that path.

It is written down as an accepted deviation: losing every dashboard edit on the next pull was judged worse. The controls named in the same breath carry the weight; three matter here.

The branch stays protected. No force-push, from anyone. A secret that reaches the history can be rotated but never rewritten out of it — which is why the guard against secrets has to work before a commit exists.

The pre-deploy gate is fail-closed, as above: the machine allowed to write to the branch is also the one that refuses to keep a tree that would not load.

The commit path refuses secret-shaped paths. The box stages everything, then matches the staged paths against a pattern list, and on any match resets the index and refuses. The commit never happens.

That guard exists because ignoring files was not enough — an unattended stage-everything commit can sweep up a path the ignore rules miss, and a backup copy of an ignored file is not covered by the original’s pattern. The refusal is deliberately total: the automatic pull takes its own safety commit first, so a blocked commit blocks the pull too, and merges stop reaching the house until the offending path is gone.

What the pipeline proves, and what it cannot

In everyday termsThe checks read the recipe and confirm it is a recipe: quantities present, no step missing. Nobody has tasted the food.

Twelve jobs are defined across two stages, lint and validate. There is no deploy stage.

Configuration validation runs Home Assistant’s own check inside the official image, pinned by tag and digest to the version production runs, and it blocks the merge request. The check alone is a false green: Home Assistant logs a schema-invalid item, disables just that item and carries on, so only a syntax error or a core-block failure exits non-zero. The captured log is therefore searched for the failure strings as well. A separate job scans the full git history for secrets.

The pipeline proves the configuration parses and loads under a pinned image, and that the local patches on the vendored code are still in place. Not one automation, template or presence rule has a test.

Much of the running instance is not in git, and cannot be. Scenes and scripts created in the interface, dashboards, every integration’s config entry and every credential live in the hub’s own storage, deliberately untracked. For the files git tracks, the repository is the running configuration — except while a gate is holding the box on an older tree, which is the point of the gates.

Nothing here compiles firmware either: nine node configs are validated in CI, but the builds happen in a container on the host, and a compile that never runs is invisible to git and CI alike. Nothing proves the firmware on a given node was built from the config git holds for it. What those nodes carry, and which of them run firmware written here rather than merely configured here, is the subject of the radios note.

One more thing changes the tree without a person in it. The third-party integrations installed on the hub update themselves in a weekly window on Sunday morning, and the automatic commit then publishes the new versions to the branch — an unreviewed change to the running house, by design, on the same accepted deviation as the interface edits. The window is narrow on purpose. It skips firmware of every kind, because an over-the-air update would overwrite the lamp firmware written here, wipe the credentials off the sensor nodes and push a development build onto the radio coordinator. It skips an update that reboots the storage box. And it skips the handful of components carrying local patches, because installing over one of those clobbers the patch and the commit publishes the clobber.

One press showed how little the repository controls. On 2026-08-24 the vendor’s own update entity was pressed; it flashed the stock factory image, which carries no credentials for anything to reach the hardware with, and all four Apollo multi-sensor nodes left the network at once. That entity and the vendor’s firmware button are now hidden on all eight vendor nodes — hiding the entity alone leaves the identical flash one press away, because the button calls it. The nodes have since gone further than hiding: the firmware’s own web updater is disabled outright and the fallback access point each node raises when it cannot find the network now carries a password, so the same flash is no longer reachable over the air at all.

What is missing, and what is planned

The off-box backup has been decrypted and booted in isolation; a from-scratch rebuild onto an empty box has not. The bootstrap restores the untracked state — from the off-box backup, decrypted with the emergency key held for it — onto an empty configuration directory, and the sync takes over from there. The backup note has both halves: the restore that has actually been performed, and the fleet-wide proof still owed.
Planned: one pass over the physical controls — inventory every button, dial and pad, then decide the mapping per room alongside retiring the lighting behaviours still running inside a vendor bridge, so two systems never drive the same lamps.

Both items name the same absence: nothing here has been proved by watching it work. The gates prove that a tree loads — not that the house running on it could be rebuilt, and not that its physical controls agree with it.

Deploy integration, pipeline and node configs read in the smart-home configuration repository, and the pinned validation images in the shared CI templates, on 2026-09-08; dated incidents carry the date recorded with them, and planned items are marked as planned.

Four radios, and where local stops

Thu, 03 Sep 2026 00:00:00 +0200

Four radios carry the devices in this flat, and each is good at something the others are not. This note counts them: what each one actually carries, the devices whose firmware is built in the flat — nine built nodes, two air purifiers and, since September, the reader sitting on the electricity meter — and what measuring a radar taught about what it can and cannot see. It covers how the household’s electricity is measured too — off the utility meter’s own optical port now, with the per-phase clamps demoted to a cross-check — and the written rule against counting the same watt twice. And it is honest about the seam. Several bridges exist only to keep bought hardware useful without the vendor’s cloud, two of them running firmware written here after the manufacturer’s servers went dark. But a lighting bridge still runs its own behaviours on a wall clock beside the hub, and the Amazon Echo units in the flat are cloud devices carrying the household’s actual daily voice traffic. The local speech pipeline really is local. The house as a whole is not. As of 2026-09-08.

A count is not an architecture. Zigbee holds the switches, Thread rides the Zigbee coordinator’s own border router, Matter turns out thinner than the word suggests, and Bluetooth LE only ever seconds a decision another sensor has already made. This note goes through them one by one, then through the devices whose firmware is built in the flat, and ends where keeping a bought product local did not work.

Four radios

In everyday termsFour separate radios in one flat sounds like a mess, but a household already runs several without noticing: the doorbell, the car key, the headphones. Each is built for its own job, and none of them would do the others'.

Zigbee carries the switches, buttons, knobs and metering plugs. It runs natively inside the hub rather than through a separate bridge daemon — a program of its own, running beside the hub. A version-controlled quirks directory covers the devices the upstream library models badly: a metering clamp needed one.

Thread runs on the Zigbee/Thread coordinator’s own border router. No border-router container runs beside it on the host, and the host’s configuration is written to keep it that way.

Matter is thinner here than the word suggests. The lighting bridge is exposed as a Matter node, and the Matter twins of three motion sensors carry battery entities only. Matter is not carrying the house.

Bluetooth LE is one leg of presence. The phone’s own beacon is located room by room across eight receivers — all of them built sensor nodes doing a second job — computed on the LAN, with nothing sent outward.

Firmware built here

In everyday termsFirmware built here is a recipe that names the exact version of every ingredient — because a supplier quietly changing one is how last month’s dish stops working.

Nine nodes run firmware built in the flat, and eight of them are bought hardware running the vendor’s own published firmware package with local overrides: four multi-sensor nodes reading mmWave presence per range gate, carbon dioxide, illuminance and environment, and four air-quality nodes reading carbon dioxide, particulates and environment. A range gate is one distance band out from the radar, and each band is reported separately.

The ninth is hand-written and is not a sensor. It is an ESP32 on the air conditioner’s own RS485 bus, turning two indoor units into ordinary climate entities and publishing the outdoor unit’s power, energy, current, voltage and error code. Its external component is pinned to a commit hash of upstream’s main branch rather than a release tag, because upstream keeps its compatibility fixes there and warns that its own tagged code no longer builds.

Three more devices carry firmware written in the flat rather than merely configured here, and all three exist because a vendor stopped. Two are air purifiers whose manufacturer’s backend went dark; they run a firmware built for them in this household and published under the household’s own name. The third is the reader on the electricity meter — a community project, built here with a set of local fixes, talking to the house broker and to nothing else.

The two dodecahedron lamps in the living room belong on the list as well. Their vendor ships a closed fork whose effects come with no source, so the effects were read back out of the factory image and reimplemented on the stock open firmware; the tour has that story.

None of that proves the binary on a node was built from the file beside it. That gap belongs to the deploy note.

What a sensor can actually see

In everyday termsFootsteps heard through a wall prove somebody is in the building, not which room they are in. So the instrument that can only see its own room is the one allowed to say a person arrived, and the one that hears through walls may only say they have not left.

A radar sees through a wall, and did: people in the hallway were reported as occupants of the next room, while that room’s infrared sensor stayed off and the light lit an empty room. No threshold separates the two cases, because a body close to the sensor saturates several range gates at once. So only the infrared sensor may start presence, and the radar may only sustain it — which is what a motionless occupant needs.

Another radar was dropped from the presence inputs altogether. Its moving channel flapped too often to be trusted as an input.

Room-level Bluetooth is live and deliberately not trusted on its own: it disagrees with the presence sensors often enough that a Bluetooth area may never drive a light without a presence sensor agreeing, and the walk-through that would qualify it is unlogged.

What the house then does with presence — five states derived from a door, a phone and these sensors — is the subject of the house-states note.

What the house measures

In everyday termsIf two meters watch the same washing machine, adding both to the bill does not make the reading better — it makes the house look like it uses twice the electricity it does.

Grid draw is read from the meter the utility bills against. A small radio reader sits on the meter’s optical port. The three current clamps that carried that job until September are a cross-check now, and they earn it by disagreeing: they read a couple of percent over the meter. That is how you find out that a clamp is a clamp and a meter is a meter. The loads worth watching are metered at the plug or the circuit.

The rules written around that dashboard are mostly about not counting a watt twice: never a derived counter beside the native counter it duplicates, never a channel that is the sum of two other channels, and exactly one row for a load two meters can see.

The uninterruptible power supply is the worked example. It reports no wattage, so a derived sensor multiplies its reported load percentage by the nameplate rating, in coarse steps. It is a headroom gauge, not an instrument, and barred from the dashboard because a plug already measures that socket for real. It fails to unavailable rather than zero, because a zero would read as an idle rack. And even deduplicated, the metered rows do not account for all of the grid draw — some of the flat’s load simply sits on no meter at all.

Where local stops

Some bridges exist to keep bought hardware useful without the company behind it. The living-room presence sensor is paired over a local protocol rather than a vendor account. The air conditioner speaks its own bus instead of the manufacturer’s app. Two air purifiers run replacement firmware written here and publish plain integers to the house broker, because the vendor’s backend went dark and there was no other way to keep them — and the configuration admits the cost: one switch runs optimistically, since the firmware never reports that state back. One of the two now stands outside on the balcony and blows through the wall into the bedroom, which makes it the only mechanical air exchange the flat has.

Then the seam. The lighting bridge still runs its own behaviours on a wall clock in two rooms while the hub drives the same lamps — two systems on one set of lamps, which is the one arrangement this house rules out. Retiring them is an operator action inside the vendor’s own app; no configuration change can perform it.

The voice seam is wider than that. The Amazon Echo units standing in the flat are cloud devices: what is spoken to them goes to a company’s servers and the answer comes back. They carry the household’s actual daily voice traffic — the spoken light commands. The local speech pipeline genuinely is local: the speech-to-text and text-to-speech services and the satellite all run in the flat. It is not what the household talks to. The house as a whole is not cloud-free.

What is planned

One air-quality channel left unconsumed. A paired comparison found its humidity reading well off a reference. Nothing reads it until it is checked.
Retiring the bridge-native lighting behaviours, tied to one pass over every button, dial and pad so the two do not fight.

Where local stops is not a principle here but a position, held device by device. It moves only when someone moves it: which is why the second item waits on a person in a vendor’s app rather than on a merge request.

Radio and node configurations, package files, the energy plan and the component inventory read in the smart-home configuration repository on 2026-09-08; the sensor measurements are recorded there beside the rules they set. Planned items are marked as planned.

What runs, and why

Sat, 22 Aug 2026 00:00:00 +0200

A tour of the services in the apartment datacenter, sorted by what they are for rather than by vendor — with the reason each one is there and the things that were deliberately left out. This is what runs as of 2026-09-08, not a recommendation; every guest named here was seen running on the Proxmox API on that date, and every container comes from a compose file in git.

Two computers under a desk in Cologne run the things a household usually rents from Google, Apple or Amazon. Nothing here came as a bundle: every service on the list is here because something in the flat needed it, and everything that can be pinned to a version, rebuilt and rolled back is.

The shape of it

In everyday termsTwo computers do the work, a storage box keeps the copies and breaks ties, and every service runs in its own small container — like tenants in one building, each with their own door.

Two physical Proxmox nodes carry everything. A Synology NAS — the storage box above — sits beside them for shared storage, backups, the cluster’s tie-breaker vote and a handful of services of its own. Everything that runs on the nodes is a guest: one virtual machine or one container. Guests come in two kinds: a handful of Docker VMs for stacks that want to be composed together, and small LXC containers for single-purpose services that need nothing but an operating system and one daemon.

What runs where, by role — the guest list as of 2026-09-08 first, the NAS’s own services after it:

Role	What runs
Edge	two OPNsense VMs, one per node — the firewall pair that owns the WAN legs
Smart home	one Docker VM (`tychome`) with Home Assistant and the voice stack
Cameras	a detection guest on the second node, beside the recorder on the NAS
Sites and ingress	one Docker VM (`tycfront`) for the Traefik front door and the static sites
Observability + CI	one Docker VM (`tycdocker`) carrying the Grafana stack, plus `tycops`, the VM the stack is moving onto
Secrets	an OpenBao LXC (`tycbao`)
Media	a Plex LXC
Network services	the UniFi controller LXC, an mDNS reflector LXC, a UPS-monitor LXC
Backup	a backup-server VM, and since September the nightly target for every guest but two
Development	one VM for development work
CI runners	two general-purpose runners on the Docker host, four runner containers on the nodes, plus one on the NAS as the recovery path
NAS services	the household photo library and file sync, the identity provider behind the Proxmox login, a network-boot container and the two backup applications — on the NAS itself, not as guests

One guest still runs twice over: the host that carries both observability and ingress is up beside the two new guests taking over its halves. That is the migration, not an accident: each replacement runs beside its predecessor until a separate, reviewed retirement, and the architecture note explains why the cutover is staged that way.

The room all of this runs in — real light, real hardware, no render.

August 2026: the switches mounted vertically in the gap between the desk and the hi-fi tower; the final cabling is still to come.

Smart home

In everyday termsLights, heating and sensors answer to one hub in the flat, not to a company’s servers. The hub can listen and answer without anything leaving the building — but the Echo speakers still standing in the rooms cannot: everything said to those goes to Amazon and comes back.

Home Assistant is the hub, and it runs as a plain container rather than Home Assistant OS. The reason is pinning: HAOS add-ons cannot be version-pinned, rebuilt and rolled back like any other service, and everything else here can. The price is that the “add-ons” have to be run as ordinary containers alongside it — which is exactly what the voice stack is.

Mosquitto — the broker for MQTT, the small message protocol devices use to post their readings so that anything interested can subscribe to them. Only a handful of devices and bridges here need it; the Zigbee radio and the DIY sensors talk to the hub directly instead.
Matter server — the Matter fabric for the devices that speak it.
Zigbee and Thread — the Zigbee radio is driven by the hub itself, and Thread has no stack of its own: it rides the border router built into the same coordinator, and the deployment refuses to stand a second one up beside it. The four radios in the house note has the detail.
ESPHome — the firmware builder for the DIY sensors; its dashboard is the git checkout, so device configs are versioned like everything else.
tycwled — the household’s own firmware for the two dodecahedron lamps in the photo above. The vendor ships a closed fork of WLED whose effects come with no source, so they were read back out of the factory image and reimplemented on stock WLED, published under the MIT licence. Every change has to prove itself against a second lamp still running the vendor build.
Whisper and Piper — speech-to-text and text-to-speech, locally. Nothing the local assistant hears leaves the machine. It is not what the household actually talks to: the Amazon Echo speakers still in the flat carry the everyday spoken commands, and they are cloud devices by construction.
Music Assistant — the multi-room audio layer for the speakers.
govee2mqtt — the bridge that keeps the Govee lights on the local broker instead of a cloud app.

What the house does with all of that hangs off one variable. A house state is derived from the front door, the phone and the presence sensors — never from a clock — and the welcome light, the lock, the air conditioning, the cameras, the vacuum and the bathroom fan all read it. That has its own note: the house decides, and no clock does.

Home Assistant is also the one service that deploys itself, and the one place in the house where a change reaches the branch without a reviewer: dashboards edited in the hub’s own interface are committed straight back to main, because losing every interface edit on the next sync was judged worse than the missing review. The fail-closed check the container runs before it accepts a commit is what stands in for the reviewer. The whole loop is in the note on configuration that deploys itself.

Cameras and detection

In everyday termsThe recorder still records. The new part only looks at the picture and says “that is a person” — what happens next is decided in the flat, by the hub.

The cameras record to the surveillance software on the NAS, as they always have. Beside it runs a second, local detection plane: an object detector in its own guest on the second node, using that machine’s integrated graphics for inference and keeping no recordings of its own. The split is deliberate. The detector detects, Home Assistant decides what a detection means, and the surveillance software keeps recording and raising alarms.

Every camera in the flat is enrolled now, the fisheye at the front door included, and the detector recognises faces on the same stream it detects on — so an alert raised while the flat is empty can say who it was rather than only that somebody was there. What the hub does with a detection, and why the recorder’s own Home Mode is deliberately left alone, is in the house decides.

Energy

In everyday termsThe flat measures its own electricity, where it comes in and at the sockets, so the bill is never the first news of something left switched on.

The house total is read from the utility meter itself: a small radio reader sits on the meter’s optical port, runs firmware built in the flat, and publishes to the house broker with no vendor account anywhere in the path. The three current clamps that used to carry that role are now only a cross-check. The loads worth watching still report per socket or per circuit, all of it inside the hub. What keeps the picture honest is that no load is counted twice — the metering rules, and what is still owed, are in the four radios in the house note. The figures themselves stay off this page.

Media

Plex serves the household library from an LXC; the bulk media stays on the NAS and is mounted in. Synology Photos runs on the NAS itself and holds the household’s photo library — the pictures stay in the flat rather than on a photo service’s servers.

Sites and ingress

In everyday termsOne front door for everything public, including this page. Behind it, each website is a sealed package that only changes when the new package is approved in writing.

Traefik is the public web front door. It routes this site, a second Hugo site and a couple of pro-bono ones. Every one of them is a container image pinned by tag and digest — the second site moved off a copied folder onto the same image path, so they all ship the same way now. What is served changes only when the pin changes in git and the gated apply runs. Public ingress is IPv4 only, and that is now a decision rather than a stage: publishing these names over IPv6 as well was ruled out in September.

Homepage is the internal dashboard — the page that answers “where was that again” for the household.

The front door is in the middle of a move: tycfront is the new ingress VM in the published zone. The sites are served from both during the coexistence period, and the copies retire once the move is reviewed.

Identity and secrets

In everyday termsNo master key on a hook by the door: every job gets a single-use key that expires minutes later and opens only its own room.

OpenBao holds every credential the fleet uses. No merge-request pipeline ever holds a long-lived token. Each job proves who it is with an OIDC identity — a signed statement from GitLab that this job is this job — against a per-repository, read-only role, and receives a short-lived credential scoped to exactly that job. Plan material and apply material are separate roles. The few long-lived tokens that remain — the ones that talk to the forge and the package hosts rather than to a machine — live in the vault and are handed to protected-branch jobs as masked variables, never typed by hand. The vault takes a nightly snapshot to the NAS, and its restore job has actually been run; only the smart-home hub can say the same, and most things on this page cannot.

Synology SSO Server on the NAS is the identity provider behind the Proxmox login: the cluster carries a realm that maps the NAS’s administrators group onto a scoped admin role, and that mapping is declared in the infrastructure code like the rest of the cluster. The application itself is set up by hand in the NAS’s own interface — the mapping goes through review, the thing that authenticates does not.

Observability

In everyday termsEvery machine writes down what it is doing and how it is feeling, in one place — the way a ward keeps a chart at the foot of each bed, so nobody has to walk round asking.

Grafana, Loki and Prometheus, with Alloy shipping logs and metrics from the hosts the configuration management reaches, plus the usual exporters (node, Proxmox). The firewalls, the switches and the modem have no agent to install and send their logs the old way instead. The Grafana alert rules are provisioned from git like everything else. Coverage is incomplete: fleet coverage is a tracked work package rather than a finished state.

Network services

In everyday termsThe network keeps its own doorman and its own phone book: one hands each arriving device its room number, the other looks names up and refuses the ones on the house list. Both live in the flat, so no device can quietly ask somebody else’s phone book instead.

Most of the network is not a service at all — it is the RouterOS fabric and the OPNsense pair, both driven from git and written up in the architecture note. The parts that run as services of their own:

Kea DHCP and Unbound DNS on the edge pair — Kea hands out the addresses, Unbound answers the name lookups — one set per zone, with the host registry generated from the same network model that generates the switch configuration. The resolver also filters advertising and malware domains for the zones that should have them. Filtering only works if it cannot be walked around, so the firewall sends those clients back to the resolver, refuses the encrypted lookup protocols outright, and the resolver blocks the public services that answer them. A filter a device can step around is theatre. There is no Pi-hole; the resolver lives where the policy lives.
A WireGuard tunnel — the household’s own remote access into the flat terminates on the edge pair.
UniFi controller — one LXC for the one surviving UniFi device, the access point.
mDNS reflector — so casting and discovery cross the zone boundary where it is explicitly allowed to.
UPS monitor — a small web interface over NUT, the daemon that reads the uninterruptible power supply; the nodes read it too, so that they can shut down cleanly when the power goes.
netboot.xyz — network boot: a machine with no installer to hand pulls one over the wire. It is a container on the NAS rather than a guest on the nodes, committed as a compose file and started by hand in the NAS’s container manager, the same shape as the runner kept there for recovery.
Cloudflare is the DNS authority, managed as code; the registrars are only registrars.

CI

In everyday termsThe robots that carry out approved changes live in the flat; one lives in the storage box so that something still works when both servers are down.

CI is the pipeline that checks a change and then carries it out; GitLab runners are the machines it runs on, and they sit where the work is. General-purpose ones take merge-request and build work. Dedicated ones run as their own containers on the nodes: one for the infrastructure applies, one for the smart-home repository, and one per edge node, because each firewall’s own identity — its interfaces, its addresses, its dial-up session — has to be written from the node it lives on. One more is a container on the NAS, kept for a single purpose: being the path that still works when both nodes are down. Apply jobs are manual, environment-scoped and serialised per OpenTofu state; merge-request pipelines are read-only.

Storage and backup

In everyday termsOne cupboard in the flat holds what every machine needs to reach, and each machine keeps a small shelf of its own for the things needed to put that cupboard back. Nothing that rescues the cupboard is allowed to live inside it.

The NAS shares storage over the network two ways: NFS, which the guests mount for shared storage, and SMB, which the Home Assistant backups are written over. The nodes keep the repair-path guests on their own disks instead, on ZFS — a filesystem that checksums what it stores — so that the NAS is not on the path to recovering the NAS.

The household’s own files live on the NAS too. Synology Drive keeps them in step across the machines that use them. Active Backup for Business captures the household workstation, and Hyper Backup replicates from the NAS to a target in another building — the off-site copy. All three are set up by hand in the NAS’s own interface, not through the reviewed pipeline the rest of this page runs on. What those copies cover, and what is still owed, is the backups note’s question rather than this page’s: Backups: three copies, one missing proof.

What was deliberately left out

Kubernetes. No workload here needs it; it comes back on the day one does.
Home Assistant OS. See above — pinning beats convenience.
A cloud backup target. Not yet, and not before a restore has been proven against the copies that already exist.
Sponsorship. Nothing on this page is paid for or for sale.

Those four are decisions rather than gaps, and most of them name the condition that would reverse them. Nothing on this page runs because it was expected to be here: it runs because something in the flat needed it, and the day that stops being true is the day it comes off the list.

Guest inventory, storage and the nightly backup jobs read from the Proxmox API on 2026-09-08, and the household automations from the running hub the same day; the containers against the compose files in git, and the runner list against the declarations that register them. The identity provider’s realm on the cluster is declared in the infrastructure code and can be read there; the NAS’s other applications are as the operator stated them on 2026-09-09, and were read from neither the NAS nor the code. Planned items are marked as planned.

Backups: three copies, one missing proof

Sat, 22 Aug 2026 00:00:00 +0200

The apartment datacenter measured against 3-2-1: what is backed up, where it lands, what watches it, and the gap — a full restore drill that has not happened yet. The backup that silently stopped for thirteen days is the reason the monitoring exists. Since September the archives go to a dedicated backup server that deduplicates them, re-reads them to prove they are still readable, and takes them encrypted, so the storage box underneath holds copies it cannot open — which moves the hardest open question from ciphertext to who holds the key. As of 2026-09-08.

The rule everyone quotes is 3-2-1: three copies of the data, on two kinds of media, one of them somewhere else. Each number answers a different way of losing the data — the copy that corrupts, the shelf that fails as a batch, the building that burns. What the rule counts is copies, not proof: whether any of the three has ever been opened again is the question this note ends on.

What is backed up

In everyday termsThe rule asks for three copies, kept on two kinds of shelf, with one of them in another building. There is now something in that other building — and still nobody has fetched it back to check that it opens.

Almost every guest, nightly. Since September the archives no longer go straight onto a network share. They go to a dedicated backup server running beside the rest of the fleet, which deduplicates them, keeps the last five per guest, and re-reads and checksums what it holds on a schedule of its own — so “the copy exists” and “the copy is readable” stopped being the same claim. The bytes still land on the storage box underneath — the NAS, which is where the space is. What changed is that they are encrypted on the node before they leave it, so the NAS holds archives it cannot read.

The backup server is the one guest that cannot use it: an archive of the backup server, stored inside the backup server, is not a backup. It writes its own nightly copy to the share directly, and that one archive is the declared exception rather than an oversight.

One machine is deliberately left out of the nightly run altogether. It stays out until its secret handling matches the rest of the fleet, because the rule here is that no backup store holds a secret in the clear.

The nightly run is five jobs, not one. They start one after another in the small hours, so two dumps never write to the same share at once. Each slot is as wide as the slowest run measured over a recent stretch of nights, plus margin — not as wide as a good night. Every job runs in snapshot mode, on storage that supports snapshots — which is the fix for something learned the hard way: a guest the cluster’s HA layer had taken over silently stopped backing up the day it joined, while its neighbour in the same job kept working. Two of the five are pinned to a node and three are not, and that is also a lesson: an unpinned job fires on every node at once, and the concurrency is exactly what the staggered slots exist to avoid.

The backup jobs live in git. A backup schedule that exists only in the Proxmox GUI is a schedule that a rebuilt node comes up without — and nothing fails, the backups just stop. So the job list is declared in the hub repository and converged on every deploy, adopting the GUI-generated job IDs verbatim so that no guest is ever backed up twice. If the cluster carries a job that git does not know about, the deploy fails loudly rather than deleting it: an undeclared job might be the only backup of something.

Home Assistant backs itself up to a dedicated share on the NAS, and the container mounts that share so a restore is a native operation rather than an upload. Before the hub was allowed to move to another machine, one of those off-box backups was decrypted and booted in isolation, so this is a restore that has actually been performed. It is also the half of a rebuild that git cannot supply: the hub’s own deploy path carries the reviewed configuration, and this backup carries everything that path never sees.

The secrets vault takes a nightly snapshot to the NAS, riding the always-armed schedule that also runs the drift check. Its restore job has actually run too, most recently on 2026-08-20.

Two more streams run on the NAS itself. Hyper Backup replicates from the NAS to a target in another building; Active Backup for Business backs up the household workstation to the NAS. Neither is declared in the repository or converged by a deploy — both are set up in the NAS’s own interface, by hand, which puts them in the same class as a backup schedule that lives only in a GUI and carries the same exposure. What the off-site job reaches is not established here: the third copy counts as a third copy, not yet as cover for any particular thing.

The firewalls and switches are code, so their backup is the repository plus the generated network model.

What watches it

In everyday termsNothing had gone wrong, so nothing complained — but nothing had happened either. What gets checked now is the date on the newest copy, the way one reads the date on the milk rather than trusting the fridge to be cold.

The failure that shaped this section: between 2026-07-13 and 2026-07-26 the nightly backup did not run, and nothing said so. Proxmox mails on failure, but a job that never starts does not fail. A healthy fleet and a fleet with no backups looked identical on every dashboard.

So every node now publishes the age of the newest archive per guest as a metric, read from the Proxmox API against the same shared storage. A guest with no archive at all is published as infinitely old — “never backed up” alerts exactly like “stale”, which means a new guest alerts until the next night’s run. That is deliberate; an un-backed-up guest is the condition. The collector refuses to write a partial file: if the API errors, the previous file goes stale and the freshness alert fires instead of a guest quietly disappearing from the list.

The smart-home hub applies the same rule to itself: its own nightly copy is judged by the entities it keeps about the outcome, never by the call having returned.

Counting, honestly

3-2-1 asks for	What exists
Three copies	Three, as far as the off-site job reaches: the guest’s own disk, the nightly archive on the NAS, and the copy Hyper Backup writes to another building
Two kinds of media	Yes: node SSDs on one side, the NAS array on the other
One copy off-site	Yes: Hyper Backup replicates from the NAS to a target in another building — set up on the NAS by hand rather than declared in the repository, and no restore from it is on record
A proven restore	Partly — the vault, the smart-home hub, and one guest pulled back out of the encrypted store; the whole-fleet drill is still owed

That last row is the one that governs the plan. A backup that has never been restored is a hopeful copy, so the whole-fleet restore proof stays an open gate. One guest has now been pulled back out of the encrypted store, which proves the encryption and the tooling; it does not prove the fleet.

The off-site row changed recently and the restore row did not. A copy in another building raises the count the day it is written; it raises the confidence only on the day something is pulled back out of it.

The table asks about copies, media and distance and says nothing at all about encryption — which is the row this setup would now score best on, and the row where the remaining work is not ciphertext but custody. The key that opens those archives is held in the fleet’s own secrets vault. A copy on paper, off the fleet, is owed: a key kept only inside the thing it unlocks is not yet a key you have.

What is planned

In everyday termsThe archives are locked, and the only key hangs on a hook inside the same building. A copy of it somewhere else is the part that is still owed.

Custody for the archive key, off the fleet and on paper. Losing the NAS and the cluster together, with the key only in the vault, would leave every archive unreadable — the failure the encryption was supposed to make survivable, arriving from the other direction.
The remaining cleartext. Getting every backup store to hold ciphertext only is still the largest open backup workstream; the guest archives are done, other stores are not.
The restore drill itself. Snapshot, export, wipe, restore, then count and value checks, driven from a path that does not depend on the NAS — because the drill has to work on the day the NAS is the casualty.
An off-site copy of the vault export with a defined refresh point. The export job runs; stashing it off-site is a human act, and a human act without a recorded date counts as unproven — and whether the NAS’s own off-site job already carries it is one of the things not established.

None of it waits on hardware: what the list is short of is acts performed and dates recorded.

Job declarations, monitoring and plan checked against the hub repository, the Proxmox API and the backup-age metric on 2026-09-08; the archive store, its retention and its verification state read from the cluster the same day. The two backup jobs that run on the NAS itself — the off-site replication and the workstation backup — are as stated by the operator on 2026-09-09; they are configured in the NAS’s own interface and were not read from the repository, and what the off-site job reaches is not established. Planned items are marked as planned.

The apartment datacenter

Fri, 21 Aug 2026 00:00:00 +0200

How one person runs a private datacenter: git records intent, CI gates every infrastructure change, AI agents do the typing — and a human keeps merge and apply authority. What is live, what is target, and why the design ended up this way. Checked against the running system on 2026-09-08; what is planned is called planned.

TycStation keeps the smart home, media, websites, secrets and much of the network of one Cologne apartment under local control. The unusual part is not the hardware; it is the operating model. Live verification, not a merged change, decides whether something actually worked, and the household keeps functioning when the internet does not. This note answers three questions: what runs where, who is allowed to change it, and how anyone knows a change worked.

What it runs

In everyday termsThe services a household usually rents run at home instead.

Smart home — Home Assistant and the household automations run as containers on a dedicated VM; the devices reach it over Zigbee, Thread, Matter and MQTT. The hub’s own voice assistant runs locally; the Amazon Echo speakers still in use are cloud devices by construction.
Media — Plex serves the household library; bulk media stays on the NAS.
Sites and ingress — Traefik serves websites (including this one) and selected public services. Public ingress is intentional and IPv4 only — a decision taken in September, not a stage still waiting on address-plan work.
Identity and secrets — OpenBao issues short-lived, per-repository CI credentials through GitLab OIDC: the job proves which repository it runs for and is handed a credential for that alone, instead of carrying a stored password.
Storage and recovery — the Synology NAS supplies shared storage, and a dedicated backup server now takes the nightly archives and encrypts them before they leave the node; a fleet-wide restore-proven drill is still a gate on the road ahead.
Observability — Grafana, Loki and Prometheus are live. Fleet coverage is expanding and not yet complete.

The physical footprint is one vertical mini rack’s worth of ordinary gear — the hardware section below lists it.

The hardware

In everyday termsTwo shoebox computers, three switches, one storage box, one Wi-Fi puck — and a noise budget set by the loudspeaker beside them.

The constraint that shaped this list is the noise floor. This build grew up in the gap between a desk and a hi-fi tower, and a machine that whines is disqualified before its spec sheet is read. That rules out rackmount chassis and their 40 mm fans whatever they cost per core, and it pushes the whole build toward small, low-wattage machines — heat is what forces fans.

Device	Role	The part that matters
2× MS-02 mini-PC	Proxmox nodes	24 cores and 48 GB each; an active/standby two-NIC bond into the fabric, a third NIC cabled straight to the modem for the WAN, and a fourth on a cable running direct to the other node
MikroTik CRS317	Core switch	The 10G backbone and the root of the loop-avoidance tree (RSTP); every trunk terminates here
MikroTik CRS320	PoE switch	Power over the data cable at its highest tier (PoE++), a budget the previous consumer gear could not supply; it feeds the wired AV bank, the KVM, the prepared camera seats, and the desk and the household’s other wired clients
MikroTik CRS310	Access switch	2.5G ports for the Wi-Fi access point — fed through an inline PoE injector, because the PoE switch is 1G copper — and the NAS’s quorum leg, with five seats held free for growth
Synology DS1525+	NAS	Shared storage, backup target, and the cluster’s tie-breaker vote
UniFi U7 Pro Max	Wi-Fi	The sole UniFi hardware survivor; the radios were worth keeping when the switches were not
Vigor 166	G.fast bridge modem	Separately powered on purpose: a firewall failover must never wait on a modem resync
UPS	Power	Clean shutdown time for the wall, not runtime heroics

The room the rack shares: the gear sits in the gap between the desk and the hi-fi tower.

October 2025, before the rebuild: the original mini rack in that same gap.

July 2026: the three MikroTik switches arrive and get their first cabling on the desk — the core switch on top, the PoE and access switches beneath.

The switches arrived carrying the same problem in miniature: 40 mm fans, the exact part the noise budget had already ruled out at chassis scale. They were opened on the bench the day they came out of their boxes and refitted with Noctua NF-A4x20 PWM fans — a 40 × 20 mm four-pin part — before any of them was cabled in.

It is not a clean drop-in everywhere, which is the part worth knowing. The stock fans are not all the same depth: the deeper ones are 40 × 28 mm, and against those the Noctua is eight millimetres shallower. It lands on the same frame and the same screw pattern, but it does not fill the housing, so that one is a partial fit rather than a straight swap.

July 2026: both switches open on the day they arrived. The fans sit at the edge of the board, next to the heatsink they are there to cool.

23 August 2026, the night this page was revised: the fabric is in, mounted vertically in the same gap as before — and not yet fully installed. Cabling and the final mount are still to come.

Two things this list does not have. First, no noise figure — that has never been measured; power draw is metered continuously, but per-device idle figures are not broken out here. Second, there are no purchase links.

Why it ended up this way

In everyday termsEvery layer is the answer to one thing that actually broke; nothing was added for show.

Each layer answers a failure, limitation or recovery gap observed in the earlier system.

One failed gateway took the apartment offline. → One fixed OPNsense VM per physical node. CARP and pfsync protect the service everything else depends on: the two firewalls answer to one shared address, and each carries a copy of the other’s open connections. The single DSL line still redials.
The first segmentation model became fragile and leaked at layer 2. → Eight routed zones with policy at the edge: fewer risk classes, one VLAN each, explicit flows, and generated cross-plane invariants.
Consumer-switch control fought the “git owns intent” rule and hit a PoE ceiling. → A RouterOS fabric; keep only the UniFi access point. MikroTik is fully code-driven and supplies the required PoE++; Wi-Fi stays with the stronger radio platform.
Long-lived automation credentials create silent, fleet-wide risk. → OpenBao plus per-repository GitLab OIDC. Jobs receive short-lived, narrowly scoped tokens; plan and apply material stay separate.
Home Assistant OS add-ons could not be pinned and rebuilt like ordinary services. → Home Assistant and its companions moved to containers, versioned as code; the instance validates and pulls main itself.
No restore had ever been proven. → Backup verification and a full recovery proof are gates, not afterthoughts.

Complexity was also rejected. No Kubernetes without a workload trigger, no firewall config authority outside git, no physical node reinstall, and no pretend seamless WAN failover.

View 01 — system context.

The operating principles

In everyday termsKeep it private, keep it running when the internet is not, keep it recoverable — and do it the way a professional team would, just smaller.

Privacy — local processing by default. Resilience — household services survive an ISP outage. Ownership — data and configuration stay recoverable. Craft — professional platform practice at apartment scale.

None of these patterns is unusual in a cloud platform: jobs that prove who they are instead of carrying a password; a floor plan of public, private and restricted zones; a firewall pair that shares its state so either half can take over; rules that live in git rather than in a console; a recovery that has actually been performed; and telemetry from everything. Four of the six are in place; full-stack observability is live but not yet fleet-wide, and a restore-proven recovery is still owed. Putting the set behind one apartment desk is the unusual bit.

The dates say when something was last checked.

What depends on what

In everyday termsFoundation, plumbing, tools, then the rooms people use — the same order as a house.

View 02 — the stack. Layers show dependency, not rollout status; planned elements say so.

Layers

The tools sort into four lanes — which tool owns which concern. This map answers “which technologies?”; it deliberately does not imply placement — the deployment view does that.

Lane	What it is for	Tools
Experience	what people use	Home Assistant, Plex, the websites, local voice, Grafana
Platform services	shared capabilities	OpenBao, Traefik, Prometheus, Loki, Alloy, MQTT; a dedicated backup server takes the nightly archives, and the fleet-wide restore proof is still owed
Automation	how change happens	GitLab CI, on-LAN runners, OpenTofu, Ansible, OIDC, Renovate, the network model
Foundation	what everything stands on	Proxmox VE, Docker, OPNsense, RouterOS, Synology, Cloudflare

Today versus the plan

In everyday terms“Live” means seen running today; “planned” means written down and not yet built — a permit granted for an extension not yet poured.

Repository declarations prove intent, not rollout. As of the 2026-09-08 evidence snapshot:

What runs today

Two physical Proxmox nodes carry the fleet, and the third cluster vote is a tie-breaker service on the NAS; the small helper member that used to hold it is retired.
The two OPNsense edge VMs run one per physical node and own the WAN legs. They are a CARP + pfsync pair: both hold every address the network sends traffic to, firewall state syncs between them, and DHCP runs hot-standby — the second copy is loaded and current, not started when the first fails. On 2026-08-23 a real swap and swap-back moved the master role and its dial-up session with it.
The eight-zone switch fabric is applied — the renumbered VLANs, the zone gateways and the inter-zone policy are what the network runs today.
Guests are moving onto their zone addresses and storage tiers one at a time; each replacement coexists with its predecessor until a separate, reviewed retirement.
High availability is armed, with twelve guests under it. Both drills passed on 2026-09-01. One node was rolled through a planned reboot — its guests drained to the peer beforehand — and came back with its own guests; the other was crashed outright, and every enrolled guest was running on the surviving node about two minutes later. Replication mirrors every repair-path guest between the nodes, the nightly drift checks run, and staged targets are expected differences, not defects.
The nightly archives go to a dedicated backup server, which deduplicates them, verifies what it holds by re-reading it, and takes them encrypted on the node before they are written. The storage box still holds the bytes and can no longer read them. One guest has been restored out of that encrypted store; the fleet-wide drill has not been run.

What is planned

The edge pair’s hard failure drills: WAN cable of the master pulled, the node holding master powered off, a trunk member down — each with a recorded time to failover. The graceful swap is proven; the hard reads are owed.
The full restore proof, fleet-wide, and custody for the key that opens the archives: it is held in the fleet’s own vault, and a copy off the fleet is owed. A key kept only inside the thing it unlocks is not yet a key you have.
The cluster’s third vote moves off the storage box and onto the access switch. The reason is concentration, not availability: quorum, every storage tier and every backup lane are the one appliance today, and the vote is the cheapest of those to move. Nothing answers at the new seat yet, and the old one serves until it does.
The existing nodes converge in place: no reinstall, no rename.

View 03 — deployment; the edge pair, the fabric and the tie-breaker vote are live. Cluster names remain TYCPVE and Tyc2pve; the lowercase names are DNS role aliases.

WAN never crosses a switch: the modem’s two LAN legs cable directly to the two nodes, one edge VM inside each. The node bonds carry the LAN trunks; storage and the tie-breaker vote ride separate NAS legs.

A cloud-style landing zone, inside one apartment

In everyday termsEight rooms, locked doors, and a written rule for every door that opens — a hotel floor plan for devices.

View 04 — network and landing zone; guest moves in progress.

A landing zone is a network floor plan settled before anything moves in: which zones exist, what may cross between them, and who may change that. This is the same public/private/restricted pattern the cloud ones use, scaled down to one physical fabric. Every inter-zone path crosses the edge and is denied unless an explicit flow allows it. The published front door lives in Z-DMZ (Traefik) — the one zone the internet may reach, held apart so that a break-in there lands nowhere else. A few documented NAS resilience ports intentionally bypass that zone. Plex sits in the smart-home and AV zone with the devices it streams to. The private zones separate the household’s trusted hosts, smart-home devices, guests, and one “far side not ours” segment that gets internet and nothing lateral. The restricted zones carry the secrets vault and quorum, switch, AP and KVM management, and a camera zone. Stable internal identity rides a private ULA /48 — an IPv6 range generated for this network alone and never routed on the internet. The ISP prefix rotates; names and policy do not.

Honest limits: intra-zone traffic is not filtered, the trusted zone remains broad, and per-workload microsegmentation — a rule per workload rather than per zone — is a deliberately separate hardening phase after the migration completes.

How a change reaches reality

In everyday termsThe building-permit process: application, plausibility check, reading, stamp, inspection. The AI writes the applications; the stamp stays with the human — who is also, in a household of one, the person who ordered the drawing.

AI changes the typing capacity, not the authority. Agents discover, propose and review. A human approves merge and gated infrastructure mutation. Verification observes the resulting system.

View 05 — change delivery.

Change delivery

Infrastructure mutation is CI-only except for documented break-glass recovery: an emergency path written down in advance, for the cases where the pipeline cannot reach what is broken. A clean plan proves intent; acceptance against the live system proves completion. Home Assistant is the deliberate exception in transport: its container pulls and validates main itself, fail-closed — there is no CI push into the smart home.

View 06 — automation. Five OpenTofu states, one lock each.

Delivery and automation

An OpenTofu state is the record of what the tool believes it has built. Five of them — the virtualisation hub, the network fabric, the edge pair, DNS, and the Wi-Fi controller — are separate lock domains: independent changes run in parallel while each state stays serialised. The per-node edge states were retired on 2026-08-23, and the pipeline now refuses to run against them. Merge-request pipelines are read-only — plan and check only. A plan reads the shared state and the plan-side secrets; the credentials that can change anything are bound to the protected branch and reach only the apply job. Apply is a human pressing a button on an on-LAN runner, with short-lived OpenBao credentials scoped to exactly that job. Shared CI templates provide the defaults, but review and protected branches remain part of the control — a consumer pipeline can still weaken an inherited rule, so the guard is the branch rule, not the template.

Two guards run fail-closed. One scans the project’s entire git history for credentials, not just the change in front of it, and it runs on every pipeline in every repository; the other reads every infrastructure declaration for misconfiguration, in the repositories that carry one. Neither is allowed to warn instead of failing.

Verification is its own graded pass, not a look at the logs. Eleven invariants are read back off the switches themselves and graded one by one; a twelfth is checked against the source text before anything runs. The run then prints its own blind spots beside the results. A verifier that reaches the switches only through the plane it audits cannot see a break in that plane. The graders themselves run against a synthetic network with a fault seeded for each invariant, so a grader that has stopped noticing gets caught.

Drift is checked in two directions. One comparison holds the running world against the record and catches a change made around the pipeline; the other holds the record against itself and catches a change that was merged and then never applied, which is a different failure and needs its own alarm. Neither is allowed to soft-fail — a drift check that warns instead of failing is a false green.

The pipelines are shaped differently from repository to repository, and none of them carries every stage — a repository runs the ones its content needs. The kinds of check, and the jobs that carry them:

What happens	What it does	Jobs
Guard and lint	shared checks, repository drift protection, and fail-closed scans for credentials and misconfiguration	`lint:all` `lint:config-drift` `secrets:scan`
Model and plan	derived network consistency plus change preview	`model:check` `tofu:plan`
Apply	manual, environment-scoped and state-serialised	`tofu:apply` `deploy:hosts`
Verify	cross-plane invariants and service checks	`net:verify` `validate:ha-config`
Detect drift	scheduled comparison in both directions: the world against the record, and the record against itself	`tofu:drift` `dns:delegation`
Recover	explicit reconcile and credential-safe recovery paths	`edge:reconcile-import` `bao:snapshot`

The migration: build the target without reinstalling the nodes

In everyday termsRenovating while living in the house: build the new kitchen beside the old one, move over, and only then rip the old one out.

The sequence is deliberately staged and reversible until final retirement, in four stages: establish the new substrate, converge the existing nodes in place, prove recovery and failover, then seal.

View 07 — the migration; the substrate stage is done and guest moves are in progress, high availability is armed and both cluster drills passed, the restore drill is still owed. No physical node reinstall and no node rename.

Migration plan

As of 2026-09-08 the substrate stage is done: console access and backups were proven before anything mutated, the fabric renumber is applied, both replacement firewalls are live with the WAN recabled directly to the nodes, and the shared storage plane exists. Guests are moving in recovery-tier order. Then each node is drained and converged behind its working firewall peer — the helper member is already retired and the tie-breaker vote is live. High availability is armed and both drills have passed; the whole-fleet restore drill is still owed. Rollback assets are retired only after a stability soak, and that step is the only irreversible one.

The decisions that define the platform

In everyday termsThe house rules, written down once so nobody has to argue them again at 2 a.m.

Git records intent; observation decides truth. Plans and documentation never substitute for live acceptance evidence.
The edge is a fixed pair, not a floating guest. One OPNsense VM per node; CARP and pfsync move the service, not the VM. One firewall is the configuration authority and pushes every synced section to its partner over the vendor’s own sync channel; git is the only origin of that configuration, and nothing is ever authored on the standby.
Storage follows recovery tier. Repair-path guests stay NAS-independent on replicated local ZFS; suitable VMs use shared NFS.
ULA carries identity and policy. The ISP prefix rotates; stable internal names and rules use a private ULA /48.
The quorum vote is a vote, not a compute node. Two physical members plus a dedicated tie-breaker service — on the storage box today, ruled in September to move onto the access switch, which will carry it in a container of its own.
Eight zones now; workload isolation later. Risk-class boundaries are fixed without pretending intra-zone microsegmentation already exists.

Accepted residuals

One DSL line means firewall failover redials; it is node redundancy, not link redundancy.
Simultaneous NAS and node loss removes quorum and needs confirmed-peer-down break-glass.
Loss of the core switch chassis isolates switched traffic, but not the pair itself: two of the cluster’s three internal links and the firewalls' shared state ride cables that run directly between the nodes.
Intra-zone traffic is not filtered and the trusted zone remains broad.
Per-workload default-deny is a separate hardening phase after the migration completes.

Three incidents that changed the design

Sole-edge outage — promoted the firewall pair from “later” to the plan of record.
Docker on NFS — established explicit local-ZFS exceptions and storage tiers.
Layer-2 leak — replaced fragile segmentation with eight routed risk zones and derived invariants.

Audited from the outside

In September the whole estate was audited against an enterprise rubric, from the outside in, and scored 45 of 132. The method, the cost and the places where the audit was wrong about itself are in 45 of 132; the findings themselves are not published.

Everything above was checked on 2026-09-08. What runs was read from the cluster’s own API and from the smart-home hub; the switch and firewall planes were read from the declarations CI applies to them, not from the devices themselves on that date. Planned items are marked as planned.