Which box went off
A VM running on the Tower was switched off on 29 August. The Hub VM was not, and neither was the Tower host itself — the Goose container on it never missed a beat. Two earlier versions of this page got that wrong in two different ways, so the corrections, and how the estate talked itself into each, come first.
Start here
The estate did not lose the host it thought it had
It said the Hub VM died at ~12:00 BST of its own accord and was not coming back. That came
from the decommission plan, which inferred the death from two vault artifacts —
_generated/status.md last written 12:00:01 BST against a claimed
15-minute cadence, and _generated/asana-tasks.md last written 06:45, with "the
12:00 run never happened".
Both readings were wrong. status.md commits three times a
day — 06:00, 11:00 and 17:00 UTC, the same pattern the day before — so a
12:00 BST timestamp read at 16:00 BST is an ordinary gap. asana-tasks.md commits
only when its content changes, and the 12:00 run found the same 96 open tasks as the 06:45 one.
A cadence was assumed rather than checked, and a healthy box was pronounced dead.
That is the estate's own signature failure, arrived at from the other side: a confident conclusion from a stale read. It is the third time this pattern has produced a wrong diagnosis here, after the 13 August briefing's dead-Tower story and the 8 August map's outage narrative. The tell is the same each time — an artifact's timestamp treated as a heartbeat without confirming what writes it, or how often.
Read live from the vault and the public endpoints, 21:00 UTC
What is up, and what is down
| Host | State | Evidence |
|---|---|---|
| Hub VM | Up — last confirmed 18:00 BST | Its own 18:00 render: mcp-server ok, context-push "pushed digests", cron-watchdog, signal-freshness-watch, vault-sync 17:40, d1-backup 04:00. |
| goose container | Up | Health published every 15 minutes through 20:45 UTC; 19 jobs, 17 of them ok. |
| GitHub Actions | Up | calendar-sync 18:56 UTC; asana-tasks' first scheduled fire 19:33 UTC. |
| Cloudflare edge | Up | api.mallory.chat answers Access; design.mallory.chat serves. |
| Tower — the host | Up | It runs the Goose container, which published throughout. |
| Tower — the VM on it | Off — deliberately | ollama-tower.mallory.chat → HTTP 530, Cloudflare error 1033 (tunnel down). |
Ollama is confirmed gone with that VM. Whether Hermes was on it cannot be
checked from here — hermes-api and tower-hermes sit behind
Access and answer 302 whether their origin is alive or dead. If Hermes was on that VM, the Q
Delegate lane is down with it, and that is the user-visible loss. The Tower's own
bare-host crontab (hermes-backup at 04:30, model-research on Mondays)
still fires, because the host is up — though what hermes-backup has to copy
depends on the same open question. Chat, recall, nudges and the briefings do not touch the Tower
at all.
Every other estate hostname sits behind Cloudflare Access, which answers 302
whether the origin behind it is alive or dead — so hermes-api and
tower-hermes look identical up or down. ollama-tower is not
Access-gated, so it surfaces the tunnel's real state. It is currently the only
externally decisive liveness probe the estate has, and it is one config change away
from not being.
An earlier version of this page argued the opposite — the container kept publishing while
the Tower "was off", so schedule.tsv's host note calling goose "the
TOWER's own" looked wrong. The inference was wrong, not the note. Per Alex: the
Tower is the host, Goose is a container on it, and what was switched off was a
VM running on that host. That is exactly why the container never noticed while Ollama's
tunnel died. Goose can run Ollama itself if it is ever wanted there; it does not today.
The lesson is the same one twice: up and down are properties of a named thing, and "the Tower is off" named the wrong one — just as "the VM is dead" read the wrong cadence.
Where that leaves the plan
The VM decommission is still a plan, not a history
The dependency sweep behind it stands: the boxes hold a filesystem, not compute, and of the VM's fourteen tracked jobs it tables eleven as safe to stop, two to move and one — the vault MCP — as the genuine blocker. None of that was tested by an outage, because there was no outage. It remains an argument from reading the repos, which is what it always was.
Two things that were reported as settled are not. The MCP is fine — it
runs on the VM, which is up, and reported ok at 18:00; the KEEP verdict and the
decision it blocks stand exactly where the plan left them. And self-update,
the heartbeats and mcp-probe are still running rather than gone with a
dead box.
asana-tasks and d1-backup are GitHub Actions workflows now, and
every secret they need is set. But asana-tasks' first scheduled fire
skipped itself: GitHub ran the 18:00 UTC cron 93 minutes late,
the London-hour gate saw hour 20, and the run logged "the DST pair skipping its off-season
half, not a failure."
The gate could not tell a delayed live fire from the off-season twin, and reported the wrong
reason when it dropped one. That fire was a harmless twin — the damaging case is the same
delay on a target fire, and for d1-backup, with one target a day, a single
mis-skip is the whole day's backup with nothing red anywhere.
Fixed the next morning in mallory-runtime #155: one cron entry per
candidate hour, and the gate judges each fire by the hour it was scheduled for rather
than the hour it arrived, so a delay of any length cannot change the verdict. Widening the
window would not have worked — the DST pairs are one hour apart by construction, so any
tolerance big enough to absorb a delay also admits the twin.
The other half of the fortnight, and unaffected by any of the above
Who is allowed to interrupt, and when
The estate believed it had a quiet-hours rule. It had one in two places out of six:
shouldSendNow and the nudge flush enforced it, while the briefing, the end-of-day
wrap, the dead-man's-switch and the reminder push all called notify() with no clock
at all. Four of the six producers were outside a window the estate thought it had.
The gate now sits in notify(), the one function every push passes through.
| Change | What it does |
|---|---|
| 09:00–22:00, London | The boundary the vault always stated. The one copy that could actually buzz a phone said 07:00, mid school-run. high priority no longer pierces it. |
| Withheld, not dropped | A held push writes a push_withheld row naming the gate and the local hour. The flush and the reminder push deliver on the next in-window tick. |
| Mutes key on the member | One roll-up covered the briefing, the wrap, the flush and every trigger nudge, so muting it was all-or-nothing. Keep the briefing, drop the trigger nudges. |
| Piercing is a named tier | Two things cross the boundary: the hub dead-man's-switch and the weekday traffic warning, which self-gates to 06:00–09:00 and would otherwise never fire again. |
It exposed three quiet failures, all fixed the next day
| Found | Fix |
|---|---|
| The morning briefing pushed into the closed window, every day. The Updates row is written before the push, so the content existed and only the interrupt died — invisibly, unless someone read the withheld counts. | Move the job, not the boundary. The briefing now fires at 09:00. |
Held urgent nudges had no reader at all. A held nudge was written queued keeping its own urgency, while the only reader of queued rows selected the batch urgency. |
The queue is keyed on status alone, plus a tested invariant: nothing may be written to a state no reader will ever select. |
| The all-clear could wake him at 07:15 on a Saturday. Roughly half the dead-man's-switch's volume is "signals are back", which is never urgent. | Split the exemption by member: alerts pierce, all-clears are held to the first in-window tick and say how long they waited. |
Every one of these is the same move: act on the member, not the roll-up. Mutes, the exemption tier, the withheld ledger and the all-clear split all resolve one collapsed category into the things it actually contained.
Where the wrong diagnosis reached the code
Four signals became two, and one of them shouldn't have
| Signal | State | Verdict |
|---|---|---|
agenda | Live — ships from the Actions calendar-sync run. | Watched, 6h threshold. |
tasks | Producing — the VM copy still runs. | Watched, 18h threshold. |
activity | Genuinely dead since 25 Aug — asana-activity fails with "OAuth session expired and could not be refreshed". | Retired correctly. |
status | Still being produced. context-push shipped digests at 18:00. | Retired on the false premise — and stays retired, decided 30 Aug. |
The argument for retiring status was that its producer had been killed deliberately,
so the dead-man's-switch would page every 24 hours forever about a job nobody was going to fix.
That argument was sound — and its premise was false. The producer is alive, so what shipped
is that Mallory stopped watching the health signal of a running host.
Decided 30 August: it stays retired. Alex's judgement is that the signal never worked well enough to be worth watching. So it is now a deliberate retirement of a weak instrument rather than an accident downstream of a wrong diagnosis — which is the distinction worth having on the record, whatever the outcome.
Independent of which box is off
What is actually broken
| What | Since | Cause |
|---|---|---|
asana-activity, decay, heartbeat | 20–25 Aug | One expired claude -p OAuth session, three jobs. Re-authing the CLI fixes all three. |
connector-google_mk, mk-inbox | 28 Jul | The MK Google connector's refresh fails; the inbox job fails downstream of it. |
/status overall | — | Reads RED, and cron-watchdog reports 3 overdue. |
Seven paused jobs on the VM are not in that list: they are twins superseded by the goose container and are paused on purpose.
Designed, not built
The estate is getting a way to be told it was wrong
One design cycle landed in the same fortnight, and it is in the archive as Conversational Feedback. Today, when the briefing puts the wrong thing in the morning list, there is no way for Alex to say so. The spec is the plumbing for a correction spoken in the conversation already happening: a deterministic gate, a spoken confirm before anything is written, and a routing decision between a standing preference and a task for Q.
Its own history is the argument. Four self-reported surfaces — the journal, the daily note's own text slot, a focus field, any feedback route — have returned four zeros. So the spec forbids itself the obvious answer: no form, no widget, no marker convention, and a stop condition that retires the channel outright if it produces fewer than three confirmed items in a month. Status: unapproved, and nothing is built.
Still open
What this leaves
- Where Hermes runs — and therefore whether the Q Delegate lane and
hermes-backupare affected. Not answerable from outside Access. - One expired CLI login holding three jobs down, and the MK connector holding two more.
- Step 2 of the Tower exit —
MALLORY_STATE_SYNCis set on the Actions host and on neither the VM nor the container, soruntime_job_healthis a list of hosts that are reporting, never a census. - The decommission plan's §2a, which still records the outage that did not happen.
How much to trust this
Confirmed, inferred, and unchecked
Confirmed. Everything in the state table was read directly: the vault's
_generated/ artifacts and their commit history at 21:00 UTC on 29 August, the
GitHub Actions run logs, and live HTTP probes of every *.mallory.chat hostname the
two repos name. The code claims — the delivery window, the exemption tier, the retired
signal set, the briefing's new cron — were read in the repositories at this weekend's
merge commits.
Inferred. That the Tower being off is what takes Hermes and Ollama down follows from where those services are documented to run, not from an observation of each one: their hostnames are Access-gated and cannot be probed from outside.
Not checked. The Hub VM's liveness after 18:00 BST. Its vault writes are thrice-daily, so the next one is due around 07:00 — there is simply no fresher evidence to read, and the absence of one proves nothing either way. That is the same gap that produced the error at the top of this page, so it is stated rather than filled in.
Verdict
The estate is in better shape than the last two briefings claimed, and its instruments are worse. Nothing is down that was not switched off deliberately. What keeps failing is the reading: three times now a stale artifact has been read as a dead host, and this time the misreading reached production and retired a live signal.
The cheap fix is not more monitoring. It is that every artifact used as a heartbeat should state its own cadence, in the file, next to the timestamp — because the instrument that gets read at 4pm on a Saturday is a Markdown file, and it currently gives a reader no way to tell a normal gap from a corpse.
design.mallory.chat/reviews/briefings · 2026-08-29, corrected twice — 22:00 UTC (the Hub VM was not dead) and 2026-08-30 (Goose is on the Tower; a VM on it was what went off).