Reliability & Ops
Observability & Heartbeats
Every cron — edge and Hub — is watched for silence, and the operator is paged once when one goes overdue.
Situation
Two gaps: edge crons record nothing (if the Worker's scheduled() silently stops, or a handler throws, there is zero signal — GET /status reports signal freshness, not cron execution), and Hub jobs go stale silently (status.sh only alerts on a hard fail; a job that stops running quietly degrades ok→stale→amber, which never pages).
The estate already has a pager (alert() → ntfy) and a single external HEALTHCHECK_URL ping proving the VM+cron is alive generally, but nothing per-job. This spec adds an in-estate watchdog that reuses everything already built rather than adding a third-party per-cron SaaS.
Key decisions
In-estate watchdog, not external SaaS
Edge crons beat into KV; a Hub cron-watchdog.sh polls + pages, reusing alert()/ntfy, record_status, and the state/jobs/*.json plane already in place.
External HEALTHCHECK_URL stays the total-VM-death backstop
A watchdog running on the VM can't report its own VM dying, so the existing single external ping covers the catastrophe case.
Alerts via alert()/ntfy only, not FCM
Keeps ops noise out of Mallory's user-facing notifications; FCM-via-/nudge for ops alerts is a later optional add.
status.sh stays untouched
The watchdog carries its own page-budgets, deliberately more conservative than status.sh's amber-render thresholds, so there's no shared-state coupling with the critical renderer.
Edge crons are watchdog-only for v1
Not folded into the vault status.md render — lower risk; that fold-in is a noted later nicety.
Architecture
Edge crons write a best-effort heartbeat to KV after each job; a 30-minute Hub cron pulls both edge beats and Hub job state, classifies each against a silence budget, and pages once per transition.
CONTEXT_KV.put('cron:hb:'+name, iso) after each of the 5 logical edge jobs; swallows KV errors so a heartbeat write never breaks a cron.{name: iso|null} for all 5 edge crons, read via CONTEXT_KV.list.classify($now;$budgets) → ok/overdue/unknown/paused per cron; transitions($prev) emits only down/recovered events, deduping repeat pages.state/jobs/*.json, classifies via the jq lib, pages via alert() on transitions, persists new state.Risks & guardrails
Edge-endpoint unreachable
If /cron/heartbeats is unreachable or non-200, the watchdog pages once (synthetic edge-endpoint name) and continues with Hub-only classification rather than failing silently.
Watch-the-watchman gap
A silently-dead watchdog would itself go unnoticed; a second healthchecks.io check (WATCHDOG_HEALTHCHECK_URL) is recommended but held for Alex to configure.
nudgeFlush overnight false alarms
A flat 13h budget was chosen specifically to survive the real 21:00→07:00 gap without paging; active-window modelling is deferred.
Delivery
PR-2a — mallory (Worker)
src/cron/heartbeat.ts, scheduled() wiring, src/routes/cron.ts + index registration, vitest coverage.
PR-2b — mallory-runtime (watchdog)
lib/watchdog.jq, jobs/cron-watchdog.sh, jobs/tests/watchdog.test.sh, config additions, README schedule row.
Offline test coverage
Fixture-driven (crons + budgets + prev-state) → exact transitions, covering ok→overdue, dedup on repeated overdue, unknown/paused silence, and the nudgeFlush 10h-vs-14h boundary.
Held for Alex
Confirm HEALTHCHECK_URL is set (done), optionally add the watch-the-watchman check, install the 30-min watchdog cron, and deploy the Worker for edge beats + the new route.
Out of scope
docs/superpowers/specs/2026-06-21-observability-heartbeats-design.mdplan ·
docs/superpowers/plans/2026-06-21-observability-heartbeats.md