Estate
Backup & Recovery
Daily D1 + Hermes SQLite exports to R2, a tiered rebuild runbook, and a break-glass kit for total estate loss.
Situation
This is the operational runbook implementing decision D14 (backup, DR & operational safety net). Backups are produced by mallory-runtime scripts (jobs/d1-backup.sh for D1, tower/hermes-backup.sh for Hermes) and uploaded to the R2 bucket mallory-backups, retained as 30 daily + 12 weekly snapshots. The document explicitly corrects a wrangler-syntax assumption: there is no wrangler d1 import — restore goes through wrangler d1 execute --file.
Only D1's full export is edge-canonical and not derivable from the vault (conversations, messages, threads, memories); Hermes SQLite, vault_notes, and FTS5 indexes are all derived/rebuildable, so their backups exist for fast recovery rather than as the sole copy.
Key decisions
D1 export is the one truly canonical backup
Conversations/messages/threads/memories live only in D1 — not recoverable from the vault — so its daily export is the backup that actually matters most.
Tiered rebuild runbook
Tier A (corruption/bad migration) uses D1 Time Travel for a seconds-fast point-in-time restore with no artifact needed; Tier B rebuilds from an R2 backup via d1 execute --file plus manual FTS5 rebuild; Tier C is total estate loss, following the full break-glass order.
Prune manifest is transient and safe-by-construction
A local, gitignored manifest drives pruning and can only ever select backups beyond the retention window — a lost manifest just means pre-existing R2 objects become invisible to pruning, not that backups are affected.
Break-glass kit inventories secrets by name only
The kit lists which secrets exist (Anthropic, ElevenLabs, VAPID, Google refresh token, service tokens, GitHub bot token, etc.) without ever storing values — actual recovery relies on a separately-held encrypted blob.
Estate-recovery order
Regain Cloudflare account/Access first, then restore secrets, then D1 (Tier B), then deploy Worker + web, then bring up Hub + Tower and restore Hermes, then re-auth connectors.
Risks & guardrails
IdP lockout is the fatal SPOF
Cloudflare Access / IdP recovery is called out as the single point of failure that no amount of uptime engineering can fix — the break-glass kit exists specifically for this.
Encrypted secret backup location is a TODO
The document flags an open TODO to record where the encrypted secret blob lives and how to decrypt it — the kit is not yet fully actionable without that.
FTS5 shadow tables don't round-trip
Rebuilding from a D1 export requires manually re-running the FTS5 rebuild command against the live schema, since shadow tables are not carried by export/import.
docs/estate/06-backup-recovery.md