Feature Waves

Wave 5 — Recall & Search

One more retrieval feeding the classify → route → inject → answer pipeline, unifying past conversations and vault notes under one BM25 ranking.

Dated 2026-06-13Programme Feature Waves

Situation

Mallory could hold context and dig deep asynchronously, but had no fast, in-conversation way to search his own memory — past conversations or vault notes. This closes the two limits the persona apologises for: searching the wider note archive and pulling up separate past conversations.

The guiding move is that search rides the pipeline Mallory already runs — recall is one more retrieval feeding the context block, not a new subsystem — and it stays edge-local: the Worker reads D1, the Hub pushes, the Worker never calls the Hub.

Key decisions

Two corpora, one ranked set

Past conversations (already in D1) and vault notes (pushed by the Hub) are queried through one SQLite FTS5/BM25 index with a recency boost, not two separately-ranked lists.

FTS over Vectorize for v1

Chosen because both corpora share one BM25 scale (no cross-engine score reconciliation) and need no new infra; semantic search via Vectorize is the deferred relevance upgrade, kept swap-in compatible.

Hub-pushed index over live Worker→Hub call

A live call would break the edge-local invariant and make the hot path depend on the VM being up; rejected. Trade-off accepted: vault results are up to ~15 min stale.

Capture provenance, don't classify yet

vault_notes.folder/tags and a nullable messages.context column are captured raw now; work/personal classification is deferred to a vault-refactor task that also feeds Wave 6 modes — no backfill needed later.

Results presentation: quiet footer (option A)

Inline citations in the prose plus a compact "Sources" footer, reusing the Wave 2a link vocabulary, over richer source cards (deferred).

Conversation deep-link is new, in-scope work

No message-level deep-link exists today (only ?dig=); a ?msg=<id> route pages history until the message loads, then scrolls to it — with a non-navigating snippet as fallback if pagination proves fiddly.

Architecture

New FTS5 tables for both corpora plus a pure searchCorpus/rankHits core injected into the context block the same way signals already are.

messages_fts (D1)

FTS5 virtual table over message content, kept in sync via triggers; maps back to conversation_id/message_id/created_at.

vault_notes / vault_notes_fts (D1, new)

path/title/body/mtime/folder/tags pushed from the vault, plus its FTS5 index — deliberately not named notes (that table already means Mallory's own captured notes).

POST /vault/sync

Access/service-token gated Hub→Worker ingest endpoint; batch upserts by mtime + deletes; pure payload parser is unit-tested.

searchCorpus(query, filter?)

Sanitises the query into a safe FTS5 expression, queries both FTS tables, and calls rankHits to merge BM25 + recency across sources; filter is a no-op in the MVP.

Recall context-block builder

Injects top-k hits into a new "Recall" context section, same mechanism as context.ts uses for signals; persona cites only injected hits.

Web Sources footer + ?msg= route

Renders source label + snippet + link (obsidian:// for notes, new ?msg= deep-link for conversations).

Risks & guardrails

Fabricated citations

Guarded by persona instruction: Mallory may cite only hits actually injected into context; if there are no hits he says so and offers the async dig.

FTS syntax injection

Raw user text would throw on FTS5 syntax if passed directly; a dedicated sanitiser converts it into a safe match expression first.

Vault staleness

Vault results run up to ~15 minutes behind the Hub's push cadence — accepted as fine for MVP with a clear upgrade path.

Large corpus growth

FTS5 index size on D1 needs monitoring; an exclude-list is deferred to the same refactor as context classification.

Delivery

Task 1

D1 migration — FTS tables, triggers, context column.

Task 2

Pure FTS query sanitiser (src/lib/search.ts).

Task 3

Pure ranking + row mapping (src/lib/search.ts).

Task 4

D1 search queries + searchCorpus (src/db.ts, src/searchCorpus.ts).

Task 5

Recall block + sources builders (src/lib/search.ts).

Task 6

Wire recall search into the chat route (src/routes/chat.ts).

Task 7

/vault/sync ingest endpoint.

Task 8

Persona nudge (src/persona.ts).

Task 9

Web — capture sources + render the quiet "Sources" footer.

Task 10

Web — ?msg= conversation deep-link.

Out of scope

Semantic search via Vectorize embeddings (kept swap-in compatible)Agentic search tool (model-refined mid-turn search)Work/personal context classification (needs a vault storage reorg — separate task, feeds Wave 6)Richer source cards (option B — natural fit once 2c link previews land)Any change to the async dig

spec · docs/superpowers/specs/2026-06-13-wave-5-recall-search-design.md
plan · docs/superpowers/plans/2026-06-13-wave-5-recall-search.md

Mallory · Estate briefing