Evals & Models

Eval Battery — Operationalisation

Turning the already-built eval harness into a standing regression safety net — a committed baseline plus a change-driven gate and a scheduled canary.

Dated 2026-07-18Programme Evals & Models

Situation

The Round 1 estate review flagged an eval battery as a do-before-heavier-automation priority, and the roadmap still lists it "still open." But the code tells a more precise story: a substantial v1 harness already exists (evals/, PRs #65/#66), exercising the real edge Worker turn, scoring four dimensions with hard assertions plus a Sonnet judge, and already able to compare a run against a baseline and exit 0/1.

So the harness is not the gap. Three things are missing that turn it from a script you can run into an actual safety net: no committed baseline.json (so it can score a run but not say it got worse), no trigger (drift is only caught if someone remembers to run it), and a conflation of two different surfaces — this edge-chat battery tests the persona turn, while the Goose migration's vault-artifact parity check is a separate, un-built harness and an explicitly out-of-scope backlog item.

Key decisions

Scope = operationalise, not rebuild

Operationalise the existing edge battery only — no new eval surface, no new scoring engine, and explicitly not the Goose-parity harness.

Both triggers

A manual change-driven gate for local pre-merge runs, plus a periodic Tower canary that runs unattended — not one or the other.

Cost is not a constraint

30 scenarios ≈ 30 reply turns + 30 judge calls + 30 fast-classify calls per run, cents-to-low-dollars; the canary runs the full battery, no lighter subset.

Thresholds kept as-is

Soft floor 3.5 and delta tolerance 0.5 in evals/config.ts are sensible and unchanged; any hard-assertion failure still fails the run.

Single-run baseline

Established from one reviewed run, not a multi-run average — multi-run averaging is a token-cost fix for a flakiness problem not yet observed (YAGNI), revisit only if the canary proves flaky.

Architecture

Three additive pieces around the existing harness: a committed baseline with a re-bless ritual, a path-aware local gate, and a scheduled remote canary that reuses existing transport and alerting rather than building new ones.

evals/baseline.json (commit)

The unblock: run once, review the scorecard by eye, commit — activates the regression check for both triggers.

Re-bless ritual (same PR)

An intentional persona/context/recall/routing change runs --update-baseline in the same PR so the scorecard delta is visible in review.

pre-push git hook

Fires only when the push touches eval-relevant paths (persona/context/recall/routing source); nudges and can soft-block rather than being pure memory or a non-path-aware checklist line.

Hermes weekly Tower cron

Runs npm run eval on the Tower where the repo is already cloned; captures exit code + Markdown report.

Existing Hermes digest inbound port

Reused with a new eval payload type to carry the result back to the edge — deliberately not a new endpoint, so no new inbound token to rate-limit or rotate.

/status + notify()

Last-run verdict and per-dimension means surface on /status; only a regression pages the operator via the existing FCM/email notify(), reusing the dead-man's-switch alerting pattern.

Risks & guardrails

Auth durability on the Tower

npm run eval needs its own durable ANTHROPIC_API_KEY on the Tower (separate from Hermes's OPENROUTER_API_KEY), and it must sit under the same watchdog/heartbeat as the migration's other auth — an unmonitored, silently-lapsed eval key would recreate the exact 2026-06-30 silent-auth-lapse failure mode this whole migration exists to eliminate.

Judge stochasticity

The judge is inherently stochastic; the 0.5 delta tolerance over 6–10 scenarios per dimension is expected to absorb ordinary jitter, but this is unproven at canary cadence.

Two eval surfaces, one name

"Eval battery" colloquially covers both this persona-turn battery and the unbuilt Goose vault-artifact parity check — keeping them named and scoped separately avoids the roadmap conflating the two.

Delivery

A. Establish the baseline

Run npm run eval once with a real key, review the per-dimension means by eye, investigate anything surprising, then commit evals/baseline.json.

B. Local gate

Document the rule (run before merging persona/context/recall/routing changes) and add the path-aware pre-push hook — the cheapest part of the work.

C. Tower canary

Wire the weekly Hermes cron, the eval payload on the existing digest inbound port, /status surfacing, and notify() paging on regression only.

Watchdog coverage

Slot the canary's ANTHROPIC_API_KEY into the existing Tower watchdog set, verified by a deliberately-expired-key test.

Document the re-bless ritual

Record the in-PR --update-baseline process in evals/README.md.

Out of scope

Goose routine-parity harness (separate backlog item — different surface, diffs vault artifacts, not built)The eval:models model-comparison trackAuthoring new scenarios (existing persona/recall coverage is already adequate)

spec · docs/superpowers/specs/2026-07-18-eval-battery-operationalisation-design.md
plan · docs/superpowers/plans/2026-07-18-eval-battery-operationalisation.md

Mallory · Estate briefing