Evals & Models

Eval Battery

A repeatable, real-model scenario battery across four dimensions — the quality-regression baseline the council wanted before trusting heavier automation.

Dated 2026-06-19Programme Evals & Models

Situation

Mallory had no repeatable way to catch a regression when persona, context, or routing changed. The eval battery is a fixed scenario set run against real model output, scored deterministically where possible and by an LLM judge where not, so drift shows up before it ships — and so the later model-eval track (cheaper models) can reuse the same harness unchanged.

Mallory is he/him (Gareth Mallory / "M"), a commonly-mis-defaulted detail — so it is load-bearing for the eval: the voice rubric and the judge prompt must use he/him throughout, the judge is told his pronouns so it doesn't "correct" correct usage, and at least one scenario checks he self-refers correctly.

Key decisions

Four dimensions, all scored

Persona/voice fidelity, recall quality, capability honesty, and reasoning/answer quality — reasoning gets the full ~5–8 scenario set, equal to the others.

In-process, real models

A layered harness via npm run eval (outside the vitest unit suite) exercises the real persona/context/turn logic as far as practical per dimension — bugs live in the plumbing, so the model isn't mocked.

Hard asserts + judged scores

Deterministic assertions are hard pass/fail; an LLM-as-judge gives 1–5 rubric scores tracked against a committed baseline.json plus a soft floor.

Judge model = Sonnet 4.6

Fixed as a config constant, deliberately not Opus, so the model isn't grading its own Mallory answers, and it stays cheaper to run repeatedly.

Scoring gate

A run fails on any hard-assertion failure, a dimension mean below the 3.5 soft floor, or a drop of more than 0.5 versus baseline — never exact equality, since the judge is stochastic.

Architecture

A new top-level evals/ directory, outside the Worker bundle and outside the vitest unit collection, with a clean split between pure/testable scoring and the live, real-model runner.

evals/scenarios/*.ts

Data-only scenario files per dimension (persona, recall, capability, reasoning) — the contribution surface for adding new cases.

evals/runner.ts

Per scenario: assemble real context, call the real model, run assertions, call the judge.

evals/judge.ts

LLM-as-judge (Sonnet), temperature 0, strict rubric-anchored prompt → { score, rationale }.

evals/score.ts

Pure aggregation: baseline delta, soft floor, pass/fail — holds all scoring logic so the runner doesn't.

evals/report.ts

Renders a run to timestamped JSON + Markdown with per-dimension pass-rate, mean score, and failure rationale.

evals/baseline.json

Committed scorecard; re-snapshotted deliberately via --update-baseline after a reviewed improvement.

evals/fixtures/

Seeded recall corpus (conversations + vault notes) loaded into a local D1, never touching prod.

Risks & guardrails

Cost

Real model + judge calls cost money — on-demand only, never CI, with a concurrency cap and an estimated-token-cost line printed before/after each run.

Nondeterminism

Temperature 0 where the provider allows; floor and baseline-delta checks use a tolerance band rather than exact equality, and v1's baseline is a single reviewed run (multi-run averaging deferred).

Misgendering in the judge

The judge prompt must be told Mallory's he/him pronouns explicitly, or it risks defaulting to she/her or falsely "correcting" a correct reply.

Reuse dependency

The model-eval track (comparing cheaper models) depends on this harness's provider registry staying swappable — designed in now, built later.

Delivery

Phase 1 — pure core

Types, assertion evaluators, score.ts, judge-output parser, and the report renderer — all unit-tested in the normal vitest suite.

Phase 2 — live harness

Config + provider wiring, per-scenario prompt assembly, a local-D1 recall environment via getPlatformProxy, and the judge call + runner itself.

Phase 3 — baseline + docs

Seed scenarios and fixtures, run the battery for real, review the scorecard by eye, and commit the first baseline.json.

Out of scope

Model-eval comparison track (cheaper/local models) — separate spec, reuses this harnessWeb-admin-for-OAuthCI integration (real model calls cost money)Multi-run baseline averaging (single-run baseline for v1)

spec · docs/superpowers/specs/2026-06-19-eval-battery-design.md
plan · docs/superpowers/plans/2026-06-19-eval-battery.md

Mallory · Estate briefing