Evals & Models
Model Eval
Runs the real Mallory turn on candidate cloud models via OpenRouter and scores each with the existing battery — a quality × measured-cost comparison per dimension.
Situation
Extends the eval battery (built on main) to answer the roadmap's cost question. Rather than benchmarking raw models, it routes the Orchestrator's fast/capable/deep tiers through each candidate so it tests Mallory-on-model-X, and reports quality per dimension against measured (not estimated) cost.
This is informational only — it does not touch the Worker's production provider layer or auto-change routing. Live cost-routing is a separate, parked track; this produces the evidence a future routing decision would use.
Key decisions
Cloud models only, via OpenRouter
One OpenAI-compatible API reaches all candidates and returns per-call cost/usage, so cost is measured rather than estimated.
Eval-only provider
The OpenRouter AIProvider lives entirely in evals/, never in the Worker's src/providers — the production path is untouched.
Runs on the Tower
Where Hermes's OPENROUTER_API_KEY already lives and the repo is cloned, so "Hermes triggers it" needs no new plumbing.
Full Mallory turn per candidate
Classify + reply both point at the candidate model, testing the persona-on-model behaviour rather than a raw completion.
Fixed judge across all runs
Sonnet via OpenRouter judges every candidate identically, keeping scores comparable across the matrix.
Output shape
A model × dimension matrix (mean score + pass-rate), total measured cost per model, and a per-dimension "which can downgrade" verdict.
Architecture
New eval-only pieces layered on the existing battery; the shared per-scenario pipeline is extracted so the single-model and multi-model runners don't duplicate logic.
AIProvider against OpenRouter's chat-completions API; captures per-call usage/cost and exposes a running total.malloryOn(modelId) builds an Orchestrator whose tiers all use the OpenRouter provider for the candidate; the judge becomes a fixed OpenRouter Sonnet call.npm run eval:models entry — runs the battery per candidate, collecting { scorecard, costUsd }; supports --models and --scenarios overrides.runner.ts so both entry points reuse it rather than forking the logic.Risks & guardrails
Unconfirmed provider response shape
OpenRouter's exact usage/cost field (inline usage vs a /generation lookup) is confirmed against a real call at build time and parsed defensively.
Model id drift
Candidate ids are sourced from OpenRouter's live catalog and pinned where possible, but availability must be reconfirmed at build/run time.
Informational only
The report never auto-changes production routing — a human still decides whether a downgrade verdict is acted on.
Cost of the run itself
~6 models × ~23 scenarios × (turn + judge call) ≈ ~280 calls per run, mostly on cheap models but still a real spend, held for an authorised Tower run.
Delivery
OpenRouter provider
Build the eval-only AIProvider with unit-tested request shaping and usage/cost parsing against a stubbed fetch.
Extract the shared pipeline
Pull the per-scenario run loop out of runner.ts so runModels.ts reuses it cleanly.
models.ts + reportModels.ts
Wire malloryOn and the fixed OpenRouter judge, then build the pure model × dimension × cost matrix renderer.
runModels.ts
The multi-model runner entry point with --models/--scenarios flags.
Verify, docs, PR
Held for an authorised live run on the Tower with OPENROUTER_API_KEY; the single-model Anthropic npm run eval stays as-is.
Out of scope
docs/superpowers/specs/2026-06-19-model-eval-design.mdplan ·
docs/superpowers/plans/2026-06-19-model-eval.md