Evals & Models

Model Settings & Persona Routing

One screen to choose how the persona reply path routes across models, with cost/speed/quality bars grounded in real usage — replacing the hardcoded provider order.

Dated 2026-07-03Programme Evals & Models

Situation

Persona replies currently route through a hardcoded switch in providersForTier. This spec gives the operator one screen to pick how the fast/capable/deep tiers route, replacing that hardcode with config-driven data and surfacing a real cost/speed/quality trade-off per option — directly serving the estate's cost constraint.

Scoped to persona reply routing only; a second "doer model settings" section (which models Q/Goose uses for dev tasks) is an explicit follow-on. This spec is written as a portable handoff package: intended as the first real dogfood task for the doer (Goose/OpenHands/Claude-Code-headless) once the estate action plan's Phase 0/1 land, and its UI requires the operator's explicit visual sign-off against the brand guide before building.

Key decisions

Presets-as-slider

Four options — thrifty (local-first), balanced (cloud-first with local fallback, today's interim default), sharp (Sonnet/Opus-led), and custom — not a continuous 0–100 slider and not a granular per-provider matrix.

Custom is lightweight

Custom only picks the lead model per tier; the system auto-appends the remaining available providers as fallbacks in a sensible order.

Quality bar from the eval battery

Shown as indicative and reads "n/a" when a model has no eval score, which is expected for local leads until they're evaluated.

Preset orderings composed, not eval-only

Orderings come from a model profile (cost × speed × quality × role), grounded in the real provider inventory, not from eval scores alone — cost/latency/role-fit matter too.

Config-driven routing

A RoutingConfig ({ preset, customLead? }) persists in CONTEXT_KV under config:routing; the Worker reads it per request instead of the hardcode. Default absent → { preset: 'balanced' }.

Architecture

A pure resolver expands the active preset or custom leads into ordered provider lists; the Orchestrator consumes the resolved ordering, and a discovery route feeds Custom's live model picker.

src/routing.ts

PRESET_ORDERINGS map and the pure, unit-testable resolveRouting(config, registry) — skips any unavailable provider.

src/providers/factory.ts

providerFromId(id, env) maps a family-prefixed id (anthropic/·openrouter/·ollama/) to a live provider; null for unreachable/unknown, reused by the future eval-on-demand feature.

src/orchestrator.ts

providersForTier reads the resolved ordering instead of the hardcoded switch.

src/routes/config.ts

GET/POST /config/routing (config + resolved orderings + bar data, auth-gated, validated) and GET /config/models (live discovery: Ollama /api/tags, curated OpenRouter shortlist, static Anthropic set).

src/lib/spend.ts (extended)

Cost-projection helper: recent turn volume × each option's lead-provider rate, scaled to a monthly £ figure.

web Settings → Model

The segmented-slider UI plus Custom's three lead-model dropdowns, cost/speed/quality bars, and the £/month figure — design-gated on brand-guide sign-off.

Risks & guardrails

Design sign-off is a hard gate

The UI must be mocked and explicitly approved against the brand guide before building — this is called out as a blocking requirement, not a suggestion.

Quality bar gaps

Local and newly-picked models show "n/a" until eval'd; the future "Evaluate this model" action (cost-estimate → confirm → run → cache) is deferred to v2 and depends on a job runner that doesn't exist yet.

Custom picker scope

Full OpenRouter catalogue browsing (400+ models) is deliberately deferred as a footgun; v1 ships only openrouter/auto plus a curated shortlist.

Dependent on doer substrate

Handoff assumes Phase 0/1 of the estate action plan have landed; the implementation plan is deliberately not written yet to avoid staleness.

Delivery

Resolver + factory

resolveRouting and providerFromId, unit-tested for each preset's expansion, unavailable-provider skipping, and custom-lead fallback to balanced on an invalid lead.

Route + orchestrator wiring

/config/routing and /config/models, auth-enforced; providersForTier consumes the resolved list; backstop still terminates every tier (regression guard).

Cost/speed/quality bars

Server-side computation from spend.ts, the Ollama latency ring buffer, and evals/baseline.json, so the client only renders.

Web UI (design-gated)

Settings → Model segmented slider, Custom dropdowns fed by live discovery, bars + £/month recompute live from the chosen leads.

Dogfood dispatch

Built as the first doer dogfood task — dispatched from Mallory, PR opened under the bot identity, reviewed and merged by the operator.

Out of scope

Doer model config (which models Q/Goose uses for dev tasks) — separate follow-on spec, same principlesArbitrary user-created/named presets beyond the one editable Custom slotFull per-tier drag-reorderChanging ElevenLabs/TTS or the classifier modelFull OpenRouter catalogue browse/search (v1 ships curated + auto)Evaluate-a-model-on-demand (v2+, needs a job runner)

spec · docs/superpowers/specs/2026-07-03-model-settings-persona-routing-design.md
plan · docs/superpowers/plans/2026-07-03-model-settings-persona-routing.md

Mallory · Estate briefing