A cheaper model is only safe if it passes
Cheaper-per-token is not the same as cheaper-in-practice. We decide a switch with shadow testing, the same prompt across runs, and a quality gate that fits the task type.
We shadow-test a candidate model on your real tasks using the exact same prompt across runs, then apply a per-task quality gate: JSON schema checks, unit tests for code, accuracy/F1 for classification, and an LLM judge plus human sample for prose. We switch only when the candidate passes your quality bar.
- Shadow test on real tasks, not toy prompts
- Same prompt, multiple runs for stability
- Quality gate varies by task type
- Switch only when the candidate passes your bar
The procedure
- 1
Shadow test
The candidate model runs in parallel with your current model on live or replayed traffic. Production is never affected — the candidate is observed, not serving.
- 2
Same prompt, multiple runs
We hold the prompt constant and run the candidate several times to measure stability, not a lucky single shot. Variance matters as much as the average.
- 3
Real-task evaluation
Outputs are scored against the task's own success definition — the same one your production code would use — not a generic benchmark score.
- 4
Quality gate per task type
Each task type has its own gate (see below). If the candidate clears the gate at a lower cost, it is a candidate for switch.
Quality gates by task type
| Task type | Quality gate |
|---|---|
| Structured extraction | JSON schema validation — output must parse and satisfy the schema |
| Code generation | Unit tests must pass against the generated code |
| Classification | Accuracy / F1 against a labeled set above your threshold |
| Prose / open-ended | LLM judge plus a human-sampled review for tone, factuality, and instruction-following |
The rule
Switch only when the candidate passes your quality bar. A lower price is never, by itself, a reason to switch.
The gate is yours to set. Harpd reports the candidate's pass rate and the cost difference; the decision to switch stays with your quality threshold. Nothing is switched automatically without a gate passing.