Measure on real tasks, publish the method
Every number Harpd shows traces back to a documented method. We measure on real tasks, with published methods, and we separate measured results from modeled estimates — always.
Harpd uses four methodologies: sourcing verified model prices, deciding when a cheaper model is safe to switch to, computing cost per successful task, and running reproducible real-task benchmarks. Each is written down and reproducible.
- Verified prices from official provider pages
- Real-task quality gates for model replacement
- Cost per successful task, not per token
- Reproducible benchmark runs with published artifacts
The four methodologies
Each links to a full, detail-level write-up.
1 · Model pricing
How we source every model price from official provider pages, attach a verification timestamp and source URL, flag deprecated models, and publish a single machine-readable source of truth.
Live data2 · Model replacement
How we use shadow testing and per-task quality gates to decide when a cheaper model is actually safe to switch to — and the rule that we only switch when the candidate passes your bar.
3 · Cost per successful task
The metric that matters: total workload cost divided by successful tasks. Cheap-per-token models that fail often are not actually cheap once retries are counted.
4 · Benchmarks
How we run reproducible, real-task evaluations: same prompt across multiple runs, published success definitions, and CSV/JSON artifacts so anyone can replay the result.
PlannedOur measurement principle
We hold one line across all four methodologies:
We measure on real tasks, with published methods, and we separate measured results from modeled estimates.
Three consequences follow from that principle:
- Real tasks, not toy prompts. A model is judged on the work you actually run — JSON extraction, code generation, classification, prose — not a generic leaderboard question.
- Published methods. Every methodology page states the exact procedure, the success definition, and the data used, so the measurement is repeatable.
- Measured vs. modeled, never mixed. If a figure is modeled or not yet measured, it is labeled as such and visually distinguished. We never present an estimate as a result.