Benchmarks anyone can replay
Real tasks, the same prompt across runs, a published success definition, and CSV/JSON artifacts. The run is reproducible because every input and output is saved.
Quick answer
What is Harpd's benchmark methodology?
Harpd benchmarks run real tasks with the same prompt across multiple runs, publish the success definition and prices used, and save CSV/JSON artifacts so the whole run is reproducible. The benchmark is currently PLANNED / MODELED — no measured results are published yet, and modeled estimates are visually distinct and must not be cited as results.
- Method
- Real tasks, published success def
- Artifacts
- CSV + JSON per run
- Status
- Planned / modeled
Data source: Harpd benchmark methodology
Why it matters
A methodology page is only credible if the procedure is specific enough to replay. By publishing the exact steps, the success definition and the saved artifacts, Harpd lets anyone reproduce or challenge a benchmark result instead of taking a headline number on faith.
Limitations
- The benchmark is currently PLANNED / MODELED — no measured results are published yet.
- Modeled estimates on /benchmarks/ are visually distinct and must not be cited as measured results.
- Sample sizes are 80–240 tasks, not thousands; success criteria are task-specific and do not transfer to every workload.
Data source: Harpd benchmark methodology
Harpd benchmarks run real tasks with the same prompt across multiple runs, publish the success definition and the prices used, and save CSV/JSON artifacts so the whole run is reproducible. IMPORTANT: the benchmark is currently PLANNED / MODELED - no measured results are published. Modeled estimates on /benchmarks/ are visually distinct and must not be cited as results.
- Real tasks, same prompt, multiple runs
- Published success definition + prices used
- CSV/JSON artifacts for every run
- Currently PLANNED / MODELED — no measured results
The procedure
- 1
Pick real tasks
Tasks come from actual workloads — extraction, code, classification, prose — not synthetic leaderboard questions.
- 2
Hold the prompt constant
The same prompt is used across every run and every model so differences reflect the model, not the wording.
- 3
Run multiple times
Each task is run several times per model to capture stability and variance, not a single lucky output.
- 4
Score against a published definition
Success is defined up front (schema valid, tests pass, F1 threshold, judge + human sample) and applied identically to every model.
- 5
Save artifacts
Every input, output, score, and the prices used are written to CSV/JSON so the run can be replayed exactly.
Reproducibility
- Success definition is published alongside the results, so "pass" is not moved after the fact.
- Prices used are the verified values from /data/llm-pricing.json, captured at a known
verifiedAt. - Runner repo is open: clone it, point it at the same task set, and replay.
- Steps to reproduce: install the runner, load the task manifest, run the evaluation, then compare the generated CSV/JSON against the published artifact.
Critical: this is not yet measured
The Harpd benchmark is currently in a PLANNED / MODELED stage (target ship 2026-09-15). We have published no measured results. Any numbers on /benchmarks/ that are modeled are visually distinct from measured data and must not be cited as results.
This honesty is the entire point of the research program. When the measured benchmark ships, results will be published with the artifacts above and clearly separated from anything modeled beforehand.