Benchmarks anyone can replay
Real tasks, the same prompt across runs, a published success definition, and CSV/JSON artifacts. The run is reproducible because every input and output is saved.
Harpd benchmarks run real tasks with the same prompt across multiple runs, publish the success definition and the prices used, and save CSV/JSON artifacts so the whole run is reproducible. IMPORTANT: the benchmark is currently PLANNED / MODELED — no measured results are published. Modeled estimates on /benchmarks/ are visually distinct and must not be cited as results.
- Real tasks, same prompt, multiple runs
- Published success definition + prices used
- CSV/JSON artifacts for every run
- Currently PLANNED / MODELED — no measured results
The procedure
- 1
Pick real tasks
Tasks come from actual workloads — extraction, code, classification, prose — not synthetic leaderboard questions.
- 2
Hold the prompt constant
The same prompt is used across every run and every model so differences reflect the model, not the wording.
- 3
Run multiple times
Each task is run several times per model to capture stability and variance, not a single lucky output.
- 4
Score against a published definition
Success is defined up front (schema valid, tests pass, F1 threshold, judge + human sample) and applied identically to every model.
- 5
Save artifacts
Every input, output, score, and the prices used are written to CSV/JSON so the run can be replayed exactly.
Reproducibility
- Success definition is published alongside the results, so "pass" is not moved after the fact.
- Prices used are the verified values from /data/llm-pricing.json, captured at a known
verifiedAt. - Runner repo is open: clone it, point it at the same task set, and replay.
- Steps to reproduce: install the runner, load the task manifest, run the evaluation, then compare the generated CSV/JSON against the published artifact.
Critical: this is not yet measured
The Harpd benchmark is currently in a PLANNED / MODELED stage (target ship 2026-09-15). We have published no measured results. Any numbers on /benchmarks/ that are modeled are visually distinct from measured data and must not be cited as results.
This honesty is the entire point of the research program. When the measured benchmark ships, results will be published with the artifacts above and clearly separated from anything modeled beforehand.