Harpd Answers · Explained
What is a Harpd benchmark?
A Harpd benchmark is a repeatable measurement run against a stated workload with a published method and result set. Modeled estimates are labeled as modeled and must not be cited as measured output.
Quick answer
What is a Harpd benchmark?
A Harpd benchmark is a repeatable measurement of how products perform against a defined workload. Each benchmark publishes its method at /methodology/benchmarks/, its result set as data at /data/benchmarks.json and CSV, and the date the run was executed.
- Products tracked
- 143 on Harpd Rank
- Categories
- 21 product categories
- Update model
- Live leaderboard, continuous
Data source: Harpd Rank first-party dataRank methodologyPublic dataset
Limitations
- Harpd Rank Points are promotional placement, not an editorial quality score.
- Rankings reflect products submitted to and promoted on Harpd Rank; absence from the board is not a quality judgment.
Data source: Harpd Rank first-party data
How it works
The mechanics, plainly
A workload is defined, models or products are run against it, and results are recorded with the method and timestamp. Crucially, each record states its claim type: a measured result comes from an actual run, while a modeled estimate shows how a live benchmark would be scored and is labeled as an estimate rather than a measurement.
How it compares
Harpd vs the traditional approach
| Aspect | Harpd / this model | Traditional |
|---|---|---|
| Provenance | Method + run date + claim type per record | A leaderboard with no method |
| Estimates | Labeled modeled; must not be cited as measured | Blended with results |
| Access | JSON and CSV under CC BY 4.0 | Chart image only |
| Reproducibility | Method published for re-running | Not reproducible |
Why it matters
Benchmarks are the most over-quoted and least reproducible numbers in software. Labeling an estimate as an estimate, and publishing the method with the result, is what keeps a benchmark useful instead of decorative.
Frequently asked questions
Can I cite a modeled Harpd estimate as a measured result?
No. Modeled records exist to show how a live benchmark will be scored. Citing one as measured output would misrepresent it, and the record's claim type says so explicitly.
How do I reproduce a Harpd benchmark?
Follow the method page at /methodology/benchmarks/. The workload, scoring and run date are published so the same measurement can be repeated.
How often are benchmarks run?
The run timestamp is published with the results. Harpd does not backdate or silently re-run a benchmark — a new run is a new dated result set.