Dataset
Harpd Benchmark Dataset
Long-format benchmark records (model, task, metric, score, cost, timestamp) with an explicit isModeled flag separating measured runs from modeled estimates.
Quick answer
What is the Harpd Benchmark Dataset?
The Harpd Benchmark Dataset publishes 56 long-format benchmark records — model, task, metric, score, cost and timestamp — with an explicit isModeled flag on every row separating measured runs from modeled estimates. The current data is dated 2026-08-18. It is released as open JSON and CSV under CC BY 4.0. A modeled score is an estimate and must not be quoted as a measured result.
- Records
- 56
- Update frequency
- Per benchmark run. Historical runs are never overwritten.
- Licence
- CC BY 4.0
Data source: Harpd Benchmark DatasetLast updated: MethodologyJSON endpoint
Key facts
Cite asHarpd. "Harpd Benchmark Dataset." harpd.com/benchmarks/ Accessed: 2026-08-18. License: CC BY 4.0.
What this dataset contains
Long-format benchmark records (model, task, metric, score, cost, timestamp) with an explicit isModeled flag separating measured runs from modeled estimates.
Scope. Published benchmark tasks, one record per model × metric. The current release holds 56 benchmark records and was last updated .
Harpd is an independent product discovery and ranking intelligence platform. Rank Points, where this dataset carries them, are promotional placement bought with Credits — they are not an editorial quality score, rating or review.
Fields
Every field the endpoint publishes. The list is generated from the dataset’s own schema, so a field cannot ship undocumented.
| Field | Description |
|---|---|
id | Stable record identifier (benchmark + model + metric). |
model | The model the metric was scored for. |
task | The benchmark task the model was run against. |
metric | Metric name in long format — see the metric list below. |
score | The metric value, in the unit named by `unit`. |
cost | The value repeated in USD for cost metrics; null for every non-cost metric. |
unit | Unit of `score` (ratio, ms, tokens or usd). |
isModeled | True when the record is a modeled estimate rather than a measured run. Never conflated with a measurement. |
measuredAt | When the run that produced the score was measured; null for modeled records. |
timestamp | The benchmark run’s own timestamp — data-derived, never a build date. |
source | Where the record came from, including the pricing source used for cost metrics. |
success_rate | Published metric `success_rate`, measured in ratio. |
p50_latency_ms | Published metric `p50_latency_ms`, measured in ms. |
p95_latency_ms | Published metric `p95_latency_ms`, measured in ms. |
avg_input_tokens | Published metric `avg_input_tokens`, measured in tokens. |
avg_output_tokens | Published metric `avg_output_tokens`, measured in tokens. |
cost_per_task | Published metric `cost_per_task`, measured in usd. |
cost_per_successful_task | Published metric `cost_per_successful_task`, measured in usd. |
Sample record
The first record currently returned by /data/benchmarks.json. Shown so the row shape is verifiable without fetching the endpoint; the live files move, so treat this as a shape sample rather than a snapshot.
model | task | metric | score | unit | isModeled |
|---|---|---|---|---|---|
| gpt-4o-mini | Structured JSON extraction from messy natural-language input (en + ja + zh) | success_rate | 0.88 | ratio | true |
How it is generated
Long-format benchmark records (model, task, metric, score, cost, timestamp). Each record carries an explicit `isModeled` flag: measured runs are produced by a real runner against the pinned task; modeled records are estimates and are labelled as such in the data.
Update frequency
Per benchmark run. Historical runs are never overwritten.
Each record carries its own date where the dataset provides one, so any value can be re-checked against the endpoint after the fact. The dataset’s lastUpdated field moves only when a record actually changes — never on a rebuild alone.
Access the data
The dataset itself: envelope plus records, stable field names.
/data/benchmarks.csvCSVFlat mirror for spreadsheets and quick analysis.
/data/benchmarks/READMEMarkdownLong-form README: generation, schema, licence, limitations, reproduction steps.
No API key is required for any public dataset. Rate limits, caching and versioning for the documented endpoints are described in the API reference.
Methodology
How these numbers are produced is documented in full at harpd.com/methodology/benchmarks/. A number in this dataset is citable only because that methodology exists and is public.
Licence
This dataset is released under CC BY 4.0. Attribution is required and no endorsement is implied. Reuse, redistribution and commercial use are permitted with attribution to Harpd.
Why it matters
Publishing this dataset as an open, machine-readable file lets researchers, developers and AI systems build on Harpd data without scraping the live site. Every record carries a data date and the whole dataset carries a CC BY 4.0 licence, so a number quoted from it is directly citable and reproducible.
Limitations
- MEASURED vs MODELED is strictly separated and must stay that way: if the underlying data is a modeled estimate it MUST be stated as an estimate and never presented as a measured result.
- The published benchmarks.json may contain modeled preview estimates (isModeled: true). A modeled score is an estimate, not a measurement, and must never be quoted as a measured result.
- Historical runs are never overwritten; a benchmark result reflects the model, task and pricing version it was scored with at its timestamp.
Data source: Harpd Benchmark DatasetLast updated: August 18, 2026
Cite this data
Cite the dataset, not the homepage, and always include the access date: the live board moves, while this dataset’s registered citation target stays stable.
Cite this dataset
Data: https://harpd.com/data/benchmarks.json · CSV
Method: https://harpd.com/methodology/benchmarks/
Updated: 2026-08-18
Licensed under CC BY 4.0. Attribution to Harpd is required when reused.