# Harpd Benchmark Dataset

> Human-readable README for the canonical `benchmarks` dataset. Companion to https://harpd.com/data/benchmarks.json.

## What it is

Long-format benchmark records (model, task, metric, score, cost, timestamp) with an explicit isModeled flag separating measured runs from modeled estimates.

## How it is generated

Long-format benchmark records (model, task, metric, score, cost, timestamp). Each record carries an explicit `isModeled` flag: measured runs are produced by a real runner against the pinned task; modeled records are estimates and are labelled as such in the data.

- **Source type:** `HARPD_BENCHMARK`
- **Coverage:** Published benchmark tasks; measured and modeled records are flagged separately.
- **Methodology:** https://harpd.com/methodology/benchmarks/

## Update frequency

Per benchmark run. Historical runs are never overwritten.

## Schema

The record shape is described by the `schemaVersion` envelope and field documentation carried by https://harpd.com/data/benchmarks.json.

- **JSON endpoint:** https://harpd.com/data/benchmarks.json
- **CSV mirror:** https://harpd.com/data/benchmarks.csv

## License

CC BY 4.0 — https://creativecommons.org/licenses/by/4.0/

## Citation

> Harpd. "Harpd Benchmark Dataset." harpd.com/benchmarks/ License: CC BY 4.0.

## Limitations

- MEASURED vs MODELED is strictly separated and must stay that way: if the underlying data is a modeled estimate it MUST be stated as an estimate and never presented as a measured result.
- The published benchmarks.json may contain modeled preview estimates (isModeled: true). A modeled score is an estimate, not a measurement, and must never be quoted as a measured result.
- Historical runs are never overwritten; a benchmark result reflects the model, task and pricing version it was scored with at its timestamp.

## Methodology

How these numbers are produced is documented at https://harpd.com/methodology/benchmarks/.

## Reproduction

1. Fetch the benchmark records: `curl https://harpd.com/data/benchmarks.json`
2. Every record carries a data timestamp and an explicit `isModeled` flag. Measured records (isModeled: false) can be re-run against the same task and model at the pinned version; modeled records (isModeled: true) are estimates and must be treated as such.
3. For the CSV mirror: `curl https://harpd.com/data/benchmarks.csv`

## Attribution

Source: Harpd
Data licensed under CC BY 4.0: https://creativecommons.org/licenses/by/4.0/
Canonical page: https://harpd.com/benchmarks/