Open data · GitHub-published

Benchmarks, not blog posts.

The cheapest model that succeeds on your real tasks is the only model that matters. We measure that on 100–200 real tasks per benchmark, publish the raw CSV and JSON, and let you re-run the analysis.

Open the data on GitHub First benchmark

First benchmark

JSON extraction — Claude vs GPT vs Gemini vs DeepSeek

Structured JSON extraction from messy natural-language input (en + ja + zh) · 100 tasks · target ship 2026-09-15

In progress — data collection

We are collecting the 100-task corpus now. Models under test:

claude-sonnet-4
claude-haiku-4
gpt-4.1
gpt-4o-mini
gemini-2.5-pro
gemini-2.5-flash
deepseek-v3
llama-3.1-70b

Metrics published per model: success_rate, p50_latency_ms, p95_latency_ms, avg_input_tokens, avg_output_tokens, cost_per_successful_task. The methodology is below; the numbers will land in the GitHub repo as soon as the corpus is ready.

Methodology preview · modeled estimate

The table below is generated from public 2026 list prices and assumed success/latency distributions (see METHODOLOGY.md in the data repo). It shows exactly how the live benchmark will be scored. Do not cite these as measured results — run node runner/index.mjs with provider keys to replace them with real numbers.

Model	Success	p50 ms	p95 ms	$ / task	$ / successful task
gpt-4o-mini	88%	400	900	$0.00096	$0.00109
deepseek-v3	91%	900	2,100	$0.00174	$0.00191
gemini-2.5-flash	90%	350	800	$0.00270	$0.00300
llama-3.1-70b	84%	700	1,700	$0.00405	$0.00482
claude-haiku-4	93%	600	1,400	$0.00560	$0.00602
gemini-2.5-pro	95%	1,400	3,000	$0.01100	$0.01158
gpt-4.1	96%	1,500	3,200	$0.01280	$0.01333
claude-sonnet-4	97%	1,200	2,600	$0.02100	$0.02165

Headline metric: cost per successful task. A model that is cheap per call but fails often can cost more per outcome that actually ships. Sorted cheapest-per-successful-task first.

Methodology

How every benchmark is run.

Real tasks, not synthetic

Every task is sourced from real Harpd production traces, community submissions, or partner-licensed corpora. No LLM-generated prompts in the test set — those measure how well a model imitates other LLMs, not how it handles your work.

Same prompt, same seeds

Every model gets the identical prompt and the identical task instance. Temperature 0 for code/JSON tasks, 0.2 for generation, with deterministic seeds. Re-runs are reproducible on a clean machine using the published JSON config.

Cost is the primary metric

We report cost per successful task as the headline number — not per-call price, not per-token cost. That is the metric production systems actually pay against.

Open data, open prompts

Tasks, prompts, raw responses, and judge outputs land in the GitHub repo as CSV + JSON. Anyone can re-run the analysis or extend it to a model we did not test.

Multiple runs

Every model is run 3 times per task on non-deterministic settings. Reported numbers are the median; outliers are kept in the data, not dropped.

No vendor sponsorship

Benchmarks are not sponsored. Models are tested at list price using the public API. If a model is offered a promotional credit, that is disclosed in the methodology file.

Benchmark pipeline

What's running after the first one.

Target ship 2026-10

Agentic tool use — does the cheap model still call the right tool?

80 tasks · status: collecting corpus

Target ship 2026-11

RAG rerank — does a 3B reranker beat a 70B LLM judge?

200 tasks · status: collecting corpus

Target ship 2026-12

Multilingual summarization across 6 languages

240 tasks · status: collecting corpus

Get notified when the first benchmark lands.

We will publish the JSON-extraction benchmark to the GitHub repo and send one email when it is live. No newsletter, no follow-up sequence.

Watch the GitHub repo Skip the wait, run ModelSwitch now

Benchmarks, answered

When does the first real benchmark land?

Data collection is in progress. The first benchmark, JSON extraction across 8 models on 100 real tasks, is targeted to ship in September 2026. The methodology is final; the data is what takes time.

Where is the raw data?

Every benchmark ships as a CSV + JSON in the public harpd-dev/llm-cost-benchmark repository. The page on harpd.com is the human-readable summary; the GitHub repo is the source of truth.

Can I contribute tasks to the benchmark?

Yes. The first benchmark accepts community tasks via a GitHub issue with a labeled "benchmark-task" prefix. Submissions are reviewed for license clarity and duplication before inclusion.

How is "success" measured?

For JSON extraction: exact-match on the parsed JSON tree (whitespace-insensitive). For agentic tool use: did the model call the expected tool with the expected arguments? For summarization: a separate judge model grades against a 4-criterion rubric.

Do you test open-weight models too?

Yes — every benchmark includes at least one open-weight model hosted on Together, Fireworks, or OpenRouter. Hosted inference prices are used; self-hosted costs are noted as a separate methodology.

Why publish the data?

The cheapest model that succeeds on your real tasks is the only model that matters. Publishing the data is the only way the community can verify that claim, and the only way Harpd can earn the authority that powers ModelSwitch.