Token price is not the cost
A model that is cheap per token but fails often costs more once you pay for retries and throw away failed runs. The number that matters is cost per task that actually succeeded.
Cost per successful task = total workload cost ÷ successful tasks. It folds in retries, failed runs, and wasted tokens. A cheap-per-token model that fails 30% of the time is usually more expensive than a pricier model that passes the first time.
- Accounts for retries and failed runs
- Surfaces false economies of flaky models
- Comparable across models on the same task
The definition
Why cheap-per-token is misleading
Token price only describes one attempt. In production, a task may need several attempts before it succeeds — or it may fail entirely and be discarded. The cheap model's low per-token rate is multiplied by more attempts, while a more capable (and more expensive) model may finish in one or two tries.
Consider a task where a cheap model passes 70% of the time and needs an average of 1.4 attempts, versus a pricier model that passes 98% of the time in 1.02 attempts. Divide the higher per-token price by the pass rate and the "expensive" model is often cheaper per real outcome.
What goes into the denominator
- Retries. Automatic re-attempts when the first output fails a validation or quality gate.
- Failures. Tasks that never pass and are dropped — their tokens are spent and gone.
- Wasted output. Partial or malformed completions that are thrown away before a correct one is produced.
All of these are included in totalWorkloadCost. Dividing by successfulTasks — never by attempts — is what makes the metric honest about what you actually pay for.
Honesty note
The per-task failure rates above are illustrative, not measured. Harpd's benchmark that would produce real measured rates is in a PLANNED / MODELED stage (target ship 2026-09-15). Until then, use the calculator with your own observed pass rates, or plug in modeled rates clearly labeled as such.