Reduce LLM costs without breaking quality

Token cost vs outcome cost

Token cost is the unit providers bill on, but it is not the unit you should optimize. The outcome you care about — a correct answer, a completed task, a usable draft — has its own price once you include failures and retries.

When you optimize token cost alone, you can pick a model that fails often and quietly raises outcome cost. Cost per successful task keeps the target honest.

Model replacement (the biggest lever)

Most production traffic was originally built against a flagship model and never revisited. High-volume, well-scoped calls — classification, extraction, summarization, routing — rarely need the most capable model available.

• Audit by task, not by app. One application usually contains several task types with very different difficulty.
• Benchmark candidates on your data. Use Model Compare to score cheaper models against your prompts and a held-out set.
• Swap where quality holds. Replace the model only where the cheaper candidate meets your acceptance bar, and keep the premium model for the hard tail.

Harpd ModelSwitch runs this search for you: it tests cheaper models on your real tasks and reports which ones pass, so the downgrade is backed by evidence.

Prompt caching

If your prompt contains a stable prefix — a system prompt, a long set of instructions, or retrieved context that does not change per request — cache it. The first request pays full price to write the cache; subsequent requests read from it at a much lower rate.

• Cache the static parts. Put instructions and repeated context first so they form a cacheable prefix.
• Mind the TTL. Caches expire; design so the hot prefix is reused within the window.
• Best for RAG and agents. Retrieved documents and tool schemas are ideal cacheable prefixes.

Batch processing

When a task is not user-facing and can tolerate delay, batching often lowers effective cost. Instead of sending requests one at a time with synchronous latency expectations, submit them as a batch that the provider processes offline.

Prompt & output reduction

Smaller inputs and outputs cost less, but the goal is to remove waste, not to starve the model of what it needs.

• Trim the system prompt. Delete instructions that are never exercised by the task.
• Limit retrieved context. Retrieve only what the call needs; irrelevant chunks inflate input tokens on every request.
• Stop asking for discarded output. If you parse out only the JSON, do not also ask for prose explanations.
• Set a sane max_tokens. A reasonable ceiling prevents a single runaway generation from dominating cost.

Measure success rate before and after any reduction — if quality drops, you removed too much.

Retries and fallbacks

Retries are necessary but expensive. An unbounded retry loop multiplies cost on the exact inputs that already failed once.

• Cap retries. A small fixed limit (often one or two) is usually enough.
• Fail over, do not repeat. On repeated failure, switch to a different model or a safe fallback rather than re-calling the same one.
• Count retry spend. Retries are part of cost per successful task — they should show up in your numbers, not hide in a log.

Cost per successful task

Tie it all together: the metric that matters is total spend divided by tasks that succeeded. It automatically accounts for model price, cache hits, batch discounts, and the cost of failures and retries. A change that lowers token price but raises failure rate will show up as worse here — which is exactly the signal you want.

Measure it with Cost per Successful Task and validate model swaps against it rather than against token price alone.