Español

Caching Is Where Open Models Get Cheap

Open models advertise a per-token price that looks like a steal next to the frontier. Then the monthly bill lands, and it's higher than the sticker math promised. The gap almost always traces back to one setting people leave off: input caching. It's not a footnote — for a coding agent, caching is the difference between a bill that makes you flinch and one that's closer to lunch money.

Why the sticker price lies to coding agents

A coding agent doesn't ask one question and stop. It runs a loop: read the repo, make a plan, edit files, run the tests, read the failures, fix what broke, try again. On every turn through that loop it re-sends a big block of context — the same repository files, the same system prompt, the same docs — because the model has no memory between calls. So the tokens you pay the most for aren't clever new instructions. They're the same unchanging prefix, billed over and over, dozens of times per task.

That's why "cheap per token" can still produce an expensive month. The per-token rate is real, but you're paying it against a token count inflated by constant re-reads. A heavy day of agentic coding can burn roughly 20 million tokens. At full input-and-output rates, that day's bill is the one that makes you flinch.

What input caching actually does

Providers cache the stable prefix of your prompt. When a later request starts with the same content, that's a "cache hit," and those reused tokens are billed at a fraction of the normal input rate instead of the full price. Per OpenRouter's prompt-caching docs, cached reads are typically charged at roughly 0.1–0.5× regular input, depending on the provider — some enable it automatically, others need you to mark the stable blocks explicitly.

The magnitudes on open models are the whole point. On one open model cited at around $3 per million input tokens, cached input drops to about $0.30 per million — roughly a tenth of the sticker input rate. Another open model quoted cached input around $0.26 per million. Put the numbers side by side:

Now replay that heavy coding day. The reused context — the repo, the system prompt, the docs the agent keeps re-reading — is exactly the part caching discounts by about 10×. The same 20-million-token day, with caching working on the input, comes out closer to lunch money. Caching is where the open-model cost advantage actually shows up; without it, you're paying frontier-shaped bills for a discount-shaped model.

How to actually get cache hits

Caching only helps if your prompt's prefix is genuinely stable, so structure the prompt around that:

• Put the big, unchanging context first. Repo files, the system prompt, and reference docs go at the front of the prompt so the stable prefix is byte-for-byte identical across runs. Anything that changes each turn — the latest instruction, the newest tool output — goes at the end.
• Don't let small edits break the prefix. A timestamp, a shuffled file order, or a reworded system line near the top invalidates the cache for everything after it. Keep the head of the prompt frozen.
• Check the provider's separate cached-input price. It's a distinct, much-lower line item from the sticker input rate. If a provider doesn't publish one, you don't get the discount — and the open model's headline price is only telling you half the story.
• Confirm you're getting hits. Many providers report cached vs. fresh token counts per request. If your cached-token count stays near zero across a long agent run, your prefix isn't stable — fix the ordering before you trust the savings.

Rules of thumb

• Per-token price tells you the ceiling, not the bill. For agent loops, the cached input rate is the number that decides your month.
• Cached input often runs about a tenth of sticker input — treat that gap as the real reason to reuse context deliberately.
• Stable stuff at the top, volatile stuff at the bottom. That single ordering habit is most of what turns caching on.
• Before you compare two open models on price, compare their cached-input rates, not just their sticker rates.

Takeaway

The open-model cost story isn't the sticker price — it's caching. Coding agents re-read the same context on every step, so the same input tokens go out again and again. Keep that context at the front of the prompt, check your provider's cached-input rate, and confirm you're getting hits. Do that, and the model that looked cheap on paper is finally cheap in practice.

Resources

• Cost Per Task, Not Per Token — Yeda AI Tips #203
• Point Your Coding Agent at an Open Model — Yeda AI Tips #206
• OpenRouter — prompt caching