Español

The Cheaper Model That Costs More

You're an hour into a session. The next question is trivial, so you switch to the smaller, cheaper, faster model to answer it. It's the obviously sensible move, and depending on how much conversation sits behind you, it can cost more than not bothering.

Caching is what makes long sessions affordable

Every request in a conversation carries the whole history. Without caching that would be ruinous, so the entire history gets cached and re-read at the cached rate rather than reprocessed from scratch. This is why a long session doesn't cost proportionally more per message as it grows.

The critical property is that caching is a prefix match: any byte change anywhere in the prefix invalidates everything after it. And the cache belongs to a specific model. It is not a shared store that different models read from — each has its own.

What switching actually does

Change models mid-conversation and the new model has no cache for this conversation. Everything before that point is uncached input to it. So it reads your entire history at full uncached price — to answer the small question you switched for.

Then you switch back, and the first model has to rebuild too. You've now paid full uncached rates for the same context twice, once in each direction, plus the ongoing cost of whatever you actually asked.

The arithmetic flips as the conversation grows. Early on, when there are a few thousand tokens of history, switching is cheap and the smaller model's lower rate wins easily. At a hundred thousand tokens of context, the one-time cost of rebuilding cache twice can dwarf the saving on a single short answer. The instinct that says "cheaper model, cheaper answer" only prices the answer, not the re-read.

Note: The same mechanism applies to other mid-conversation switches, not just model choice. Changing the speed setting invalidates the prompt cache too, so flipping a faster mode on midway through a long session has the same shape of cost: you re-pay for the existing context at the new configuration's rates.

Choose the model at task boundaries

The rule that falls out of this is simple: model choice is a per-task decision, not a per-question one.

Decide when you start a piece of work, then stay there for its duration. If the next thing genuinely is a different job that suits a cheaper model, that's a good reason to start a fresh session on that model — you weren't going to reuse the context anyway, and clearing between unrelated tasks is the single highest-value context habit regardless.

A useful signal: if you feel the urge to switch models mid-task, it's often evidence the task should have been two sessions. The urge is usually triggered by a question that isn't really part of the current work.

And if the side question genuinely is small, there are cheaper ways to ask it than a model switch. A side-question mechanism that doesn't add to the conversation avoids the whole problem, since you never invalidate anything.

The other cache trap in the same family

Once you know the cache is what's carrying you, a second pattern makes sense: your first message after a long break can be unexpectedly expensive, because the cache has a lifetime and an idle session falls out of it. The next request reprocesses the full context.

The lifetime is an hour on a subscription and drops to five minutes once you're drawing on usage credits; on an API key or cloud provider it's five minutes by default. So the cost of stepping away for lunch mid-session isn't zero, and on a very large session it's worth being aware of when you come back.

This is also why usage can climb in a long session that feels idle: anything that starts a new turn — a scheduled task firing, a message arriving from another session, a goal check-in — sends the full context again.

The takeaway

The cheap model is only cheap at the start of the conversation. The prompt cache belongs to one model, so switching mid-task means paying full price to re-read everything you've already said, and paying again on the way back. Pick the model when you pick the task; if the next question doesn't belong to that task, give it its own session.