Garbage Format In, Garbage Answers Out
You fine-tune a chat model, run inference, and get broken text sprinkled with stray tokens or run-on turns that never end. The instinct is to blame the data — more examples, cleaner examples, a different learning rate. Often the real problem is upstream of all of that: the training examples were never formatted the way the base model expects to see a conversation.
What a chat template actually is
Every chat-tuned base model was itself trained on conversations wrapped in a specific format: special tokens that mark where a message begins, who is speaking (system, user, or assistant), and where the turn ends. That format — the chat template — isn't a convention you can improvise. It's baked into how the model learned to read structure in the first place, during its own supervised fine-tuning.
If you fine-tune further without reproducing that exact structure, the model has no reliable signal for where one turn stops and the next starts. It doesn't fail loudly — it just learns a slightly wrong version of "when to stop talking" or "whose turn it is," which shows up later as run-on generations, garbled turn boundaries, or the model echoing role markers back at you as literal text.
How to apply it correctly
The fix isn't to hand-write the template yourself. Every base model ships with a way to retrieve its own chat template, usually as a single method exposed by the tokenizer that takes a list of role-tagged messages and returns the correctly formatted string — special tokens, role markers, and all. The workflow is:
- Load the tokenizer for your exact base model — not a similar one, the exact checkpoint you're fine-tuning.
- Structure each training example as a list of role/content pairs (system, user, assistant) rather than a single blob of text.
- Call the tokenizer's chat-template function on each example to produce the final formatted string, with the model's own special tokens in the right places.
- Feed the templated output into training — not your raw text.
This applies whether you're starting from a raw two-column dataset (prompt/response) or converting from another format: the templating step comes right before training, applied uniformly to every example.
tokenizer.apply_chat_template(example)Pitfalls
- Assuming templates transfer across models. Switching base models mid-project means re-applying the new model's template from scratch — templates are model-specific, not framework-specific, and two models from different families almost never share one.
- Mixing data from multiple sources without re-templating. If you're combining datasets that were pre-formatted for a different model, don't assume the formats match. Re-apply the target model's template to everything before training.
- Confusing base and instruct variants. Instruct-tuned models already have an established template from their own fine-tuning — use it. Base models generally don't have a template baked in, so you're defining the structure yourself; be deliberate about that choice.
- Never actually looking at a formatted example. It's easy to run templating as an invisible preprocessing step and never verify it did what you expected.
The one habit that catches this early
Before you hit train, print one fully formatted example — after templating — and read it. You're checking that the special tokens are where you expect, that roles are labeled correctly, and that nothing got truncated or duplicated. This one check catches the vast majority of chat-template bugs before they cost you a training run.
Resources
Fine-tuning a model for production? Yeda AI designs, audits, and ships production LLM systems.