How Chatbots Get Their Manners: Fine-Tuning in Three Steps

Smart isn't the same as helpful. A freshly pretrained language model has read a good chunk of the internet, but ask it a question and it may complete your sentence, answer with another question, or ramble — because all it was trained to do is predict the next word. The assistant behavior you actually experience is bolted on afterward, in a fine-tuning process with three distinct steps.

Step 1 — Perfect examples

First comes supervised fine-tuning. Humans write out thousands of ideal exchanges: a realistic user request paired with the answer a great assistant should give — right format, right tone, right level of detail. The model is trained on these demonstrations the same way it was pretrained, by predicting the next word — except now the "next words" are exemplary answers. It learns the shape of being an assistant: answer the question, structure the response, stop when done.

Step 2 — Teaching taste

Demonstrations can't cover everything, and for most prompts there's no single perfect answer — just better and worse ones. So the next step teaches the model taste. The model generates several candidate answers to the same prompt, and human reviewers rank them from best to worst. Those rankings train a separate reward model: a network whose only job is to look at an answer and score how much a human would like it. In effect, human judgment gets compressed into an automated critic.

Step 3 — The automated coach

Now the reward model becomes a coach that never sleeps. The main model generates answers, the reward model scores them, and the main model's weights are adjusted to make higher-scoring answers more likely — millions of practice rounds, no human in the loop for each one. This technique is commonly called RLHF: reinforcement learning from human feedback. The "human feedback" was collected in step 2; step 3 scales it. The result is the polite, on-topic, appropriately-cautious behavior you recognize as a chatbot's manners.

Steal the technique in your prompts

You can't fine-tune a frontier model from a chat window, but the same two moves work in-context:

• Show one great example (step 1, miniaturized). Instead of describing the output you want, paste a single perfect specimen — the exact format, tone, and depth. The model imitates demonstrations far more reliably than it follows descriptions.
• Rank and explain (step 2, miniaturized). When the model gives you options, don't just pick one — say why: "Option B, because it's concrete and skips the preamble." Stated preferences act as an in-context reward signal for the rest of the session.

Fine-tuning teams spend millions doing exactly this at scale. Doing it manually, per conversation, is free.

Resources

• Wikipedia — Reinforcement learning from human feedback (RLHF)
• Anthropic — prompt engineering overview