Your AI Is What It Eats: Why Data Quality Beats Size
Here's a number that should recalibrate how you think about training data: according to Meta's AI-foundations explainer, a frontier model's pipeline can start from roughly 3,000 trillion data points of collected text and filter it down to about 20 trillion tokens of training data. That's keeping roughly 1% and discarding the rest. The single most expensive step in building a model isn't collecting data — it's throwing most of it away.
The brutal filter
Why delete 99% of what you gathered? Because training data has to satisfy three principles at once, and raw internet text fails most of them:
- Quantity — models need enormous volumes of text to learn the statistics of language. Necessary, but the cheapest of the three to satisfy.
- Quality — duplicated pages, spam, broken markup, machine-generated sludge, and factually wrong content all get filtered out. Every low-quality document that survives teaches the model a bad pattern.
- Diversity — the surviving data must span topics, languages, formats, and viewpoints. A trillion tokens of near-identical forum posts is worth far less than a smaller, broader mix.
Quantity gets the headlines; quality and diversity do the work. The filter is where the model's ceiling gets set.
Garbage in, garbage out
Think of the model as a skyscraper and the training data as the ground it's built on. You can add floors — more parameters, more compute, longer training runs — but if the foundation is quicksand, every extra floor makes the collapse worse, not better. A bigger model trained on noisy data doesn't average the noise away; it learns the noise more faithfully. Errors, biases, and junk patterns in the data become errors, biases, and junk patterns in the outputs. No amount of scale fixes a bad foundation.
The same rule applies to your prompts
You don't control what a model was trained on, but you control the second dataset it learns from: your context window. Everything you paste in — examples, documents, chat history — is data the model conditions on, and the same quantity/quality/diversity math applies:
- Three great, correctly-labeled examples beat fifteen messy, inconsistent ones. Every flawed example teaches a flawed pattern — in-context, instantly.
- One clean, relevant document beats a raw dump of an entire wiki. Irrelevant context doesn't just waste tokens; it actively dilutes the signal the model should attend to.
- Contradictory context is quicksand: if two pasted documents disagree, the model may confidently build on either one.
Power tricks
- Curate your examples. Before few-shot prompting, review each example as if it were training data — because to the model, it is. Fix labels, remove edge cases you don't want copied.
- Prune irrelevant context. Cut anything the task doesn't need. A shorter, cleaner prompt usually outperforms a longer, noisier one.
- Prefer primary sources. Paste the actual spec, doc page, or error log — not a colleague's summary of it. Quality in, quality out.
Resources
Building an AI feature? Yeda AI designs, audits, and ships production LLM systems.