There is a moment in every machine learning project where you stop looking at the model and start staring at the data, and that moment is almost always uncomfortable. The model is not behaving the way you expected. You have tuned the hyperparameters (the knobs that control how the algorithm learns, things like learning rate and regularization strength), you have adjusted the architecture, and still the outputs feel wrong in a way you can’t immediately name. Then you pull up a random sample of your training data and you realize the problem was never the algorithm. It was always the humans who assembled the dataset.

This is not a niche engineering problem. It is one of the most revealing things that has happened in computer science in a long time. If you want to understand why AI systems sometimes behave in ways their creators didn’t anticipate, the answer is almost never in the math. It is in the assumptions, shortcuts, and blind spots of the people who decided what the training data should look like.

Labels Are Opinions Wearing Lab Coats

When you build a supervised learning model, you train it on labeled data. A label is simply a tag attached to an example, telling the model what category it belongs to or what output it should produce. A photo of a dog gets labeled “dog.” A sentence gets labeled “positive sentiment” or “negative sentiment.” Seems straightforward.

Except labeling is almost never straightforward. Consider a content moderation dataset. You hire a team of annotators (people who apply labels to raw data) and ask them to classify text as “harmful” or “not harmful.” Now you have a question hiding inside every single row: harmful according to whose standards? One annotator’s “aggressive debate” is another’s “harassment.” One person’s “dark humor” is another’s “targeted abuse.” The model learns from the aggregate of these disagreements and converges on a pattern that looks like a consistent rule but is actually a statistical average of human cultural assumptions.

Researchers at major AI labs have repeatedly found that inter-annotator agreement (the rate at which independent labelers agree on the same classification) is alarmingly low on nuanced tasks. Sometimes it hovers around 60 to 70 percent, which means nearly a third of your training labels are contested even before the model sees them. You are not training the algorithm on ground truth. You are training it on a weighted poll.

The Corpus Is a Time Capsule

Large language models (the kind that power modern AI assistants) are trained on massive corpora, which is the technical term for a large collection of text gathered from books, websites, forums, and other sources. Here is what that means in practice: those corpora have a publication date problem.

Text written in 2008 reflects the language, values, and assumptions of 2008. Text scraped from a particular corner of the internet reflects whoever was writing in that corner. If a corpus over-represents technical documentation, academic papers, and English-language forums, the resulting model will be fluent in those registers and somewhat lost everywhere else. The model does not know it is missing context. It just has a very confident, very skewed picture of the world.

This is similar to the problem engineers face when writing code intended for a future reader, where the assumptions embedded in your choices are invisible to you but obvious to someone looking at the code five years later. We have written before about how deliberate opacity in code can encode institutional knowledge in ways that are rational to the author and baffling to everyone else. Training data does the same thing, except the “code” is a multi-terabyte snapshot of human expression, and the assumptions are harder to audit.

Filtering Is Where Things Get Philosophical

Before a corpus is fed into a model, it goes through filtering. Engineers write rules to exclude low-quality content, remove personally identifiable information, deduplicate repeated text, and sometimes filter for “safe” content. Every one of those decisions is a value judgment.

What counts as low quality? Usually, it means short texts, texts with many spelling errors, or texts that fail a language-identification classifier. That classifier itself was trained on labeled data, created by humans. So you are using a human-informed model to filter the data that will train your next model. The assumptions stack.

Deduplication is another interesting case. If a piece of text appears ten thousand times in your raw corpus, many pipelines will reduce it to one or a small handful of instances. This prevents the model from over-fitting to repeated content. But it also means that widely-circulated, popularly-shared ideas get downweighted relative to ideas that appear once. Consensus, in the statistical sense, gets penalized. The rare and the common collapse toward each other in ways the model cannot explain and that the original data did not intend.

These design choices accumulate. The model you end up with is less “what the data says” and more “what the data says after a series of value-laden preprocessing decisions made by a team of engineers under deadline pressure.” And if you have spent time in this industry, you know how documentation for those decisions tends to age. Six months after the pipeline is built, the reasons for half the filtering rules live only in the memory of whoever wrote them.

What the Model Learns Is What We Already Believed

Here is the uncomfortable synthesis: a trained model is not a discovery. It is a reflection. When a language model reproduces gender stereotypes in job descriptions, it is not inventing bias. It is accurately learning from job descriptions written by humans who held those biases. When a medical AI performs worse on certain demographic groups, it is usually because the training data contained fewer examples from those groups, because those groups had historically less access to the healthcare settings that generated the data.

The model is not the problem and it is not the solution. It is an artifact. It crystallizes a particular moment in human knowledge and human prejudice into a form that can be queried at scale. That is simultaneously its power and its danger.

This is why the most serious researchers in the field spend more time on data curation than on architecture design. The architecture of a transformer model (the technical structure underlying most modern AI) is relatively well understood. What is not well understood is how to construct a dataset that accurately represents the world you want the model to operate in, rather than the world you happened to be able to scrape.

What This Means If You Are Building With AI

If you are a developer integrating AI into a product, the practical implication is that your model’s behavior is downstream of decisions made by people you have likely never met, about data you have probably never seen, according to values that were never written down in your API documentation. That is a significant unknown to carry into a production system.

The responsible engineering practice here is not to assume the model is neutral and work backward when something goes wrong. It is to treat the model’s assumptions as a variable you need to actively probe before deployment. Write adversarial test cases. Sample outputs across demographic and linguistic dimensions. Build feedback loops that surface distributional failures (cases where the model behaves differently for structurally similar inputs) before your users find them.

Because ultimately, training data is one of the most honest things the tech industry produces. It is not a press release or a benchmark result. It is a record of what humans actually wrote, actually labeled, and actually decided was worth keeping. And that record, more than any algorithm, tells you who we are.