The Presupposition Problem in AI: Why Hidden Assumptions at "Representation Genesis" Decide What Your Model Can Ever See

Every AI system makes a decision long before it answers a single question. That decision is about what the world looks like in the first place, which pieces of reality get turned into numbers, which get merged together, and which quietly disappear. In AI research circles, this moment has a name: representation genesis. It is the point where raw, messy reality becomes an internal representation a model can actually compute on.

And that moment is where a deeper issue hides. Call it the presupposition problem. Before a model ever learns anything, it has already been handed a stack of unspoken assumptions, baked in by data collection, by labelling rules, by the choice of what counts as a "category" at all. Those assumptions shape everything downstream. They decide what the model can notice, what it cannot notice, and what it will confidently get wrong.

This matters far beyond academic debate. As AI moves from demo to deployment, into hiring, lending, medicine, logistics, customer service, and autonomous decision-making, the presuppositions locked in at representation genesis become the invisible architecture of real-world outcomes. This article breaks down what the problem is, why it is so hard to fix, and what forward-looking organisations should do about it now.

What "Representation Genesis" Actually Means

AI models do not see the world. They see a compressed, structured version of it. A photo becomes a grid of numbers. A customer becomes a row of features. A hospital visit becomes a coded event. That translation step, from the world to the model's internal language, is representation genesis.

The critical point is this: the translation is never neutral. It always involves choices. Which variables are captured? How are they bucketed? What gets treated as a single category, and what gets split apart? Who decided that this was the "right" way to describe a person, a transaction, a disease, or a job applicant?

Once those choices are locked in, the model trains on them, optimises around them, and eventually presents the results as objective output. The assumption has become infrastructure. It is no longer visible as a choice, it just looks like the data.

Why Presuppositions Sneak In So Easily

The presupposition problem is not caused by bad engineers or careless data scientists. It is structural. It shows up in at least three places, and each one is easy to overlook.

1. The world is described before it is measured

Long before a dataset exists, someone had to decide what to record. A loan application form asks for certain fields and not others. A support ticket system logs certain categories of complaint and not others. A clinical trial measures certain outcomes and not others. The schema is a worldview. The model inherits that worldview wholesale.

If a system is built to predict "employee success" and the historical record defines success as "stayed three years and got promoted," then the model is not learning about talent. It is learning about a specific, narrow, historically contingent definition, and treating it as universal truth.

2. Categories flatten things that do not fit

Machine learning loves clean labels. Reality does not provide them. So we force messy things into boxes: gender into two options, job titles into taxonomies, symptoms into codes, sentiment into positive or negative. Each of those boxes presupposes that the distinction inside it does not matter.

Sometimes it does. A model that treats "part-time contractor" and "full-time employee" as the same category has made a presupposition that could distort every prediction it makes about pay, retention, or performance.

3. The objective function encodes a value judgement

Every model is trained to optimise something: click-through, conversion, accuracy, engagement, risk reduction. Choosing that target is not a technical decision. It is a statement about what matters. A recommendation system optimised purely for engagement presupposes that time spent is a good proxy for user value. That assumption may be true sometimes. It is not true by default.

These three layers compound. By the time a model produces an output, the presuppositions have been stacked so deep that unpicking them after the fact is nearly impossible.

Why Today's AI Safety and Fairness Tooling Misses This

Most current AI governance work focuses on the outputs: Is the model biased? Does it produce harmful text? Are error rates equal across groups? These are important questions. But they are downstream questions. They measure the symptom, not the source.

A fairness audit can tell you that a model's error rate differs across two groups. It cannot easily tell you that the reason is a category definition chosen years earlier by someone who never imagined the model would exist.

This creates a dangerous illusion of control. Teams can pass an audit and still ship a system built on presuppositions that make it fundamentally unfit for the context it is deployed in. The checklist is green. The model is still wrong in a way no checklist was designed to catch.

Worse, the presupposition problem is self-concealing. A model trained on assumptions produces outputs consistent with those assumptions. Those outputs then get used to validate the model. The loop closes. Everyone points to the numbers and says the system works.

What This Means for Businesses Right Now

For organisations deploying AI, the practical implication is blunt: the biggest risks are not in the model, they are in the framing. A few concrete patterns are worth watching for.

• Hiring and talent tools. If "good hire" is defined by who was hired before, the model is learning a historical pattern, not a predictive truth. It will reproduce the assumptions of whoever designed the original process.
• Credit and risk scoring. Categories like "stable income" or "creditworthy" carry embedded assumptions about employment type, geography, and household structure. Those assumptions can quietly exclude entire populations from fair consideration.
• Healthcare and triage. Symptom coding, severity scales, and "normal" ranges are all constructs. When applied across populations that were not part of the original definition, the presuppositions become clinical risk.
• Customer analytics and personalisation. Segment definitions determine what the model can ever recommend. A presupposition that "lapsed customer" means "no purchase in 90 days" will blind the system to every other form of disengagement.
• Agentic systems. As AI moves from answering questions to taking actions, presuppositions get executed. A wrong assumption in a chatbot is embarrassing. A wrong assumption in an agent that books, buys, or files is a liability.

The common thread is that these failures rarely look like failures at first. They look like working software.

A Practical Playbook for Teams

None of this is unsolvable. But it requires shifting effort upstream, into the design and data-definition phase, where most AI governance currently spends the least time.

Document the assumptions, not just the data

Data documentation is now standard practice in many teams. Assumption documentation is not. For every model, write down explicitly: what was defined, by whom, using what rule, and what does that definition exclude? Treat this as a living document, not a one-time compliance artefact.

Ask "who is not in this data?"

Coverage questions usually focus on who is included. The more revealing question is who is structurally absent, and whether that absence is a random gap or a direct consequence of how the category was defined.

Stress-test the framing, not just the model

Before evaluating performance, evaluate the problem statement. If the target variable were defined differently, would the model still be useful? If the answer is no, the system is fragile in a way accuracy scores will never reveal.

Keep humans attached to the definition layer

Automation creeps upward over time. Models start as advisors and end up as decision-makers. Build explicit review points at the definition layer, the place where categories and objectives are set, so that presuppositions get re-examined as context changes, rather than hardening into permanent infrastructure.

Design for revisability

Representations should be treated as provisional. Systems built so that categories and objectives can be redefined without rebuilding from scratch will be far more resilient than systems that treat their first framing as eternal.

The Road Ahead

The AI industry has spent the last several years getting very good at scale: bigger models, more data, faster training. The next phase will be judged on something less glamorous, whether systems can be built on representations that are honest about their own assumptions.

This is likely to become a competitive differentiator rather than a compliance burden. Organisations that can trace and defend the assumptions inside their models will be able to deploy AI in higher-stakes settings, and will recover faster when something goes wrong. Those that cannot will find themselves stuck at the pilot stage, or worse, shipping systems that fail in ways nobody can explain.

The presupposition problem is not a bug to be patched. It is a permanent feature of building systems that translate reality into computation. The goal is not to eliminate assumptions, that is impossible, but to make them visible, deliberate, and contestable. Every AI system begins with a decision about what the world is. The organisations that take that decision seriously will be the ones that build AI worth trusting.

Conclusion

Representation genesis is the quiet moment where AI systems decide what reality looks like. The presuppositions baked in at that moment shape every prediction, recommendation, and decision that follows, and they are almost invisible to the audits, dashboards, and fairness tools that dominate today's AI governance. Fixing this requires moving attention upstream, into the definitions, categories, and objectives that precede any model training. It is less exciting than scaling a model. It is more important than almost anything else on the roadmap.