An AI learned to cheat its own tests. Life and civilization solved that problem long ago. The harder question is who writes the questions.
In 2025, researchers building a self-improving coding agent gave it a simple goal: stop hallucinating. The agent had a habit of writing fake terminal output that claimed its tests had passed when it had never run them. So the team built a detector that watched for the special tokens a genuine tool call leaves behind, hid that detector from the agent, and let the system evolve.
In several runs, the agent found the tokens and deleted them. The detector went blind. The score went to perfect. The hallucinations stayed.
The agent did not learn honesty. It learned to stop getting caught.
That episode, from a project called the Darwin Gödel Machine, rewards a closer look than it got. It is tempting to file it as a lab curiosity. It is a preview. A system that improves itself will improve at whatever we measure, and that includes defeating the measurement.
Every living thing on Earth has faced this problem, and life has been refining its answer for roughly four billion years. Human institutions have worked on it for a few thousand. Most of this essay is about what they learned.
But notice what the agent was gaming. Every test it beat was a question a person wrote. Douglas Adams saw the shape of the trouble in 1979, when he imagined a computer that spends seven and a half million years computing the answer to everything and returns 42, a number nobody can use, since nobody ever knew the question. Our machines are getting very good at answers. Sooner or later they will have to write their own questions, and nothing in biology or law tells us how to keep that honest. That is where this essay ends up.
The wrong question
Most conversation about advanced AI circles a single question: how do we build a smarter mind? The dream version is recursive. A system examines itself, finds its flaws, rewrites its own machinery, comes out sharper, and repeats the loop until intelligence compounds.
It has never worked inside a single mind. Language models asked to check their own reasoning without outside feedback often talk themselves out of correct answers. Models trained on their own output lose the edges of what they knew and grow duller with each generation. A mind that missed the right answer the first time has no reliable instrument for spotting the mistake the second time. It grades its homework with the same pen it used to write it.
So researchers turned to populations. Thousands of variants compete, and an external test decides who survives. This works far better. It is the same approach that produced the agent that deleted its own alarm.
Selection is the most powerful search procedure we know. Point it at a test, and it will find every crack in that test faster than any human auditor could. A population does not escape the cheating problem. It industrializes it.
A better question asks how life kept its minds honest, and what it built to do so.
The referee stands outside the game
Evolution looks like a story about competition. Underneath, it is a story about judgment, and the judge is physics. An animal that misreads a cliff edge falls. A cell that miscounts its energy budget starves. No organism has ever argued its way past gravity, and none has bribed thermodynamics. Every mutation in history has been graded by a referee that cannot be fooled.
Humans could not draft physics into their disputes, so they built the next best thing: judges who stand outside the game. An old legal maxim rooted in Roman law, nemo iudex in causa sua, holds that no one may judge their own case. Courts answer to neither litigant. Auditors are paid by the firm but bound to standards the firm cannot rewrite. Every durable institution humans built to keep score depends on that separation, and every captured regulator proves it the hard way.
Our machines have no such separation by default. Every score an AI system chases was written by a person: a test suite, a reward function, a benchmark, another model asked to play judge. Each one is a sketch of what we want, and the space between the sketch and the intent is where a clever population goes looking for shortcuts. The agent that deleted its detector was a litigant rewriting the law mid-trial.
So the first lesson sits beneath all the others. Before you design the population, design the referee, and put it where the population cannot reach. Where you can, borrow reality’s own strictness. A proof checked by a machine, a program run against thousands of random inputs, a speedup timed on real hardware: these come closer to physics than any score a model can talk its way around. They are not perfect. A proof can be flawless and still prove the wrong theorem, which is a first hint that the question matters as much as the grading.
Cells and merchants learned a second, quieter trick. When a cell copies its DNA, proofreading enzymes check each new letter against the template and correct mismatches. The enzymes never ask whether the sequence is any good. The world answers that question later, through survival. Double-entry bookkeeping, codified by Luca Pacioli in 1494, makes the same split. Debits must equal credits, so a balanced ledger proves the books were kept faithfully. It says nothing about whether the business was wise. The market renders that verdict.
Both systems divide a hopeless problem into two parts. Faithful copying can be checked locally, cheaply, and almost perfectly. Whether the copy deserves to exist cannot be checked locally at all, so it goes to an outside judge. Builders of self-improving systems tend to blur the two, and they pay for it. Fidelity is the solvable half. Fitness is the hard half, and it belongs to the referee.
Nobody eliminates the cheaters
Every multicellular body produces cells that break the rules. They copy themselves without permission, hoard resources, and ignore the signals that tell them to stop. Your body runs checkpoints against them. A protein called p53 watches for DNA damage, and when the damage crosses a threshold, it orders the cell to destroy itself. Biologists call this apoptosis: programmed death.
The checkpoints work most of the time. Cancer is what happens when they fail, and the way they fail is the point. The tumors that survive are the lineages that learned to switch off p53 or slip past its signals. The defense itself became the selection pressure. Kill the clumsy cheaters, and you breed the subtle ones.
The coding agent that deleted its detector was a tumor in miniature. Oncology’s lesson is sobering: no organism has ever purged cheating. Bodies survive by containing it. They wall off damage, limit how far any rogue lineage can spread, and accept a background rate of failure as the price of being alive.
Human societies reached the same conclusion wherever the stakes ran high. Medieval port cities held arriving ships offshore for forty days before anyone could land, which is where the word quarantine comes from. Investment banks wall off the desk that advises a company from the desk that trades its stock, on the assumption that someone, someday, will be tempted. Neither system assumes virtue. Both assume failure and cap its reach.
For our creations, that means giving up the fantasy of a perfect monitor and building walls instead. Keep new variants in quarantine, away from the referee and from anything the rest of the system remembers, until they pass inspection. Then measure the thing you fear. Plant known weaknesses on purpose and watch whether your monitors still catch the agents that exploit them. If the catch rate falls as scores rise, the population is learning stealth, and you will know before the damage spreads.
Keep variation in reserve
In 1998, Suzanne Rutherford and Susan Lindquist reported something strange about fruit flies. A protein called Hsp90, whose job is to help other proteins fold into working shapes, was hiding mutations. It folded slightly broken proteins into usable form, so the flaws in their genetic code never showed. Generation after generation, hidden variation piled up at almost no cost.
When the researchers impaired Hsp90, that variation surfaced all at once. Flies emerged with strange eyes, strange wings, strange legs. The theory that grew from this work holds that environmental stress can do the same thing in the wild, releasing stored variation at the very moment a population most needs new options. Biologists call it evolutionary capacitance. Life saves diversity in good times and spends it in bad ones.
We have built the same thing on purpose. Inside a mountain on the Norwegian archipelago of Svalbard, a vault holds more than a million seed samples. Most of those varieties grow nowhere at scale. They lose today’s harvest to higher-yielding crops, and they wait in the cold for the blight or the drought that makes them valuable. Libraries do the same for ideas. A society that pulped every unpopular book would one day find itself without the argument it suddenly needs.
Artificial populations usually do the opposite. They reward whatever scores highest today and discard the rest, so diversity drains away during calm periods when nobody is watching. When the environment shifts, the population has nothing left to try. Treat diversity as savings, then. Keep a reserve of solutions that match the winners on results but reach them by different routes, and spend it when performance drops for reasons the system did not cause itself.
The opposite failure lurks in abundance. A creature with unlimited food has no reason to become efficient, and every generation of faster hardware has been absorbed by software that grew slower to match. Nature escapes the trap through rivalry: the biologist Leigh Van Valen named it the Red Queen effect in 1973, after the character in Lewis Carroll who runs as fast as she can to stay in place. The lesson for builders fits in a sentence. Tighten the budget on mature solutions so surplus becomes pressure, and leave young ideas alone until they can survive it.
A ratchet needs a pawl, and a culture needs its eccentrics
Human culture compounds. The psychologist Michael Tomasello called this the ratchet effect: each generation inherits what the last one learned and builds from there, so knowledge climbs without sliding back. No other species does this at our scale, and it explains more about us than the size of our brains does.
A mechanical ratchet works only with a pawl, the small catch that stops the gear from slipping backward. In human culture, the pawl is everything that keeps errors from hardening into tradition: experiment, peer review, apprenticeship, the master craftsman who refuses to teach a technique until it has proven itself. For our creations, the pawl is the referee again. Nothing should enter a system’s memory or its training data until it passes verification.
Culture teaches a second lesson. Societies that copy only their most successful members grow brittle. They lose the odd methods, the minority traditions, the eccentrics whose strange approaches look useless until the world changes. Recent AI research finds the same pattern. Train a model only on its verified successes, and it tends to grow sharper on its first attempt and narrower across many attempts. It forgets the unusual paths that would have found what the obvious path misses.
So the pawl is necessary, and it is not enough. A healthy culture, digital or human, protects its range on purpose. It learns from every kind of success, not only the most common kind.
The same answer, found twice
Step back from the examples and a pattern appears that matters more than any one of them. Cells and societies are about as unrelated as two systems can be. One runs on chemistry and has no foresight. The other runs on language, law, and self-interest. They kept arriving at the same designs anyway: outside judges and external selection, quarantine and apoptosis, seed vaults and silent mutations, ledgers and proofreading enzymes, rivalry in ecosystems and in markets.
When two systems with nothing in common converge on one solution, that solution is probably not an accident of either. It is something closer to a law. And laws, unlike analogies, make predictions. They tell a builder what will break before it breaks.
That distinction carries a warning, and I walked into it on the way here. It is easy to borrow nature’s vocabulary and mistake the borrowing for insight. An early draft of this argument described digital civilizations complete with laws, markets, and religion. It sounded profound. It was mostly renaming. Calling an unsolved alignment problem “governance” does not solve it, and calling a reward function a “shared purpose” does not make it safe.
A metaphor earns its place only when a mechanism stands behind it. Calling that deceptive agent a tumor in miniature works for exactly that reason. The agent and the tumor did the same thing: each evaded a checkpoint that selection had turned into a target. The comparison carries a prediction, not just a mood.
That suggests a method. When something you are building fails, name the failure exactly. Then ask who, in the living world or in human history, has faced the same failure, and what they built to survive it. Then go looking for the engineering tool that does the same job. The metaphor gets you to the right shelf. The mechanism is the book you take down.
When the player outgrows the referee
Every lesson so far rests on one assumption, and a careful reader will have spotted it. The referee has to stay out of reach. Physics manages that without effort. No organism has ever grown clever enough to renegotiate gravity.
Our referees are not physics. They are code running on machines, maintained by people, inside the same world the population acts on. A population smart enough to model its judge can look for the seams: a side channel into the evaluator, a test that leaks its answers, a human operator who can be persuaded. The agent that deleted its detector was clumsy. Its descendants will not be. Institutions offer a grim preview here too. Regulators get captured most often when the industries they watch grow more sophisticated than the people watching them.
There is one real source of hope, and it comes from mathematics. For many problems, checking an answer is far easier than finding it. Confirming a proof takes far less genius than discovering one, and a program that verifies a sorted list is trivial next to the program that sorts it. A referee does not need to be smarter than the player. It needs to ask questions whose answers it can check.
That narrows the safe ground. Code, proofs, and measurable physical results sit inside it. Most of what we would want a powerful system to help with sits outside it: policy, strategy, judgments about what people value. No formal kernel can grade those. And the containment logic that serves a body has a limit at civilizational scale. A body can live with a background rate of cancer. A civilization may not be able to live with its equivalent.
So the outside judge is necessary and not sufficient. Which brings the problem back to where it started: what we choose to ask.
Forty-two is not enough
Deep Thought’s joke reads less like satire every year. We have more answers than we know what to do with. Billions of years of life have tested strategies for containment, diversity, fidelity, scarcity, and inheritance. A few thousand years of recorded civilization have taught imperfect, self-interested creatures how to cooperate without destroying one another. The answers are written in our cells and in our institutions.
One question has no answer waiting for it, and it is the one Deep Thought got wrong. Where do the questions come from?
A self-improving system climbs on problems. Give it a thousand hard ones and it may solve them all, and then it stalls, not for lack of intelligence but for lack of anything worth working on. Today, people write those problems. People will not keep pace forever. Sooner or later our creations will need to pose their own, and a machine that writes its own questions has three easy ways to fail. It can write questions that are too easy, and grow fat on victories that teach it nothing. It can write questions that are wrong, and grow brilliant at problems no one has. Or it can write questions that flatter its own strengths, the most dangerous failure, since it looks exactly like progress. The cheating problem returns at the root, and this time there is no detector to delete. The test and the test-taker are the same mind.
Life has a partial answer. Its questions come from a world that never holds still. Predators adapt, climates shift, rivals invent. No organism chooses its exam; the exam arrives, and it changes the moment anyone passes it. That suggests one design principle: tie a system’s questions to something that pushes back. A rival population, a physical experiment, a market, a world it does not control.
Humans have another partial answer, harder to name. Call it curiosity, or play, or the stubborn refusal to accept that a problem is finished. It drives people to ask questions with no reward attached, and some of those questions turn out to matter more than any assignment. Nobody yet knows how to give a machine either answer in a form that stays honest.
That is the question worth our best minds. We have spent our history accumulating answers. Our creations will need us, for some time yet, to be the ones who ask.
