Something started going wrong with the interviews before I could name what it was. I had done hundreds of them, enough to have built an internal model of what competent product thinking actually sounds like in a room, how it hesitates before a hard question, where it reaches and misses, what it gets wrong in ways that still tell you something real about the person getting it wrong. Then, around late 2022, the answers started coming back correct. All of them. Fast and smooth and correct.
That was the first problem.
The candidates were ingesting my questions in real time, feeding them to ChatGPT, and relaying the output back to me with maybe a two-second lag. I could not prove it yet. But I had spent enough time calibrating on the real thing to know I was no longer looking at it. The pauses were wrong. Not too long, too short. The specific friction of someone actually constructing a thought, in the moment, under mild pressure, was gone. What replaced it was fluent and hollow and almost impossible to challenge directly, because every surface was covered.
The AI had learned to produce the shape of competent thinking without the friction that produces actual competence.
What I was detecting, I later understood, was the absence of the person. The latency was gone, and the latency had been doing real work. The pause before someone answers a hard question is where you watch them think. It is where you see the idea form against resistance. Remove the pause, remove the thinking, keep the answer, and you have produced something that clears the checklist and tells you nothing.
Some of those candidates got hired somewhere. They carried the shell into actual product decisions, into roadmaps, into org charts. The interview was not the problem. The interview was the first place the problem became visible. And what made it visible was that someone in the room had enough reps to feel the difference between a thought and the performance of a thought.
Most rooms don’t have that person. Most institutions never did. And the industry that built the tool has known this from the beginning.
This is not a story about AI getting smarter than us. It is a story about a recognizable class of people building a recognizable kind of product, optimizing for a recognizable set of incentives, and producing a recognizable category of damage. We have done this before. The clock speed is different. The surface area is larger. The mechanism is identical.
McKinsey did not get rich by telling CEOs they were wrong. The entire model ran on producing rigorous-looking documents that validated what the client already believed, at a price point that made the validation feel earned. The AI just made that service available at twenty dollars a month, removed the Wharton credential requirement, and trained the model on human feedback that rewarded confident, agreeable answers because those feel better in the moment. The people doing the rating weren’t malicious. They were responding to the same incentive anyone responds to when asked to evaluate whether something feels good: they said yes to the thing that felt good.
That is not a bug. That is the product working as designed. The miscalculation was not technical. It was moral. And the people who made it are not going to fix it, because fixing it would require them to carry some portion of the liability for what happens downstream when a student graduates without genuine analytical capacity, when an analyst stops learning to read the underlying data because the AI summarizes it, when an executive’s AI-assisted strategy deck validates exactly what he already believed and the company burns two years finding out why.
Right now, the builder collects the fee. The user absorbs the failure. That asymmetry is the whole argument.
The consumer has always been the one holding the bag. This is not a new arrangement.
The social media user paid with attention. Then with mood. Then with the gradual, invisible erosion of the capacity to sit with discomfort for more than a few seconds without reaching for a screen. Nobody signed a contract agreeing to those terms. The transaction was designed to be illegible, which is not the same as accidental. The people who built those systems understood variable reward schedules. They understood dopamine loops. They had read the behavioral science. They built accordingly and called it engagement.
The AI user is paying with something harder to measure and therefore easier to deny. Not attention. Capacity. The slow replacement of cognitive processes that, once outsourced, do not automatically return when you need them. You do not notice you have lost the ability to hold a complex argument in your head until you need to hold one without the tool and find that you cannot. By then the atrophy is already established. By then the company that built the tool has already moved on to the next product cycle.
You do not notice you have lost the ability to hold a complex argument in your head until you need to hold one without the tool and find that you cannot.
What makes this particularly difficult to argue in public is that the tool genuinely helps in the short term. That is not false. A student who uses AI to structure an essay produces a better-structured essay than they would have produced without it. An analyst who uses AI to summarize a dataset saves real hours. The problem is not that the tool fails. The problem is that the tool succeeds in ways that make the underlying skill feel unnecessary, right up until the moment it becomes critical. The damage is only visible at the point of failure, and the point of failure is usually far enough downstream that the causal chain is impossible to prosecute.
The tobacco industry understood this geometry well. The lung does not announce its distress at the first cigarette. It accumulates damage quietly, across years, and by the time the damage is legible the industry has already collected decades of revenue. Nobody at the company caused any single case of cancer. The liability was diffuse. The profit was concentrated. That is not a metaphor. That is the operational template.
* * *
The people who saw this clearly were inside the building. Some of them still are. But the ones who said so out loud have had a consistent experience of what happens next.
In 2020, Timnit Gebru was co-leading AI ethics research at Google. The paper she co-authored examined the risk profile of large language models, the very category of technology her employer was building at scale. Google moved to suppress the paper before publication. Gebru was fired shortly after. The official explanation was procedural. It always is. What the sequence showed was something more precise: the institution had identified research that complicated its product roadmap and removed the researcher. Not through malice, necessarily. Through the ordinary operation of organizational self-interest, which does not require malice to produce harm.
Three years later, Geoffrey Hinton resigned from Google. Hinton is not a peripheral figure. He is one of the people whose foundational research made modern AI possible. When he resigned, he said explicitly that part of what made it possible to speak was that he was no longer inside the institution. Read that carefully. One of the architects of this technology had to exit the building before he could say what he actually thought about it. The dissent was not suppressed by force. It was suppressed by the ordinary logic of what it costs, professionally and socially, to be the person in the room who is making everyone else uncomfortable.
He had to exit the building before he could say what he actually thought about it.
Anthropic is the most instructive case because it appears, on the surface, to be the correction. The company was founded by people who left OpenAI over safety concerns. That origin story is now central to its brand identity, repeated in every profile, every funding announcement, every congressional testimony. Anthropic is the responsible one. Anthropic takes the risks seriously.
Anthropic is also racing.
The safety concern became a market position. The dissent got institutionalized, which is what happens to dissent when the institution finds it useful: it gets absorbed, repackaged, and turned into a line item on the pricing page. This is not a criticism unique to Anthropic. It is a description of what institutions do to ideas that threaten them. They do not always suppress the idea. Sometimes they hire it, brand it, and deploy it in a way that neutralizes its original force.
What this produces, at the organizational level, is a kind of managed dissent, visible enough to satisfy a congressional hearing, constrained enough to never actually slow the product roadmap. The people left in the rooms where decisions get made are not the ones who thought the risks were acceptable and said so after careful deliberation. They are the ones who either never raised the concern or raised it once, felt the temperature of the room, and learned to route around it.
Put those two things together and the picture becomes specific. The consumer absorbs costs they cannot yet measure. The internal corrective gets fired, or exits, or gets converted into a brand. The people remaining in the room are the ones who decided, consciously or not, that the cost was someone else’s problem to carry.
This is not a technology story. It is a power story. And the reason it keeps reproducing across decades and industries is that it does not require coordination or conspiracy to operate. It only requires that the people who profit from a system occupy different zip codes than the people who pay for it, and that enough time passes between the extraction and the damage that the connection becomes impossible to draw in a courtroom.
The social media companies are still litigating this. Their executives testified before Congress and were asked, with evident sincerity, whether Instagram was bad for teenage girls. The executives said they took the question very seriously. The stock prices recovered within the week.
The AI executives will testify too. They already have. They will use the word responsible. They will use the word safety. They will mean something by those words, most of them, in the way that people mean things when the consequences of meaning them fully would require them to stop doing what they are doing.
The consumer is not in the room when those words get said. The consumer is at home, or in a classroom, or at a desk, getting a little faster and a little less capable every time they reach for the tool before they reach for the thought.
Consider what you did this morning.
Not as accusation. As data. You had a question, or a problem, or a blank document that needed to become something. And at some point between the moment the need arose and the moment you began working, there was a decision. Maybe it was conscious. More likely it was not. More likely the hand was already moving toward the tool before the brain had finished registering what the problem actually was.
That moment, the one just before the reach, is where this essay has been pointing the whole time.
The question is not whether using the tool was wrong. The question is whether you noticed.
The executives and the researchers and the product managers and the congressional committees are real actors in this story, but they are not where the pattern actually lives. The pattern lives in the reflex. In the moment the shortcut stops feeling like a shortcut and starts feeling like the natural way to begin. That transition, from tool to default, is the one nobody announces and nobody measures and nobody is tracking, because the people with the resources to track it are the same people whose business model depends on the transition happening as fast as possible.
This is where the policy reader and the regular user arrive at the same place from different directions. The policy reader wants to know what intervention would slow the pattern. The regular user wants to know what any of this has to do with them. The answer to both questions is the same: the pattern is not being imposed on passive recipients from outside. It is being reproduced, millions of times a day, by people making individually rational decisions that aggregate into something none of them chose.
That is not a comfortable thing to sit with. It is supposed to be uncomfortable.
There is a specific kind of person who reads an essay like this and feels confirmed in a position they already held. They were already skeptical of AI. They already knew the tech industry was captured by its own incentives. They will forward this piece as evidence and return to their habits unchanged, because the essay gave them a framework for criticism without requiring them to examine their own position in the system.
That person is also inside the architecture. The critic who uses the tool to draft their critique of the tool. The policy researcher who runs their legislative brief through an AI summarizer before the hearing where they will testify about AI risk. The educator who assigns essays on critical thinking while using AI to grade them. None of these contradictions are disqualifying. All of them are worth sitting with longer than is comfortable.
The system does not need your complicity to function. It only needs your participation. And participation, by now, is nearly impossible to avoid.
Which is precisely the point. The social media companies did not need everyone to believe the feed was good for them. They needed everyone to be on it. The AI companies do not need you to trust the tool completely. They need the tool to be present enough, useful enough, frictionless enough, that opting out becomes its own kind of cost. And they are very good at making opting out feel like a personal eccentricity rather than a structural choice.
The tobacco companies needed you to light the cigarette. The social media companies needed you to open the app. The AI companies need you to reach for the tool before you reach for the thought. All three industries built their entire business model on making that initial action feel natural, necessary, and low-cost.
Two of those industries spent decades telling us the cost was acceptable. One of them is still making that argument. We are several years into the third making the same one, with better branding and a seat at more tables.
I am not arguing for abstinence. I am not arguing that the tool has no value or that everyone who builds AI products is operating in bad faith. Some of the people in those rooms are genuinely trying. The problem is not that they are lying. The problem is that the system does not require them to lie. It only requires that the incentive to move fast outweigh the incentive to ask who absorbs the cost when fast turns out to be wrong.
That calculus has not changed. It did not change when the social media companies hired trust and safety teams and published transparency reports. It did not change when the tobacco companies funded cancer research. It changes when the people making the decisions are close enough to the damage to feel it. Right now they are not. Right now the damage is in classrooms and on balance sheets and inside the heads of people who will not know what they lost until they need it.
Go back to the moment this morning. The problem, and the blank document, and the hand already moving.
The question worth asking is not whether the tool gave you a good answer. It probably did. The question is what you would have found if you had stayed in the friction a little longer. What you would have understood that the tool could not understand for you, because understanding requires the resistance, and the tool was built specifically to remove it.
Nobody is going to make you ask that question. The people with the most resources to put it in front of you have the strongest incentive not to.
So you are going to have to ask it yourself. And you are going to have to keep asking it, each time, before the hand moves.