Plausible Isn't True

Here's what's actually happening when you talk to an AI like Claude or ChatGPT. You type a message. The system converts your words into numbers — little chunks called tokens — show definition. Then it runs one core operation, over and over: next-token prediction — show definition.

That operation is doing two different jobs, and it's worth separating them. During training, "guess the next token" was the objective: practicing it across a staggering amount of text is how the model absorbed grammar, facts, styles, and reasoning patterns. At generation time, it's how every reply physically comes out — every sentence, every paragraph, every seemingly brilliant insight arriving one token at a time, each chosen by asking, "Given everything that came before, what's the most likely next piece of text?"

"Autocomplete" describes that base engine better than it describes the products you actually use, though. Modern reasoning systems take a base model and train it further with reinforcement learning — rewarding it for working through problems in steps that lead to answers that check out. And when you ask them something hard, they spend extra computing time before responding (researchers call this test-time compute): drafting steps, evaluating them, backtracking. Whether "just autocomplete" is still a fair label for systems like that is genuinely debated among researchers. Two things are not debated: the reply still arrives one predicted token at a time, and none of these layers adds a built-in commitment to truth.

This is why a raw language model — one without tools or reasoning steps bolted on — can write a convincing essay about quantum physics and then confidently get an arithmetic problem wrong. It's not doing math. It's predicting what text usually follows math questions. Today's products usually do get the math right, and how they do it fits the pattern above: many models are trained to write the calculation out step by step, like a student showing their work, and some products also pass precise math to a calculator or code tool. Both approaches are real fixes. Neither turns the engine into something that knows arithmetic — it can walk through a correct calculation, and it can walk through a wrong one with exactly the same confidence.

This is also why AI hallucination — show definitions — a term for when it generates information that sounds true but isn't. Plausibility is the engine's native yardstick; truth has to be brought in from outside — by tools, by search, or by a human who checks. In the Avianca case, the model was doing what the engine underneath always does: predicting plausible-sounding text based on the patterns it learned. And a fake case citation looks, statistically, a lot like a real one.

Researchers Emily Bender and Timnit Gebru coined a useful metaphor: they called large language models — show definition stochastic parrots — show definition. A parrot can repeat words that sound meaningful to you, but the parrot doesn't understand what the words mean. It's producing sounds that match patterns it was exposed to. Language models do something similar at a much larger scale — they produce text that matches patterns in training data. Whether that qualifies as "understanding" is genuinely contested today, especially for newer reasoning-trained models that visibly work through problems step by step. But the practical point still holds: the system has no built-in commitment to truth, and fluent text is not evidence of accurate text.

This doesn't make AI useless. Far from it. A prediction engine trained on the entire internet can do remarkable things. But it changes what you should expect from it and — critically — when you should trust it.

Don't let the first-person language fool you, though. When the system says "I think" or "I believe," those words are pattern-matched text, not reports from an inner life — it's producing the kind of text that typically follows a question like yours. Whatever is or isn't going on inside, it has no stake in being right, and it can sound exactly as sure when it's wrong.

Once this clicks, you'll use AI completely differently. You'll stop asking it what it "thinks" and start treating it as what it is: a powerful tool for generating, summarizing, and transforming text — one that requires a human to verify anything that matters.