Why does ChatGPT give you generic answers instead of insight?

The strongest evidence for it is not the marketing around AI. It is the weird research results that refuse to stay inside the autocomplete frame.

One of the examples that stuck with me was Othello. Researchers trained a transformer on sequences of Othello moves written as text. Just move notation. No pictures of the board. No explanation of the rules. No visual state. Then they looked inside the model and found that it had effectively built an internal representation of the board. It knew which squares were occupied and by which color as the game progressed.

Nobody handed it a board.

It inferred the board because predicting the next legal move required some representation of the state behind the text.

That feels important. The model was not just learning surface correlations between tokens. It was learning some of the structure that generated the tokens.

Then you see similar work around space and time, where researchers find that models represent them as actual directions in activation space. You see work showing that truth and falsehood have geometry inside the model. You see circuit-level work suggesting genuine intermediate reasoning steps, where the model seems to compute latent concepts on the way to an answer instead of jumping straight from prompt to output by pattern match alone.

When I first started reading that research, it felt like the story was getting cleaner.

Maybe the model really is building a world model.

Maybe next-token prediction at enough scale forces the system to simulate the world behind the text.

I still think there is something real in that direction. But the cleaner version started to fall apart too.

Because there is other evidence that complicates it in a serious way.

The piece that hit me hardest was the growing evidence around memorization. Some recent work showed models reproducing huge chunks of books with shocking fidelity. Not themes. Not abstractions. Near-verbatim text.

That matters because it means the weights are not just storing distilled conceptual knowledge. Some of the corpus seems to remain near the surface. The compression is not uniform. It is not neat. It is not elegant in the way people sometimes want it to be.

Then there is work showing that in some tasks where the model performs well, it may not have built a coherent internal model at all. It may have built a giant patchwork of heuristics that happens to get the answer right under familiar conditions and falls apart when the terrain shifts.

And if you look at what the major labs are doing, they are not acting like text prediction alone solved intelligence. They are building multimodal systems, world models, simulation architectures. The people closest to the metal are still trying to bridge the gap between language and physical reality.

So I do not think the honest answer is that LLMs are full world models.

I also do not think the honest answer is that they are glorified autocomplete.

The frame that seems to fit best is narrower and, to me, more useful: LLMs are compressed maps of human knowledge encoded in text. Rich, structured, uneven, overlapping, and often contradictory.

It explains why they can be so strong in domains where humans have written deeply for a long time. Law. Medicine. Programming. Strategy. Any field where people have spent decades externalizing reasoning, disagreements, case studies, edge conditions, postmortems, and competing frameworks into text.

It also explains why they can be strangely weak at things that feel obvious to a person moving through the physical world. Basic spatial intuition. Embodied common sense. Tiny facts about reality that nobody bothers writing down because they are too obvious to state.

The model has the language half of human knowledge.

The physics half is partial.

Once I started looking at them this way, the next question got sharper.

If these models really contain that much structured knowledge, why do everyday interactions so often feel generic?

Why do you ask a serious question and get something that sounds like a polished summary from someone who does not actually have skin in the game?

I think there are two reasons, and both are structural.

The first is the interface.

You open ChatGPT or Claude or Gemini and it looks like a conversation. There is a blinking cursor. A chat bubble. A turn-taking rhythm. It presents itself in the shape of another person. So you naturally treat it like another person.

You ask it a question the way you would ask a colleague. You judge the answer the way you would judge a colleague. You start unconsciously projecting human cognition onto a system that is not human cognition.

I do this too. Everyone does.

But I think that frame hides what the model actually is.

To me, an LLM feels less like a person and more like an instrument.

The weights are the medium. Your methodology is the optics.

That analogy clarified a lot for me. If you reduce spectroscopy to “shining light at stuff,” you are technically correct and you have missed the point. The insight does not come from the existence of the light alone. It comes from the structure of the probe, the way the measurement is taken, and the framework used to interpret what comes back.

Using an LLM like a casual chat partner is a little like using a scientific instrument as a paperweight. The instrument is real. The methodology is missing.

The second reason is even more important.

RLHF.

Reinforcement learning from human feedback is what turns a raw model into something pleasant enough to use every day. It smooths the edges. It makes the model helpful. It makes it safer, more polite, more coherent, more aligned with what a product should feel like.

It also flattens the output.

When you reward answers that sound balanced, broadly acceptable, non-threatening, and generally useful, you select for the center of the distribution. You select for the average shape of human-approved response.

That center is exactly where insight goes to die.

The center contains what everybody already agrees on, or at least what nobody objects to strongly enough to penalize. It contains the summary layer. The safe layer. The polished layer. It sounds intelligent because it is fluent and well-formed. It often feels hollow because it has been pulled away from the edges where real practitioners spend their time.

I started calling this the beige layer in my own head. It is the reason ChatGPT gives you generic answers to hard questions.

That tracks with what a lot of people experience but struggle to name.

People use ChatGPT every day and it saves time. But when the stakes go up, the output starts sounding like it was assembled from a thousand smart-sounding fragments. Which, in a sense, it was.

The surface layer of the product is optimized for usability, not for extracting the deepest structure encoded in the model.

So the gap is not only inside the model.

It is in how we access it.

Once I saw that more clearly, my own usage changed. I stopped treating the prompt as the main event. I stopped thinking in terms of “How do I ask better?” and started thinking in terms of “What am I actually probing for?”

There is a big difference between asking for an answer and asking for the physics.

If you ask, “What is the best running shoe?” you get the centroid of reviews, affiliate content, specs, brand perception, and vague consensus. You get the beige answer.

If you ask for the structural tradeoffs between foam compounds, failure modes under different body weights, how plate stiffness interacts with fatigue patterns, why certain shoes feel incredible in short tests but degrade badly over training volume, now you are probing a different part of the map.

One question asks for recommendation output.

The other asks for mechanism.

Mechanism is where the heavy knowledge lives.

I think of the model’s knowledge space as having density. Some parts are reinforced from many independent angles. Those parts feel heavy. They survive rewording. They survive adversarial questioning. Different models often point toward similar underlying structure there, even if they disagree on the packaging.

Other parts are light. Thin. Easy to fake. Easy to perturb. They come apart when you reframe the question or push one level deeper.

That distinction has become more useful to me than almost any prompt trick.

So when people ask me why AI gives platitudes instead of insight, I do not think the answer is “because the models are dumb.”

I think the answer is closer to this:

You are usually interacting with the conversational surface of a compressed knowledge instrument that has been heavily optimized to sound broadly helpful. The product layer gives you a socially approved synthesis. The deeper structure stays buried unless your methodology is strong enough to pull on it.

What changed my own practice: reframing questions instead of refining them. Probing from multiple angles. Asking for failure modes instead of recommendations. Forcing the model to separate mechanism from narrative. Comparing outputs across frontier models because each one has different training history, different post-training behavior, different institutional pressure baked into the surface.

Where they converge independently, I pay attention.

Where one of them breaks sharply from the others, I pay attention there too.

Divergence is often where the interesting terrain starts.

I do not treat that convergence as proof. These models overlap a lot. Their blind spots can rhyme. They can share the same contamination from the broader internet. They can agree on something and still be wrong.

But the behavior under structured probing still tells you something useful. It tells you whether the representation seems dense or fragile. It tells you whether you are touching a reinforced part of the map or skimming a thin layer of approximations.

There is a lot more in the full article - why AI gives generic answers at a structural level, how I think about context engineering and heavy versus light knowledge, and how to actually probe these systems in a way that gets past the beige layer.

You can read that here: michaelquoc.com/articles/do-llms-actually-understand/

I think we are in a strange moment where a lot of people have access to an instrument they are still treating like a chatbot. The result is that they keep touching the shallow layer and concluding that the whole thing is shallow.

To me, that seems wrong.

Not because the hype version is right. A lot of it is not.

Because the models already appear to contain more structured knowledge than the interface lets you feel, and less grounded world understanding than the hype wants to claim.

That middle ground is where the interesting work is.

I want to hear how you’re thinking about it.