What Your LLM Is Actually Doing When You Say ‘Think Step by Step’

You’ve probably noticed that adding “think step by step” to a prompt often produces better answers. More organized, fewer obvious errors, sometimes dramatically more accurate on math or logic problems. Most people assume this works because they’ve given the model permission to be more careful. That’s not quite right, and understanding the real mechanism will make you a much more effective prompt writer.

The Trick Wasn’t Obvious, Even to Researchers

The phrase “chain-of-thought” prompting entered the mainstream AI conversation through a 2022 Google Brain paper by Jason Wei and colleagues. The finding was simple but surprising: if you show a language model examples where reasoning is written out step by step before the final answer, it solves multi-step problems dramatically better than if you just ask for the answer directly.

The zero-shot version, just appending “Let’s think step by step” to a question without any examples, was published the same year by Takeshi Kojima and colleagues at the University of Tokyo. That paper demonstrated that the phrase alone, with no demonstrations, improved performance on arithmetic, symbolic reasoning, and commonsense tasks across a range of models.

Both papers showed measurable gains. On some benchmarks, accuracy on multi-step math problems more than doubled. That’s not a marginal improvement. Something real is happening here.

The question is what.

What the Model Is Actually Doing

An LLM generates text one token at a time. Each token is chosen based on everything that came before it, including the tokens the model itself just produced. This is the key to understanding chain-of-thought.

When you ask a model to answer a complex question directly, it has to compress all the intermediate reasoning into a single output decision. The answer token has to somehow encode the solution to a problem that might require five or six inference steps. That’s asking a lot of one probability distribution.

When you ask the model to reason out loud first, each intermediate step becomes part of the context for the next step. The model isn’t retrieving a pre-formed answer from somewhere. It’s constructing tokens in sequence, and each token it writes influences what comes next. Writing “there are 12 months in a year, and she saves $50 per month, so that’s $600 per year” isn’t just explaining its work. That text is literally the scaffolding on which the final answer is built.

Think of it as the model using its own output as working memory. Transformers have a fixed context window, but no persistent internal scratchpad between layers. By generating intermediate text, the model creates an external record of its reasoning that feeds back into subsequent computations.

Why This Is Surprising If You Think About It Carefully

Here’s what makes this counterintuitive. The model didn’t suddenly become smarter when you added those four words. Its weights didn’t change. Its architecture didn’t change. You just changed the format of the output you asked for.

And yet accuracy improved substantially. This tells you something important about where the model’s capability was getting lost in the first place. It wasn’t that the model lacked the knowledge to solve the problem. It was that the direct-answer format forced it to skip the intermediate steps that connect that knowledge to the answer.

This is also why chain-of-thought prompting works better on harder problems and doesn’t help much on simple ones. If a question only needs one inference step, there’s nothing to scaffold. “What is 2 plus 2?” doesn’t improve with step-by-step reasoning. But “A train leaves Chicago at 9 AM traveling west at 60 mph…” is exactly the kind of problem where intermediate steps matter.

[As an aside, this connects to a broader point about model size. Chain-of-thought prompting also tends to work better on larger models. Smaller models sometimes produce plausible-looking reasoning chains that lead to wrong answers, a problem explored in depth here.]

What the Model Gets Wrong Anyway

Being clear-eyed about the limits here is important. Chain-of-thought prompting reduces certain kinds of errors but doesn’t eliminate them, and it introduces a specific failure mode you need to watch for.

Models can produce confident, coherent-sounding reasoning chains that reach wrong conclusions. The intermediate steps look reasonable. The logic appears to flow. And then the answer is incorrect. This happens because the model is optimizing for plausible text, not verified truth. A reasoning chain that sounds good can still contain a subtle error in step three that corrupts everything downstream.

Researchers call this “unfaithful reasoning,” where the stated chain of thought doesn’t accurately reflect the computation that produced the output. The model might have reached the answer through pattern matching on the final token, then constructed a post-hoc explanation that sounds logical. Your LLM has no real-time verification mechanism, so it can’t catch these errors itself.

Practically, this means you shouldn’t treat a well-formatted reasoning chain as a guarantee of correctness. It raises the probability of correctness. It doesn’t certify it.

How to Actually Use This Well

Knowing the mechanism gives you real leverage. Here’s how to apply it.

Be specific about what kind of steps you want. “Think step by step” is a reasonable starting point, but you can do better. If you’re asking the model to debug code, try: “Before answering, identify the specific line where the problem might originate, then explain what that line does, then explain why that behavior might cause the issue.” You’re structuring the scaffolding yourself, which often produces more useful intermediate steps than letting the model choose its own format.

Use it for the right problems. Multi-step math, logical deductions, code debugging, and planning tasks all benefit substantially. Simple factual lookups, short summaries, and creative tasks where you want fluency over accuracy generally don’t benefit much, and the extra tokens can actually dilute the output.

Ask the model to check its work as a separate step. This exploits the same mechanism in reverse. After the model produces an answer, ask: “Now review each step of your reasoning and identify any step where you might have made an error.” This creates a second pass where the model uses its output as input for error-checking. It doesn’t always catch mistakes, but it catches some.

Match the prompt to the task structure. If your problem has a natural sequence (gather information, analyze it, make a recommendation), explicitly name those phases. “First summarize the constraints. Then identify which constraints conflict. Then propose a solution that minimizes conflicts.” You’re essentially writing the table of contents for the reasoning chain before the model fills it in.

The Bigger Picture for How You Use LLMs

Chain-of-thought prompting is one of the cleaner examples of a broader truth about language models: the format you ask for shapes the computation that produces it. This runs counter to the intuition that a model either knows the answer or it doesn’t. In reality, whether and how the model generates intermediate representations has a significant effect on what answer it reaches.

This means prompt engineering is genuinely consequential, not just cosmetic. Changing your prompt changes the computational path through the model, not just the presentation of a fixed result. When you ask for step-by-step reasoning, you’re not just requesting a different format. You’re changing what the model does.

The practical implication is that if you’re getting poor results from an LLM on complex tasks, the first thing to examine is whether you’re asking for answers directly when you should be asking for process. Often the knowledge is there. The chain connecting it to the output isn’t.

What This Means

Here’s a quick summary of what you can apply immediately:

• “Think step by step” works because each intermediate token the model generates becomes context for subsequent tokens, giving it a working memory for multi-step problems
• This effect is real and substantial on complex tasks, minimal on simple ones
• Coherent-sounding reasoning chains can still be wrong, so don’t treat them as verified output
• You get better results when you specify the structure of the steps rather than leaving it generic
• Asking the model to review its own chain after producing it is a low-cost way to catch some errors
• The format you ask for isn’t just cosmetic. It changes the computation

The next time you reach for “think step by step,” you’ll know you’re not giving the model permission to slow down. You’re giving it a mechanism to think with.