Chain-of-Thought Prompting: When It Works and When It Doesn't

Chain-of-thought prompting — asking the model to show its reasoning step by step before giving a final answer — consistently improves output quality on complex reasoning tasks. It doesn't help on simple tasks and can actually hurt performance on tasks requiring direct answers. Knowing when to use it is more useful than applying it everywhere.

What chain-of-thought is

Instead of asking: "Should I use PostgreSQL or MongoDB for this project?"

You ask: "Before recommending a database, walk through: the key differences between relational and document databases, the specific requirements of my project (I'll describe them), which requirements favor each option, and what tradeoffs matter most for a 5-person startup. Then give me your recommendation."

The first prompt gets a one-sentence answer. The second forces the model to reason through the problem before concluding — which produces more accurate and better-supported answers on questions that actually require reasoning.

Why it works

Models generate text one token at a time. When you ask for an immediate answer, the model produces what's statistically most likely to be right based on the question — which may skip important reasoning steps. When you ask the model to reason first, you're changing what it generates: it has to produce reasoning tokens before conclusion tokens. The conclusion is then conditioned on better reasoning.

This is particularly effective on: - Multi-step logic problems - Decision-making with tradeoffs - Math and quantitative problems - Analysis tasks with multiple considerations - Questions where the right answer depends heavily on context

How to prompt for chain-of-thought

There are a few common approaches:

Direct instruction: "Think through this step by step before giving your answer." "Walk through your reasoning before concluding."

Scaffolded stages: "First, identify the key factors. Then evaluate each option against those factors. Finally, give your recommendation."

Staged thinking with explicit structure: "Work through this in four stages: (1) identify what information is relevant, (2) analyze the tradeoffs, (3) consider edge cases, (4) give your final recommendation."

The scaffolded approach — where you define the reasoning stages explicitly — produces the most reliable results on complex analytical tasks.

When it actually helps

Mathematical and quantitative problems. Asking a model to show its arithmetic prevents it from making calculation errors that don't show up when it jumps straight to an answer.

Multi-factor decisions. Architecture decisions, technology choices, strategic questions — any decision with multiple competing factors benefits from step-by-step analysis.

Debugging and diagnosis. "Walk through what you think is happening in this code before suggesting a fix" produces better debugging than "fix this bug."

Complex analysis. Research synthesis, evaluating arguments, comparing nuanced options — showing reasoning improves output quality a lot.

When it doesn't help (and can hurt)

Factual retrieval. "Who is the current CEO of Apple?" — chain-of-thought adds nothing. The answer either requires reasoning over facts or it doesn't, and a simple factual question doesn't.

Format conversion. Translating, reformatting, or transforming content doesn't require reasoning through steps — the model just needs to do it.

Creative tasks. For creative writing or brainstorming, asking the model to reason first before creating often produces more constrained, less imaginative output.

Fast, simple requests. Adding chain-of-thought prompting to simple tasks makes responses slower and longer without benefit.

The 2026 caveat: always-on reasoning models

Grok 4.3 and similar models with always-on reasoning (the model reasons internally before producing output regardless of your prompt) have partially automated what chain-of-thought prompting was doing manually. For these models, explicit chain-of-thought prompting provides less incremental benefit since reasoning is already happening under the hood.

For Claude Sonnet 4.6, GPT-5.5, and models that don't have always-on reasoning, explicit chain-of-thought prompting on complex tasks still provides meaningful quality improvement.

Simple rule

If a task requires more than one logical step to get right, ask for reasoning first. If it doesn't, skip it.