If LLMs can’t reason, show your work.

Nikhil Shinday ·

I am inviting anyone who says LLMs cannot reason or understand in ways humans can to make the technical case. Define the capacities so the claim can fail. Explain what humans do when they exercise them, what the relevant biological machinery contributes, and what prevents the specified models from doing the corresponding work. Give me a test, a causal argument, or a proof with explicit assumptions. I would like something more substantial than another confident declaration that the machines are “just predicting the next word.”

I am tired of that declaration being treated as the end of an argument that has barely started. There are serious limitations to investigate. There are also people turning those limitations into universal prohibitions without doing the work between the two.

Joscha Bach’s September 24 post captures the challenge: a categorical denial of thought needs a definition and an account of the causal processes producing the behavior.

I take that demand seriously without adopting every claim in the post. Similar outputs do not establish equivalent causal processes. The absence of a proof that LLMs cannot reason does not prove that they can, much less that they reason exactly as humans do. Both claims need evidence. What I object to is treating a disputed conclusion as a premise everyone is obliged to repeat.

The claim keeps getting bigger than the evidence

This argument has been recurring for years. In October 2023, Yann LeCun described autoregressive LLMs as having “very limited reasoning and planning abilities” and predicted that size and training data would not fix them. “Limited” deserves to survive the retelling: it is a different claim from “none.”

François Chollet made a stronger claim in June 2024: “LLMs cannot plan and reason on their own,” while explicitly recognizing their usefulness alongside symbolic planners. That qualification matters too. A criticism of an unaided model cannot automatically be applied to every system that contains one.

There is real empirical work behind parts of this debate. GSM-Symbolic tested arithmetic problems under controlled changes to numbers, clauses, and irrelevant information, finding substantial fragility. The Illusion of Thinking found complexity-dependent failures in controlled puzzle environments. These are valuable findings about the tested models and conditions. They are not a September 2026 inventory of every model, and a collection of failures is not an impossibility theorem about every future language model.

A system can possess a capacity unreliably. It can also achieve a good score through shortcuts that fail to demonstrate the capacity we intended to measure. Distinguishing those possibilities is the interesting problem. Repeating “pattern matching” does not distinguish them.

Define something that can lose a test

Here are two proposed definitions. They are starting points for disagreement, not declarations that philosophy or cognitive science has settled these words.

Reasoning in a specified domain means using represented premises, relations, and constraints to derive conclusions or plans that transfer to genuinely new combinations, with conclusions changing appropriately when relevant premises change.

Understanding a specified domain means possessing and using representations of its entities, relations, and constraints that support explanation, prediction, and appropriate counterfactual judgments across paraphrases and new cases.

Those definitions deliberately require more than a fluent answer. They also avoid defining the capacity by the species or material implementing it. If understanding additionally requires embodied grounding, social intention, phenomenal experience, or a particular biological process, state that requirement and defend it. Bender and Koller’s argument that form alone cannot supply meaning is a substantive challenge to engage with. A benchmark score does not dispose of its premises.

The language of proof should carry some weight. This September post announces that Oxford has mathematically established that LLMs cannot invent anything and concludes, “LLMs don’t think. You do.” I am citing the post as an example of that rhetoric, not endorsing its description of a paper or its historical claims.

“Mathematically impossible” needs a mathematical object, assumptions, and a conclusion that follows. An empirical claim needs observations capable of defeating it. For my proposed definitions, a reproducible failure to transfer after relevant controls would count against a claim of competence in that domain. Success across those controls would count against a blanket denial. Neither result would settle consciousness, every other domain, or whether the underlying algorithms match human ones.

If biology is the argument, explain the biology

A serious human account must connect cellular computation to representations, memory, and control. “The brain is biological” does not supply those connections.

At the cellular level, the machinery already exceeds the familiar cartoon of a neuron as a switch. Gidon and colleagues recorded calcium-mediated dendritic action potentials in human cortical tissue. Their results showed nonlinear computational properties that allow individual dendrites to classify inputs a simple linear unit cannot. That is a concrete biological mechanism. It does not establish that this mechanism alone explains reasoning, or that another physical implementation cannot perform the relevant computation.

At the level of working memory, the question becomes how information remains available when it is not currently the focus of attention. Rose and colleagues found that a TMS pulse could briefly restore a decodable representation of an unattended memory item, with behavioral consequences when it remained relevant. They interpreted the results as supporting latent, “activity-silent” memory and short-term synaptic mechanisms. That interpretation is more specific than “the brain remembers,” and more cautious than pretending the experiment directly measured every relevant synaptic change.

Flexible control has its own evidence. In a study of 80 patients with focal lesions, damage associated with the frontoparietal multiple-demand system predicted fluid-intelligence deficits; language-system damage showed a different relationship, including with verbal fluency. This helps separate systems involved in flexible problem solving from systems supporting language. It does not imply that language never assists thought.

Concepts and inference also depend on memory. Disrupting anterior temporal processing with rTMS slowed picture naming and word comprehension, supporting a causal contribution to semantic processing. Hippocampal and prefrontal research links the reinstatement and integration of overlapping experiences to novel inferences. The latter evidence includes fMRI associations; it should not be presented as the same kind of causal intervention as stimulation or lesions.

These findings give us parts of an explanation: nonlinear local computation, temporary information maintenance, conceptual representations, memory integration, and control over their use. They do not add up to a complete biological definition of understanding. Listing brain regions would not fill that gap either.

Nobody needs to solve neuroscience to demonstrate that a model fails a task. But if the claim is that some biological property makes human reasoning possible and makes machine reasoning impossible, identify that property, establish its causal role, and explain why it is necessary. A description of how humans implement a capacity is not yet a proof that only humans can implement it.

Give the machine the same depth of explanation

Adrian Zidaritz’s August 2024 post begins, “LLMs cannot reason,” then points to combining a model with Prolog. The proposed combination gives us something concrete to examine: which component represents the problem, which executes the inference, and what happens when each component is removed?

For a standard autoregressive transformer, start with the actual architecture. Tokens become vectors with positional information. Learned projections produce queries, keys, and values. Attention uses query–key compatibility scores, normalized into weights, to combine value vectors from permitted positions. Nonlinear feed-forward transformations modify those representations; residual connections carry information through successive layers. A final readout produces a probability distribution over the next token. Modern models vary the details of the original Transformer design.

Training adjusts parameters to improve the objective. The objective does not, by itself, tell us which algorithm the trained network uses on a particular problem. During ordinary inference, a standard pretrained model keeps its weights fixed while its activations and context change. Generated tokens enter the context and can provide intermediate computational material for subsequent steps. Fixed parameters are not a frozen computational state.

The empirical question is what representations and operations that process learns. Othello-GPT, trained on move sequences without an explicitly supplied board, developed an internal representation of board state. Researchers intervened on that representation and changed predictions in ways consistent with changing the represented board. This is a narrow synthetic setting. It still makes “sequence prediction can only retrieve phrases” an inadequate account of the demonstrated computation.

Anthropic’s work on Claude 3.5 Haiku gives another example. For a question about the capital of the state containing Dallas, the researchers traced an intermediate Texas representation. Suppressing it and injecting California features changed the answer toward Sacramento. The report uses approximate replacement models and selected case studies; it is vendor-authored, and its authors explicitly describe existence proofs rather than a complete map of the model. Those limits belong beside the result.

A printed chain of thought is much weaker evidence. Subbarao Kambhampati shared this warning in June 2026, and the central caution is sound: seeing an explanation does not establish that the explanation reveals the process producing the answer.

The Haiku report itself includes unfaithful explanations. We need interventions, ablations, transfer tests, and competing hypotheses about internal computation. We also need to distinguish long-term weights, temporary activations, context, retrieval, and external memory. “It has no memory” is not an adequate substitute.

Human cognition occurs in embodied, recurrent biological systems continually interacting with the world. A standard text-only pretrained transformer does not reproduce that whole organization. Explain which difference causes the claimed deficit, under which conditions. “Neurons” on one side and “matrix multiplication” on the other compare different explanatory levels. Neither phrase tells us whether the systems can represent a premise, compose relations, or revise a conclusion.

A style guide does not settle comparative cognition

The AP Stylebook made this worse on September 23: “Artificial intelligence systems do not think, feel, want or understand.” It then defined anthropomorphizing as ascribing human traits, emotions, or behaviors to “non-human things, such as animals or inanimate objects.”

To be precise, AP did not literally write that animals cannot think or feel. Animals can be anthropomorphized when we project specifically human characteristics onto them. But putting animals beside inanimate objects in that paragraph invites a thoroughly unhelpful inference: that recognizing cognition or emotion outside our species is itself a category mistake.

That is a ridiculous amount of philosophical confidence to smuggle into style guidance. A stylebook can recommend precise descriptions. It cannot settle disputed questions about cognition by making the categorical answer a house rule. AP’s broader guidance to examine performance, ownership, harms, and exaggerated marketing claims is useful. The opening declaration deserves the same scrutiny it asks journalists to apply elsewhere.

Animals make the stakes particularly clear. In Robyn Crook’s octopus study, animals learned to avoid a place associated with a noxious injection and preferred a place associated with anesthetic relief after that injection. The anesthetic was not independently rewarding in the saline controls. Together with grooming behavior and nerve recordings, the findings supported an affective pain interpretation beyond a simple withdrawal reflex. The paper appropriately described that experience as likely, not directly observable from outside.

I think language that casually makes animal experience sound like human projection risks reinforcing the dismissal of suffering. That is my concern about the framing, not a measured claim about this tweet’s effects. It also establishes nothing about whether a chatbot feels pain. Reasoning, understanding, affect, and consciousness are separate questions. Converging evidence and explicit uncertainty serve all of them better than deciding in advance which subjects are allowed the verbs.

Put the claim somewhere we can examine it

This is the standard I want behind posts like the one below, which leaps from an alleged proof about reasoning to reassurance about jobs. I have not established its characterization of the study. An institution’s name and a link do not make the advertised conclusion follow.

Build a test around an explicit domain: relational inference, planning under constraints, or causal prediction, for example. Generate fresh problem families with controlled compositions. Separate familiar templates from withheld structures. Change relevant premises, rename entities, paraphrase instructions, and add irrelevant details. Specify expected invariances and expected changes before inspecting outputs. Include contamination controls rather than assuming novelty from a new wording.

Compare appropriate human participants and models under disclosed conditions. Record instructions, training examples, time, context, scratch space, tools, sampling settings, and token budgets. Resources need not be identical, but differences must be explicit. Report distributions, repeated trials, uncertainty, and failures. Choose scoring rules and thresholds in advance. Human performance provides a comparison, not an automatic definition of the correct algorithm.

Scope matters throughout. Kambhampati’s “Stone Soup” discussion explicitly warns that critiques of unaided LLMs do not automatically carry over to augmented systems. That is a useful constraint on both proponents and critics.

For a causal claim, identify the representation or operation your account predicts, intervene where possible, and test the consequences against competing explanations. For an impossibility proof, state the architecture, resource bounds, assumptions, and precise property ruled out. For a bounded performance result, publish the bounded result. There is no need to inflate it to make it valuable.

Respond on GitHub with your definitions of reasoning and understanding; the scope of your claim and exact models or systems; the relevant human evidence and biological mechanism; the corresponding machine mechanism or missing operation; a reproducible protocol, code, prompts, outputs, and controls; and the result that would change your mind. If your claim concerns behavior alone, matched human evidence is enough to start. If biology carries the impossibility argument, the biological explanation needs to carry its weight.

A reproducible, controlled gap would change my view about the capability in question. A valid impossibility argument would change my view within its assumptions. Positive evidence deserves the same discipline. I am inviting anyone to make that case, including people who think my proposed definitions are wrong. Give us definitions we can challenge, mechanisms we can investigate, and predictions that can fail. “It’s just next-token prediction” has had plenty of airtime. Show your work.

Research and drafting assisted by OpenAI Codex.