Theory of Mind in LLMs: How Models Track Beliefs, Intentions, and Perspectives

By Published Updated

How LLMs represent agent-specific knowledge, beliefs, intentions, emotions, and perspectives—and when those inferences affect interpretation, prediction, and interaction.

Introduction

Humans rarely interpret behavior from visible facts alone.

If a person searches the wrong drawer, we do not necessarily conclude that the action is irrational. We ask what the person last saw, what they were told, what they believe, and what they are trying to find. If someone says, “It is getting late,” we may interpret the sentence as more than a statement about time. Depending on the speaker and the shared context, it may function as a request to leave, a warning to finish, or an attempt to end the conversation.

Theory of Mind is the capacity to attribute mental states—such as knowledge, beliefs, desires, intentions, and emotions—to oneself and others, and to use those attributions to interpret or predict behavior. The state is inferred from what an agent perceived, learned, said, did, or failed to learn. This follows the problem introduced by Premack and Woodruff: behavior becomes intelligible through states attributed to a particular mind.

Under defined prompts and evaluation conditions, LLMs can reproduce parts of this pattern in language. A model may distinguish what different people know, represent a belief that conflicts with reality, infer a goal from an action, explain an emotion from a person’s perspective, or recover intended meaning that is not stated literally. The output can therefore depend not only on the facts available in the prompt, but also on what a particular agent takes to be true.

These behaviors follow a general dependency:

  1. determine what information was available to each agent;
  2. infer an agent-specific mental state from that information and from the agent’s behavior;
  3. use the attributed state to interpret meaning, explain conduct, predict an action, make a judgment, or adapt within an interaction.

This article maps the main forms of that functional resemblance and the evidence behind them. Behavioral similarity does not establish that an LLM uses the same cognitive mechanisms as a human or subjectively experiences beliefs, intentions, or emotions. The relevant question here is narrower: when does a model’s output depend on a mental state attributed to a specific agent, and what does the model do with that attribution?

1. What counts as Theory of Mind-like behavior in language

Theory of Mind is not a single task. A systematic review by Beaudoin and colleagues identified 220 measures across 830 studies of young children, covering seven categories: percepts, knowledge, beliefs, desires, intentions, emotions, and mentalistic understanding of non-literal communication. The categories are not interchangeable, but they provide a useful coverage map for language-model behavior.

An output is not Theory of Mind-relevant merely because it contains words such as believes, wants, or feels. The attribution must be tied to a particular agent, derived from evidence about that agent’s perspective, and used in a subsequent inference.

Functional form Agent-specific distinction Consequence for the output
Information-access tracking Who saw, heard, or learned what Determines who can know or answer something
Belief attribution What an agent takes to be true Explains or predicts action when belief differs from reality
Desire and goal inference What outcome an agent prefers or pursues Explains choices and constrains likely next actions
Intention inference What action or plan an agent means to carry out Separates intended conduct from outcome or side effect
Perspective-dependent emotion How an agent evaluates an event Explains why the same event can produce different emotions
Pragmatic interpretation What a speaker intends a listener to understand Recovers hints, irony, deception, or accidental offense
Interaction adaptation What a partner knows or is likely to do Changes the model’s own strategy within the interaction

The categories can combine in one inference. Predicting an action may require the model to represent what an agent saw, consequently believes, and wants. The map identifies which agent-specific distinctions the response depends on; it does not imply one mechanism or level of reliability across categories.

2. Tracking information access: who saw, heard, or learned what

Before attributing a belief, a model often has to determine what information was available to the relevant agent.

This differs from retrieving a fact from an omniscient context. The model can read the complete prompt, but the people described inside it may have access to different parts of the event. Perspective-sensitive reasoning requires the model to avoid assigning every known fact to every character.

The model may need to track:

  • who was present when a fact was introduced;
  • who left before an update occurred;
  • who later received the update;
  • whether a person can answer from their own information;
  • whether participants now hold different knowledge states about the same event.

FANToM tests this problem in information-asymmetric, multi-party conversations. Participants enter and leave a discussion, and multiple question types probe the same underlying situation: what a person believes, who can answer a factual question, and who had access to the relevant information. Asking related questions makes it possible to test whether the model’s answers form a consistent account rather than a set of locally plausible responses.

In the reported experiments, state-of-the-art models performed significantly below humans overall. The paper also found cases in which models answered a belief question more successfully than the access and answerability questions that should support it. A correct belief choice therefore did not always correspond to a consistent representation of who knew what.

SimToM isolates the perspective-taking step more directly. Its first stage filters the narrative according to what the target character knows; the second answers the mental-state question from that restricted context. This two-stage prompting procedure substantially improved performance over the compared methods without additional training.

The result is operational rather than mechanistic: explicitly separating a character’s accessible information from the model’s full context can improve later mental-state reasoning. If that separation fails, the model may construct a plausible belief, emotion, or intention from evidence the agent never possessed.

3. Representing beliefs that diverge from reality

Belief reasoning becomes visible when the model must preserve a difference between the state of the world and the state attributed to an agent.

If a person sees an object in one location, leaves, and misses its relocation, three variables matter:

  • where the object actually is;
  • where the absent person believes it is;
  • what that person is likely to do while acting on the belief.

Generating the sentence “the person has a false belief” is not enough. The attributed belief must constrain the explanation or prediction. A response that assigns the old location to the agent but predicts an immediate search in the unseen new location has labeled the belief without applying it.

BigToM represents this dependency through causal links among percepts, beliefs, desires, and actions. Its evaluations include forward inference from perception to belief, forward inference from perception and desire to action, and backward inference from observed action toward unobserved beliefs or percepts. The authors reported that GPT-4 mirrored human inference patterns across parts of this structure, but less reliably, while the other evaluated models struggled more.

Strachan and colleagues compared GPT-4, GPT-3.5, and LLaMA2-70B with human participants across false-belief, misdirection, irony, indirect-request, strange-story, and faux-pas tasks. GPT-4 matched or exceeded the human comparison group in several tested conditions, including false belief and misdirection, but the models showed different failure patterns across tasks and formulations.

These studies show that some evaluated models can separate an attributed belief from known reality and sometimes use it in further inference. They do not establish a persistent person model across arbitrary prompts or extended interactions.

Higher-order cases—such as what one person believes about another person’s belief—introduce recursive nesting and additional scope-control demands. Those structures are covered separately in Orders of Intentionality in LLM Evaluation. The basic dependency here is first-order: what an agent encountered can determine what the model predicts that agent will believe and do.

4. Inferring desires, goals, and intentions from behavior

Beliefs describe what an agent takes to be true. They do not, by themselves, explain why the agent acts. Prediction also depends on the outcome the agent values and the action the agent is trying to perform.

For this article, the following are working distinctions rather than universal definitions:

  • Desire: a state or outcome the agent prefers.
  • Goal: an outcome the agent is pursuing in the current context.
  • Intention: an action or plan the agent means to carry out in pursuit of a goal.

The distinctions prevent three common collapses. Causing an outcome does not prove that the outcome was intended. Performing the same action does not imply sharing the same goal. Failing to achieve a result does not erase the intention revealed by the attempt.

In language, a model may infer these states from actions, constraints, preferences, and dialogue. That can allow it to identify an unstated objective, explain why one option was chosen, distinguish an intended result from an accidental consequence, or predict a next action from the combination of belief and goal.

OpenToM extends evaluation beyond short, location-based false-belief stories. Its narratives include explicit personality traits, actions triggered by intentions, and questions about mental states in both the physical and psychological world. The evaluated models performed well on some physical-state questions but remained substantially weaker on psychological-state tracking.

NegotiationToM moves desire, belief, and intention inference into real negotiation conversations using a belief-desire-intention framework. The tested state-of-the-art models performed significantly below humans, including with chain-of-thought prompting.

A short narrative can expose the causal link between a preference and an action. In negotiation, goals may be concealed, priorities may compete, and an utterance can both reveal information and influence the other side. Success in a constrained story therefore does not imply equally strong intention inference in strategic interaction.

5. Inferring emotion through perspective, not only tone

Emotion classification and affective Theory of Mind are related but distinct.

A model can label a message as angry from explicit words, punctuation, or other textual signals without reconstructing a person’s perspective. That process is examined in How AI Tools Read Emotional Signals in Text. The Theory of Mind-relevant case begins when the model must infer how a particular person is likely to feel from what that person believes, wanted, expected, or understood about an event.

The same event can produce different emotions because agents appraise it differently. An outcome can advance one person’s goal and block another’s. An action can produce relief when its protective intention is known, or resentment when only its immediate effect is visible. Perspective-dependent emotion inference may therefore require a combination of the agent’s belief, desired outcome, interpretation of another person’s intention, and personal significance of the event.

OpenToM supplies the more direct evidence within a Theory of Mind setting. Its stories separate physical facts from psychological states and test reasoning about attitudes and emotions in relation to character traits, preferences, intentions, and observed actions. The reported weakness of evaluated models on psychological-state tracking includes the kind of perspective-sensitive inference that simple tone classification does not test.

Zhan and colleagues provide complementary evidence about cognitive appraisal. Their dataset contains 241 Reddit posts annotated across 24 appraisal dimensions, each with a natural-language rationale. This is not an interpersonal Theory of Mind benchmark: the model assesses the writer’s appraisal from the post. Its narrower relevance is that emotion-related inference can be organized around subjective evaluation rather than only an emotion label.

This supports a narrower claim than saying that LLMs generally “understand emotion”: some evaluated models can infer appraisals and reason about psychological states tied to an agent’s perspective.

6. Interpreting what speakers mean, not only what they say

Communication often contains a gap between literal content and intended meaning.

A request can be phrased as an observation. Irony can communicate the opposite of the literal sentence. Deception involves presenting information the speaker takes to be false, often to change what a listener believes. A faux pas can cause harm because the speaker lacks information that the listener possesses.

These cases can require several agent-specific inferences at once:

  • what the speaker believes;
  • what the speaker wants the listener to believe or do;
  • what the speaker expects the listener to infer;
  • what the listener knows that the speaker does not;
  • whether the social effect was intended.

This is the intersection of Theory of Mind and pragmatics. The model must infer the communicative action performed through an utterance, not only its dictionary meaning.

The results from Strachan and colleagues show why “understands non-literal language” is too broad. GPT-4 exceeded the human comparison group on hinting and irony in the reported battery, while the other models produced different patterns. The original faux-pas formulation initially produced a different result: GPT-4 and GPT-3.5 did not reliably commit to the expected conclusion that the speaker lacked the relevant knowledge.

The study’s control experiments changed that interpretation. When the question asked which explanation was more likely, GPT-4 identified the speaker’s ignorance and the unintentional offense. Further variants showed that its responses changed when the story implied that the speaker knew. The authors concluded that the original error reflected an overly conservative response policy rather than a failure to generate the relevant mental-state inference.

The result does not establish general faux-pas comprehension. It shows that question framing can determine whether an available inference becomes the explicit answer. It also anticipates a broader distinction: attributing a mental state and using it correctly are not the same capability.

7. From mental-state inference to mental-state use

The stronger functional test is not whether a model can state what someone knows or believes. It is whether that attribution changes a downstream explanation, prediction, judgment, or interaction strategy.

7.1 Applied inference

SimpleToM tests this transition directly. Each short scenario contains an information asymmetry and three questions about the same person:

  1. Mental-state inference: Is the person aware of the relevant fact?
  2. Behavior prediction: What is the person likely to do next?
  3. Behavior judgment: Given what the person knew, was an observed action reasonable?

Across the 22 evaluated models, recent frontier models often performed very well on the explicit awareness question. That success did not consistently transfer to applied questions. For several larger models, behavior-prediction accuracy was at least 30 percentage points lower than mental-state accuracy, and judgment remained difficult. The scenarios were unchanged; what changed was whether the model only had to identify the mental state or use it to reason about conduct.

The gap exposes a failure mode that a belief label alone cannot reveal. A model may correctly state that a person does not know about a hidden defect and still predict behavior as though the person had access to it. It may identify the belief correctly but judge the resulting action against the model’s own information rather than the agent’s information.

SimpleToM therefore provides direct evidence that explicit mental-state inference can be decoupled from application. BigToM and FANToM show related consistency problems across causal and access questions; SimpleToM isolates the downstream transition itself.

7.2 Adapting within an interaction

Using a partner model while participating in an interaction is a further requirement. It asks the model to update its own strategy in response to another agent’s behavior, not only to answer a static question about that agent.

Riemer and colleagues call this functional Theory of Mind in their evaluation framework: adapting in context to a new partner through a rational response to that partner’s behavior. They contrast it with literal Theory of Mind, defined there as predicting another agent’s behavior.

Many open-source models in their experiments showed strong literal performance but struggled to adapt even to simple partner policies. The difficulty increased across longer interaction horizons. The result bounds the inference that can be drawn from a static prediction task: correctly forecasting what another agent will do does not establish that the model will coordinate with that agent consistently over several turns.

Explanation, prediction, judgment, and interaction adaptation are distinct downstream operations. Evidence for one does not establish the others.

8. What the current evidence establishes

8.1 Behavioral evidence

Across the cited studies, specific models under specific conditions have distinguished agents’ information access, represented beliefs that conflict with reality, inferred goals and intentions, reasoned about perspective-dependent psychological states, and interpreted some forms of indirect or non-literal language. Other studies show that these attributions can affect action prediction, behavior judgment, and adaptation to a partner.

The evidence also shows fragmentation. A model can answer a belief question while contradicting the access conditions required for that belief. It can infer awareness but fail to apply it to behavior. It can identify an accidental offense under one question framing but avoid the same conclusion under another. It can predict a partner in a static task and still fail to adapt over time.

The defensible unit of analysis is therefore not “LLMs have Theory of Mind.” It is the relation among a particular model, context, agent-specific state, downstream use, and evaluation condition.

8.2 Evidence beyond output behavior

Most studies evaluate generated answers. Mechanistic studies ask whether model activations contain structured information about attributed mental states and whether interventions on that information change behavior.

Zhu and colleagues reported that belief status from the perspectives of self and others could be linearly decoded from language-model activations. Manipulating those representations produced substantial changes in Theory of Mind task performance, and the findings extended across several social-reasoning tasks with different causal structures.

Bortoletto and colleagues examined belief representations across model scales, training regimens, and prompts, with control tasks intended to exclude confounds. They found structured representations that improved with scale and fine-tuning but remained brittle to prompt variation. Targeted activation edits corrected some wrong Theory of Mind inferences.

These two studies concern belief representation in particular. They do not establish that the same mechanism supports emotion, intention, pragmatic interpretation, and partner adaptation.

Tsvilodub and colleagues address part of that broader question. They combined behavioral evaluation with causal-mechanistic functional localization across seven Theory of Mind categories and pragmatic reasoning. Their hypothesis-driven tests produced what the authors describe as suggestive evidence for functional integration: some mental-state and pragmatic functions may recruit overlapping, repurposed computational mechanisms. This is preliminary evidence of connection, not proof of one unified social-reasoning system.

Mechanistic results extend the evidence beyond output matching by associating some behavior with intervention-relevant internal representations. They do not establish equivalence to human cognition or subjective mental states.

8.3 The supported boundary

Current evidence supports the following claim:

Under defined conditions, LLMs can exhibit multiple forms of agent-specific mental-state reasoning and sometimes use those inferences to interpret language, explain or predict behavior, make judgments, or adapt to a partner. Preliminary mechanistic evidence suggests that some forms may share functionally relevant representations.

Current evidence does not, by itself, establish that:

  • every LLM has the same capabilities;
  • the behavior is reliable across prompts, contexts, and task formats;
  • a correct one-turn answer reflects a stable model of a person;
  • success in one mental-state category transfers to another;
  • the underlying representations or processes are the same as those used by humans;
  • the model subjectively experiences beliefs, desires, intentions, or emotions.

These limits do not remove the behavioral phenomenon. They define the level at which it can be described.

Conclusion

Theory of Mind-like behavior in an LLM appears when the model stops treating a prompt as one shared pool of facts and reconstructs a perspective tied to a particular agent.

The model may track what the agent encountered, infer what the agent consequently knows, believes, wants, intends, or feels, and use that attribution to interpret language or behavior. Research provides behavioral evidence for several parts of this process and limited mechanistic evidence for internal representations that affect performance. It also shows that inference, application, and interaction adaptation can separate.

The resemblance is therefore functional and conditional, not evidence that an LLM possesses a human mind.

The resemblance to Theory of Mind appears in the transition from what an agent encountered to what the model predicts that agent will know, believe, mean, feel, or do.

Continue exploring mental-state reasoning

Connect functional Theory of Mind-like behavior with recursive mental-state attribution, false-premise jailbreaks, and evidence-gated research review.

References