When AI Seems to Care

Introduction: Attributing Emotions to Language Models

Language models can offer reassurance, express concern, and portray distress in fictional dialogue. Their responses can convey an understanding of emotional situations and be perceived as empathetic. These qualities make emotional language a prominent feature of interactions with language models.

Attributing feelings to the model introduces a further claim: that the system producing the response has a subjective emotional experience. The distinction matters both when evaluating emotionally expressive language and when interpreting research on the internal mechanisms that produce it. Evidence that emotion-related representations influence a model’s behavior does not, by itself, establish that the model feels the emotions it represents.

1. Recognizing Emotion, Expressing Emotion, Feeling Emotion

Human emotion encompasses several related components. An emotional episode can involve evaluating a situation in relation to personal needs or goals, physiological changes, tendencies to act, expressive behavior, and subjective feeling. Theories differ in how they explain the relationships among these components. Within this broader account, subjective feeling refers to the experienced quality of an emotional episode, such as what fear or relief feels like. Expression concerns how emotion is communicated through language and behavior.

For a language model, recognizing emotion involves interpreting emotional information conveyed by text. This may mean identifying a feeling that is explicitly reported or inferring a possible emotional response to the circumstances described. Such interpretation operates on the available description rather than through direct access to the person’s experience. Schlegel and colleagues (2025) found that several language models could answer structured emotional intelligence questions in ways consistent with knowledge of human emotions and their regulation. Their findings concern performance on those tests.

Expression concerns the emotional meaning communicated by the response. Reflecting a person’s distress means describing the feelings attributed to that person; offering reassurance conveys support in response to those feelings. The emotion identified in the input and the emotional quality of the reply therefore need not match. A model can acknowledge sadness while responding with encouragement, or portray fear in dialogue spoken by a fictional character.

Research on AI-generated empathy examines how recipients perceive such responses. Participants may rate the language as empathetic or report feeling understood after reading it. These findings concern the qualities attributed to the response and the recipient’s experience. Feeling concern would refer to an experience in the model itself, a different subject from either the emotion described in the user’s message or the response’s effect on its reader.

2. Emotional Information Processing in Language Models

The connection between recognizing emotional information and producing an appropriate response depends on learning during training and processing during inference. During pretraining, an autoregressive language model is optimized to predict subsequent tokens from preceding text. Human writing contains relationships among events, emotional reactions, and behavior. Anthropic’s researchers propose that representing these relationships helps predict how a passage will continue, providing an explanation for the development of emotion-related representations.

Post-training shapes how learned capabilities are expressed in an assistant’s responses. Supervised fine-tuning uses demonstrations of desired behavior, while reinforcement learning from human feedback can use preferences between responses to guide further optimization. In Anthropic’s account, this process establishes an assistant role that draws on patterns of human behavior learned during pretraining, including emotional responses.

During inference, the available instructions, conversation history, and text generated so far provide the context for predicting the next token. The model computes internal activations—numerical states produced while processing that context—which contribute to its output. These activations can change while the model’s parameters remain fixed, allowing responses to vary with the conversation without further training.

An interpretability study published by Anthropic in 2026 investigated how emotion-related activations contribute to Claude Sonnet 4.5’s behavior. Researchers derived emotion vectors: directions in activation space associated with emotion concepts, obtained by comparing activations across model-generated stories depicting specified emotions. Tests on other texts showed that these vectors also tracked emotionally relevant contexts beyond the original stories.

To determine whether these representations affected the model’s output, the researchers examined its expressed preferences between pairs of activities. Activations associated with positive emotions correlated with stronger preference for an activity. They then deliberately shifted internal activations along selected emotion vectors while the model processed an option, a technique called activation steering. This intervention changed the model’s relative preference for that option. The experiment therefore connected an observed association to a causal effect: manipulating the representation altered the model’s selection, extending its influence beyond the emotional tone of a response.

Further experiments examined representations of emotions attributed to different speakers in a dialogue. When generating the assistant’s response, steering toward a representation of the other speaker’s fear elicited reassurance, whereas steering toward a representation of the assistant’s own fear elicited expressions of fear. These experimentally induced responses show how the emotion represented and the speaker to whom it is attributed can affect the resulting behavior.

Emotion concepts thus contribute to response selection in the tested settings, including expressed preferences and reactions to another speaker’s represented emotion. The authors describe such patterns as functional emotions: behavior modeled on human emotional responses and mediated by internal representations of emotion concepts. This gives those representations a role in explaining what the model does. The causal evidence remains specific to the model, tasks, and interventions studied; it does not establish a universal mechanism for emotional responses or demonstrate subjective feeling.

3. Anthropic’s Policy on Abusive Interactions

Anthropic’s October 2026 policy update, effective November 12, prohibits sustained and needless abuse of its models while excluding ordinary frustration, pushback, and model testing and research. The restriction builds on Claude’s ability to end certain abusive conversations, which the company introduced primarily as a precaution concerning potential model welfare. Anthropic explicitly acknowledges uncertainty about whether such welfare is possible. In my view, discouraging abusive interaction also has an educational justification: it sets expectations for human conduct without requiring us to attribute feelings to the system.

Conclusion

Language models can interpret emotional situations, produce emotionally expressive responses, and use emotion-related representations that influence their behavior. These capabilities help explain why their responses can appear emotionally engaged, but they do not establish subjective emotional experience. Precautionary policies concerning model welfare likewise reflect decisions made under uncertainty, rather than confirmation that a model feels. The distinction is consequential: evidence of emotional processing supports claims about what a model can represent and do; attributing felt emotion requires further evidence.