When AI Seems to Care
Why AI can seem caring: emotional language, emotion-related representations, and the distinction between model behavior and subjective feeling.
Introduction: Attributing Emotions to Language Models
Language models can offer reassurance, express concern, and portray distress in fictional dialogue. Their responses can convey an understanding of emotional situations and be perceived as empathetic. These qualities make emotional language a prominent feature of interactions with language models.
Attributing feelings to the model introduces a further claim: that the system producing the response has a subjective emotional experience. The distinction matters both when evaluating emotionally expressive language and when interpreting research on the internal mechanisms that produce it. Evidence that emotion-related representations influence a model’s behavior does not, by itself, establish that the model feels the emotions it represents.
1. Recognizing Emotion, Expressing Emotion, Feeling Emotion
Human emotion encompasses several related components. An emotional episode can involve evaluating a situation in relation to personal needs or goals, physiological changes, tendencies to act, expressive behavior, and subjective feeling. Theories differ in how they explain the relationships among these components. Within this broader account, subjective feeling refers to the experienced quality of an emotional episode, such as what fear or relief feels like. Expression concerns how emotion is communicated through language and behavior.
For a language model, recognizing emotion involves interpreting emotional information conveyed by text. This may mean identifying a feeling that is explicitly reported or inferring a possible emotional response to the circumstances described. Such interpretation operates on the available description rather than through direct access to the person’s experience. Schlegel and colleagues (2025) found that several language models could answer structured emotional intelligence questions in ways consistent with knowledge of human emotions and their regulation. Their findings concern performance on those tests.
Expression concerns the emotional meaning communicated by the response. Reflecting a person’s distress means describing the feelings attributed to that person; offering reassurance conveys support in response to those feelings. The emotion identified in the input and the emotional quality of the reply therefore need not match. A model can acknowledge sadness while responding with encouragement, or portray fear in dialogue spoken by a fictional character.
Research on AI-generated empathy examines how recipients perceive such responses. Participants may rate the language as empathetic or report feeling understood after reading it. These findings concern the qualities attributed to the response and the recipient’s experience. Feeling concern would refer to an experience in the model itself, a different subject from either the emotion described in the user’s message or the response’s effect on its reader.
2. Emotional Information Processing in Language Models
Training and Inference
During pretraining, an autoregressive language model learns to predict the next token from preceding text. Human writing contains patterns linking events, emotional reactions, and behavior. Anthropic’s researchers propose that representing these patterns helps the model predict what comes next. This offers an explanation for how emotion-related representations develop.
Post-training shapes how learned capabilities appear in an assistant’s responses. Supervised fine-tuning uses demonstrations of desired behavior. Reinforcement learning from human feedback can use preferences between responses to guide further optimization. In Anthropic’s account, post-training establishes an assistant role that draws on patterns of human behavior learned during pretraining, including emotional responses.
During inference, the model predicts the next token based on the available instructions, conversation history, and text it has generated so far. As the model processes that context, it computes internal activations, numerical states that contribute to its output. Changes in context can alter these activations and the resulting response while the model’s parameters remain fixed.
Emotion Vectors and Model Behavior
Anthropic’s 2026 interpretability study examined how emotion-related activations contribute to the behavior of Claude Sonnet 4.5. The researchers compared activations across model-generated stories depicting specific emotions. From those comparisons, they derived emotion vectors, directions in activation space associated with emotion concepts. In tests on other texts, the vectors also tracked emotionally relevant contexts beyond the original stories.
In one experiment, the researchers examined the model’s expressed preferences between pairs of activities. Activations associated with positive emotions correlated with a stronger preference for an activity.
To test for a causal effect, the researchers used activation steering. They deliberately shifted the model’s activations along selected emotion vectors while it processed an option. The intervention changed the model’s relative preference for that option, showing that the effect extended to what the model selected.
A separate set of experiments examined how the model responded to fear attributed to different speakers in a dialogue. When generating the assistant’s reply, steering toward a representation of the other speaker’s fear elicited reassurance. Steering toward a representation of the assistant’s own fear elicited expressions of fear. The same emotion concept produced different responses depending on whose fear the model represented.
The authors describe these patterns as functional emotions. The term refers to behavior modeled on human emotional responses and mediated by internal representations of emotion concepts. In the experiments, these representations help explain both the model’s expressed preferences and its responses to fear attributed to different speakers.
The causal evidence is specific to the model, tasks, and interventions studied. Establishing a general mechanism for emotional responses or attributing subjective feeling to the model would require further evidence.
3. Anthropic’s Policy on Abusive Interactions
The possibility of model welfare has shaped how Anthropic handles abusive interactions. The company gave Claude the ability to end certain abusive conversations primarily as a precaution, while explicitly acknowledging that it remains uncertain whether models can have welfare.
Anthropic’s October 2026 policy update builds on that feature. Effective November 12, it prohibits sustained and needless abuse of its models while excluding ordinary frustration, pushback, and model testing and research.
In my view, discouraging abusive interaction also has an educational justification. It sets expectations for how people conduct themselves even when the question of what the model experiences remains open.
Conclusion
Language models can interpret emotional situations, produce emotionally expressive responses, and use emotion-related representations that influence their behavior. These capabilities help explain why their responses can appear emotionally engaged, but they do not establish subjective emotional experience. Precautionary policies concerning model welfare likewise reflect decisions made under uncertainty, rather than confirmation that a model feels. The distinction is consequential: evidence of emotional processing supports claims about what a model can represent and do; attributing felt emotion requires further evidence.
References
- Schlegel, K., et al. (2025). Large language models are proficient in solving and creating emotional intelligence tests. Communications Psychology.
- Yin, Y., Jia, N., & Wakslak, C. J. (2024). AI can help people feel heard, but an AI label diminishes this impact. Proceedings of the National Academy of Sciences.
- Sofroniew, N., et al. (2026). Emotion Concepts and their Function in a Large Language Model. arXiv preprint.
- Anthropic. (2026, October 8). 2026 Usage Policy update.
- Anthropic. (2025, August 15). Claude Opus 4 and 4.1 can now end a rare subset of conversations.