Memory Architecture for LLM Agents
A technical guide to memory architecture in LLM agents, covering working state, persistent memory, episodic, semantic, and procedural roles, representation, storage, retrieval, updates, and evaluation.
Abstract
Memory in an LLM agent is not a single component. The term is routinely used to describe the context window, conversation history, persistent user data, retrieved documents, model parameters, and external databases. These mechanisms differ in lifetime, representation, access path, and operational role. Treating them as one layer creates category errors: a larger context window is mistaken for long-term memory, a vector index is treated as a complete memory architecture, and stored conversation history is assumed to be useful simply because it exists.
This article presents a vendor-agnostic systems model for agent memory. It distinguishes four separable analytical dimensions: temporal scope, functional role, representation form, and storage and access substrate. It then maps those dimensions onto a memory lifecycle that includes capture, write selection, representation, validation, storage, indexing, retrieval, context assembly, use, update, consolidation, and deletion.
The model is a synthesis of peer-reviewed agent-architecture literature, retrieval-augmented generation research, long-context evaluation, and agent-memory benchmarks. It is not presented as a universal taxonomy. Its purpose is to provide a precise engineering vocabulary for analyzing and designing memory-dependent LLM systems.
Memory is a systems capability, not a single store
An LLM does not acquire durable memory merely because an application sends it a long prompt or preserves a transcript. At the system level, memory requires mechanisms that allow information from an earlier observation, interaction, or task to affect a later inference or action.
A useful systems definition is:
Agent memory is the set of mechanisms used to represent, retain, retrieve, select, and update information so that it can influence future model invocations or agent actions.
This definition separates memory from any one implementation. A memory system may use a relational database, an event log, a document store, a vector index, a knowledge graph, executable procedures, model parameters, or several of these together. The storage technology does not determine the functional type of the memory, and persistence alone does not guarantee that stored information will be retrieved or used correctly.
CoALA provides one of the clearest formal architectures for this distinction. It models a language agent as a system with working memory, optional long-term memory components, a structured action space, and a decision process. Within its long-term memory model, episodic, semantic, and procedural memory serve different functional roles [1].
Why “memory” is overloaded in LLM systems
Several mechanisms are commonly grouped under the same label even though they solve different problems.
| Term | What it describes | What it does not establish |
|---|---|---|
| Context window | The bounded token capacity available to a model invocation, including supplied input and generated output and, where applicable, additional model-side tokens | Persistence across future invocations or sessions |
| Conversation state | A mechanism for carrying prior messages, tool calls, outputs, or other items across turns | A structured memory model, retrieval policy, or update policy |
| Persistent store | Data retained beyond the current invocation or session | Whether the data will be retrieved, selected, or used correctly |
| Retrieved context | Records selected from an external source and inserted into the active model input | That the records are correct, sufficient, current, or trusted |
| Parametric knowledge | Information represented implicitly in learned model parameters | An inspectable record that can be updated or deleted as ordinary application data |
| Compacted or summarized state | A reduced representation of earlier context used to continue a long interaction | A complete source-of-record history or a general long-term memory architecture |
A precise architecture must therefore answer more than “does the system have memory?” It must identify what information is active, what persists, how the information is represented, when it is retrieved, how conflicts are handled, and what changes when a new record supersedes an older one.
A multidimensional model of agent memory
A useful correction to the common “memory layers” diagram is to stop treating short-term, long-term, episodic, semantic, and procedural memory as five parallel categories.
They are better analyzed as separable dimensions that interact in implementation.
Dimension 1: Temporal scope
This dimension describes when information is active or available for reuse.
- Working state contains information active for the current task or decision process. It may persist across several model invocations within the same task.
- Persistent memory contains information retained outside the active working state so that it can be retrieved or reactivated in later task cycles, sessions, or interactions.
Dimension 2: Functional role
This dimension describes what the information represents or how it influences behavior.
- Episodic memory represents events, interactions, trajectories, and outcomes.
- Semantic memory represents facts, concepts, entities, attributes, and relations.
- Procedural memory represents rules, skills, strategies, workflows, or action policies.
In the CoALA framework, episodic, semantic, and procedural memory are long-term memory categories rather than peers of long-term memory itself [1]. When information is retrieved or reactivated from these memory components, the selected representation becomes part of the active working state or becomes accessible to the controller responsible for the current task.
Dimension 3: Representation form
This dimension describes the form in which information is encoded.
- Parametric representation encodes information as distributed patterns in model parameters.
- Textual representation encodes information as messages, documents, notes, summaries, or other text.
- Structured representation encodes information as typed records, entities, attributes, or relations.
- Vector representation encodes information as embeddings used in similarity operations.
- Graph representation encodes information as nodes, edges, and typed relations.
- Executable representation encodes procedures as code, rules, prompts, policies, or workflow definitions.
Dimension 4: Storage and access substrate
This dimension describes where a representation resides and how the system addresses or retrieves it.
Possible substrates include:
- the current model context;
- model parameters;
- relational or key-value databases;
- document stores;
- event stores;
- search indexes;
- vector indexes;
- graph stores;
- file systems;
- code or configuration repositories.
These dimensions should be analyzed separately, although they interact in implementation. A user preference can be persistent in temporal scope, semantic in function, structured in representation, and stored in a relational database. A completed tool trajectory can be episodic, represented as structured events, and stored in an event store. A reusable workflow can be procedural, represented as executable configuration, and stored in a code repository.
The same functional memory type can therefore be instantiated through several representation forms and storage or access substrates.
Working state: the information active now
Working state is the information currently available to the model or controller for the active task. Depending on the architecture, it may include:
- system and developer instructions;
- the current user request;
- selected conversation history;
- active goals and task state;
- retrieved memory records;
- tool results and observations;
- intermediate plans or structured state;
- constraints applied by the application.
The context window is the model-side token capacity available to a model invocation. Anthropic describes the context window as the text a model can reference while generating a response and uses “working memory” as an operational analogy [8]. OpenAI separately documents mechanisms for carrying conversation state across responses, including persistent conversation objects and response chaining [6].
These descriptions support an important architectural distinction: the context window limits the tokenized material available to one model invocation, while working state is the broader system state from which model-facing context is selected and assembled. Working state can span multiple model invocations within a task and may include controller state or tool-accessible information that is not placed verbatim into the model input.
Conversation history may contribute to working state, but the two are not identical. A system can exclude old turns, retrieve only selected episodes, inject structured application state, or reconstruct the active model context from external records. Conversely, a transcript may be preserved in storage without being loaded into the current invocation.
Persistent memory: retained information available for future use
Persistent memory is information retained beyond the current model invocation or active task state. Its persistence horizon may be one workflow, one user session, multiple sessions, a project, an organization, or the lifetime of a system.
Persistence does not imply continuous presence in the context window. In most scalable architectures, persistent information remains outside the active context until a retrieval or application rule selects it. This creates three separate questions:
- Was the information stored?
- Was it retrieved for the current task?
- Did the model or controller use it correctly?
These questions correspond to different failure modes. A correct record can exist but remain undiscovered. A relevant record can be retrieved but ranked below irrelevant material. A selected record can enter the context and still be misread or ignored.
Persistent conversation state is one possible implementation. It can preserve prior messages and tool interactions across sessions, but it does not by itself define semantic structure, retrieval criteria, conflict resolution, retention rules, or memory evaluation. It is a state-continuity mechanism, not a complete memory architecture.
Episodic memory: events, interactions, and outcomes
Episodic memory represents temporally situated experiences. In an agent system, an episode may describe a conversation, a task attempt, a tool-use trajectory, a decision, an observed event, or a result.
A useful episodic record may include:
- event or task identity;
- timestamp or temporal range;
- participating actors or systems;
- relevant context;
- observations;
- actions taken;
- outcome;
- source or provenance;
- evaluation, feedback, or reward signal.
The defining property is not the storage schema. It is the connection between information and a specific event or experience.
Generative Agents provides a concrete research example. Its architecture records observations in a memory stream, retrieves records using recency, importance, and relevance, and generates higher-level reflections that can influence later planning [5]. This is one implementation pattern, not a universal design requirement. It demonstrates that episodic memory involves more than transcript retention: the system needs a method for deciding which past experiences are relevant to the present state.
Semantic memory: facts, concepts, and relations
Semantic memory represents information that is not defined primarily by a single remembered event. Typical examples include:
- user or system attributes;
- stable preferences;
- facts about entities;
- domain concepts;
- relationships between records;
- constraints that remain applicable across tasks.
An episodic record and a semantic record can refer to the same underlying information at different levels. For example, “the user selected option B during the review on 10 July” is episodic. “The user prefers option B for this project” is semantic only if the system has a justified rule for extracting and maintaining that preference.
This distinction matters because event storage does not automatically produce reliable semantic knowledge. Converting repeated episodes into a generalized fact is a consolidation operation. It may require evidence thresholds, conflict handling, temporal qualification, or explicit confirmation.
Semantic memory also should not be described as “meaning understood by the system.” That wording implies a stronger claim than the architecture requires. The system needs a representation that can support later retrieval and computation; the definition does not depend on attributing human-like understanding.
Procedural memory: rules, skills, and ways of acting
Procedural memory represents information governing how an agent performs tasks or selects actions. It may take the form of:
- workflow definitions;
- tool-use procedures;
- decision rules;
- planning strategies;
- prompts or policies that guide behavior;
- executable code;
- learned action patterns encoded in model parameters.
Procedural memory is broader than a fixed list of sequential steps. A procedure can include branching, preconditions, exception handling, loops, tool-selection rules, or policies that map states to actions.
The procedure itself must also be distinguished from its execution. “Call tool A, validate the result, then choose tool B or C” can be procedural memory. A specific run of that procedure, including its inputs and outcome, is an episode. The two may be linked, but they are not the same record.
CoALA explicitly treats procedural memory as a long-term component that can include both implicit knowledge in an LLM and explicit code or policies in the surrounding agent system [1]. This is particularly important in production architectures, where behavior is often distributed across model parameters, prompts, workflow definitions, and controller logic.
Parametric and external memory have different representation and update properties
The original RAG paper uses a distinction between parametric memory and non-parametric memory. In that architecture, the parametric component is a pretrained sequence-to-sequence model, while the non-parametric component is a dense vector index of Wikipedia accessed through a neural retriever [2].
The architecture demonstrates one way to combine information encoded in model parameters with information retrieved from an external system. It does not establish that all external memory must use embeddings or a vector database.
External memory may be implemented with:
- relational tables;
- document collections;
- event streams;
- key-value stores;
- search indexes;
- vector indexes;
- knowledge graphs;
- file systems;
- code or rule repositories.
A vector embedding is a representation used for similarity operations. A vector index is an access structure that supports retrieval over those representations. Neither is, by itself, a complete memory architecture. A production memory design still needs policies for writing, scoping, validating, updating, deleting, retrieving, ranking, and using records.
Parametric memory has different operational properties. Information encoded in model parameters is distributed and is not exposed as a conventional application record. It therefore cannot ordinarily be inspected, versioned, updated, or deleted through the same mechanisms used for database records. Training, fine-tuning, model editing, and external memory writes should be treated as distinct operations with different controls and failure modes.
The memory lifecycle
No single lifecycle is standardized across all agent systems. However, the literature supports a clear separation between storage, retrieval, and downstream use. LongMemEval, for example, decomposes long-term memory design into indexing, retrieval, and reading stages [4]. A broader engineering lifecycle can make the remaining decisions explicit.
1. Capture
The system observes an input, event, tool result, task outcome, or state change that could be useful later.
Capture does not imply persistence. Raw observations may be transient, duplicated, incomplete, or irrelevant to future tasks.
2. Write selection
A write policy determines whether information should become persistent memory.
The decision may consider:
- expected future utility;
- record type;
- scope;
- duplication;
- temporal relevance;
- source quality;
- required retention period;
- whether an existing record should be updated instead.
A system that writes every message or tool result has a history store, but not necessarily a usable memory system. Unbounded accumulation increases retrieval noise and makes stale or contradictory records harder to manage.
3. Representation
Selected information is converted into a representation appropriate to its function and future access path. This may be:
- a raw event;
- a structured record;
- a normalized entity or relation;
- a summary;
- an embedding;
- a procedure;
- an update to model parameters.
Representation is not neutral. Summarization can remove detail. Entity extraction can introduce errors. A semantic fact inferred from several episodes can overgeneralize. The architecture should therefore distinguish source records from derived records.
4. Validation and provenance
Before or after persistence, the system may validate the record and attach metadata such as:
- source;
- creation time;
- effective time;
- scope;
- version;
- confidence or status;
- relationship to earlier records.
There is no universal schema for this metadata. The architectural requirement is that the system retain enough information to interpret and update the record correctly.
5. Storage and indexing
Storage preserves the record. Indexing creates one or more paths for finding it.
These are separate operations. A record may be stored in a database and indexed by entity identifier, time, keywords, graph relations, embeddings, or several signals at once. Similarity search is one retrieval method among many.
6. Retrieval and ranking
At a later task, the system generates candidate records and ranks them for relevance. Retrieval may use:
- entity identity;
- lexical matching;
- semantic similarity;
- temporal constraints;
- recency;
- importance;
- task type;
- record status;
- combinations of structured and unstructured signals.
Retrieval quality depends on the query, index, metadata, ranking function, and the completeness of the stored representation. The existence of a relevant record does not guarantee that the system will retrieve it.
7. Context assembly and reading
Selected records are inserted into the working state or exposed through a tool. The model or controller must then interpret them in relation to the current task.
This stage is distinct from retrieval. LongMemEval separates retrieval from reading because a system can surface the correct evidence and still answer incorrectly [4]. Long-context research reinforces the same point: information can be present in the input without being used robustly [3].
8. Use
The active system uses the selected information to generate a response, continue a task, choose an action, personalize behavior, or resolve a current decision.
The effect of memory should be observable at the task level. A record that is stored and retrieved but does not change the relevant decision has not provided operational value for that task.
9. Update and consolidation
New observations may confirm, refine, contradict, or supersede existing records. The architecture needs explicit behavior for:
- replacing outdated values;
- preserving temporal history;
- merging duplicates;
- retaining conflicting claims;
- deriving semantic records from repeated episodes;
- revising a procedure after evaluated outcomes.
Update logic is part of the memory model. Appending a new record without defining its relationship to existing records can leave the system with several incompatible versions of the same fact.
10. Expiration and deletion
Some records should stop influencing future behavior. The system may expire records by time, scope, status, explicit removal, or a domain-specific retention rule.
Deletion is not equivalent to excluding a record from one prompt. A complete design must identify which representations, indexes, summaries, and derived records are affected when a memory is removed or invalidated.
Reference flow
The lifecycle can be represented as a system flow rather than as five parallel “memory layers”:
Current inputs ───────────────────┐
Instructions ─────────────────────┤
Tool results ─────────────────────┤
Application task state ───────────┤
Retrieved memory records ─────────┤
▼
Working state
│
│ selected tokenized material
▼
Model context
│
▼
Inference / decision / action
│
▼
Outcomes and feedback
│
▼
capture → select → represent → validate
│
▼
store and create indexes
│
▼
Persistent memory records
├─ Episodic
├─ Semantic
└─ Procedural
│
▼
retrieve → rank → select
│
└──────────────► Retrieved memory records
Storage and access implementations:
model parameters · databases · documents · event stores
search indexes · vector indexes · graph stores
file systems · code repositories
Cross-cutting operations:
update · consolidate · supersede · expire · delete
The diagram separates five questions that are often collapsed:
- What information is active in the current working state?
- What information persists beyond that active state?
- What functional role does each memory record serve?
- How is the information represented, stored, and accessed?
- What operations connect retained information to future behavior?
Long context is not long-term memory
A larger context window increases the amount of information that can be supplied to a model in one invocation. It does not create a durable memory lifecycle.
The distinction is empirical as well as conceptual. Lost in the Middle found that model performance could change substantially with the position of relevant information in a long input. Performance was often strongest when relevant evidence appeared near the beginning or end and weaker when it appeared in the middle [3]. The result shows that availability inside the context does not guarantee robust use.
Long-context systems therefore still need selection and organization. Passing an entire transcript can consume capacity while making the relevant evidence harder to identify. For many tasks, a smaller, well-constructed working state may be more useful than an indiscriminately larger one.
Conversation state and compaction are context-management mechanisms
Conversation-state APIs can preserve messages, tool calls, tool outputs, and other items across turns. OpenAI, for example, documents persistent conversation objects and response chaining as mechanisms for sharing context across responses [6]. This continuity can support memory-dependent applications, but it does not decide what should be retained as durable semantic, episodic, or procedural memory.
Compaction addresses a different problem: how to continue a long interaction when accumulated context approaches a model limit. OpenAI documents compaction items that carry forward selected prior state in fewer tokens [7]. Anthropic documents server-side compaction that summarizes earlier conversation content so an interaction can continue [8].
Compacted state should therefore be treated as a derived context representation, not automatically as the authoritative memory store. When exact history, provenance, or reversible updates matter, the source records and the compacted representation serve different purposes.
Memory failures occur at different stages
Treating memory as a lifecycle makes diagnosis more precise.
Write failure
The system does not store information that should persist, or stores an incorrect interpretation of the source event.
Representation failure
The stored record loses necessary detail, encodes the wrong entity or time, or turns a conditional observation into an unconditional fact.
Retrieval failure
A relevant record exists but is not returned for the current task.
Ranking failure
The correct record is retrieved but displaced by less relevant or stale records.
Reading failure
The correct evidence is present in the working state, but the model or controller interprets it incorrectly or fails to use it.
Update failure
A newer record does not supersede an older one, or conflicting records remain unresolved without temporal or status information.
Temporal failure
The system remembers the content but applies it to the wrong time period or fails to reason about sequence and change.
Abstention failure
The required information is absent, but the system answers as though it had a supporting memory.
LongMemEval directly evaluates information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention [4]. These categories demonstrate why a single retrieval-recall score is insufficient for evaluating end-to-end memory behavior.
Architecture decisions that must be explicit
A professional memory architecture should define the following decisions before selecting a database or retrieval framework.
| Decision | Required question |
|---|---|
| Write policy | Which observations become persistent records, and which remain transient? |
| Memory type | Is the record episodic, semantic, procedural, or a combination represented through linked records? |
| Scope | Does the record apply to one task, session, user, project, tenant, or system? |
| Granularity | Is the source preserved as a full event, a structured fact, a summary, or several linked representations? |
| Storage model | Which substrate supports the required queries, updates, provenance, and retention behavior? |
| Indexing | Which access paths are required: identifiers, time, text, embeddings, graph relations, or hybrid indexes? |
| Retrieval trigger | When should memory retrieval occur, and what should form the query? |
| Ranking and selection | How are relevance, recency, status, and task fit combined under a context budget? |
| Conflict handling | How are contradictory, duplicate, or superseded records represented? |
| Consolidation | When may episodes produce or revise semantic knowledge or procedures? |
| Expiration and deletion | When should a record stop influencing behavior, and which derived representations must also change? |
| Evaluation | How will the system test storage, retrieval, reading, temporal correctness, updates, and abstention? |
These decisions are logically prior to implementation. Selecting a vector database first answers only part of the storage-and-indexing question.
Evaluating an agent memory system
Memory evaluation should separate component performance from end-to-end task performance.
Component-level evaluation
Engineering metrics may include:
- write accuracy;
- duplicate and contradiction rates;
- retrieval recall and precision;
- ranking quality;
- record freshness;
- retrieval latency;
- storage growth;
- context-token cost;
- provenance retention.
These metrics identify where the pipeline fails, but they do not establish that the agent uses memory correctly.
End-to-end evaluation
LongMemEval provides a stronger behavioral frame by testing five capabilities [4]:
- information extraction;
- multi-session reasoning;
- temporal reasoning;
- knowledge updates;
- abstention when the required information is unavailable.
A complete evaluation should also test whether the system behaves correctly when records conflict, when relevant evidence is mixed with distractors, and when a stored item is no longer valid. The benchmark design should match the intended memory scope rather than assuming that generic conversational recall is sufficient for every agent.
The key distinction is:
Retrieval success is not equivalent to memory success.
The system succeeds only when the right information is retained, retrieved, interpreted, and applied to the correct task and time.
Common category errors
Context window = memory
A context window is bounded model input capacity. It can hold working state, but it does not define persistence, retrieval, update, or deletion.
Conversation history = memory architecture
A transcript is one possible source of episodic records. Without selection, structure, retrieval, and update rules, it remains history rather than a complete memory system.
RAG = long-term memory
RAG is a family of retrieval-augmented generation architectures. It can provide access to external information, but it does not by itself define memory writes, temporal updates, consolidation, or retention.
Vector database = memory
A vector database supports a storage and similarity-retrieval pattern. It does not determine which records should exist, what they mean, how they change, or how the agent should use them.
Retrieval = use
A retrieved record may be ignored, misread, or outweighed by other context. Retrieval and reading must be evaluated separately.
Procedure execution = procedural memory
Procedural memory describes the reusable rule, skill, or policy. A specific execution and its outcome are episodic records.
Storing everything = learning
Persistence is not equivalent to learning. A system learns operationally only when stored or updated information changes future behavior through a defined access and use mechanism.
Persistence = relevance
A record can remain stored after it becomes stale, superseded, or irrelevant. Retention and selection are different policies.
Conclusion
Memory architecture in LLM agents is best understood as a coordinated information system rather than a single store or a fixed stack of layers.
Working state and persistent memory describe temporal availability. Episodic, semantic, and procedural memory describe functional roles. Parametric, textual, structured, vector, graph, and executable representations describe implementation substrates. A complete architecture connects these dimensions through explicit write, retrieval, selection, update, consolidation, and deletion operations.
This model also clarifies what memory is not. It is not synonymous with a context window, a transcript, a vector database, or RAG. Each can participate in a memory system, but none provides the entire lifecycle.
The practical standard is therefore not whether an agent can claim to “remember.” It is whether the system can explain, test, and control:
- what was retained;
- how it was represented;
- why it was retrieved;
- whether it was used correctly;
- how it changes when newer information arrives;
- and when it should stop influencing future behavior.
Suggested reading
- Parallel Exploration in LLM Systems Is an Orchestration Pattern
- LLM-Led vs Orchestrator-Led Tool Execution
- File Upload Is Not Full-File Review
References
-
Sumers TR, Yao S, Narasimhan K, Griffiths TL. Cognitive Architectures for Language Agents. Transactions on Machine Learning Research. 2024.
https://openreview.net/forum?id=1i6ZCvflQJ -
Lewis P, Perez E, Piktus A, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33. 2020.
https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html -
Liu NF, Lin K, Hewitt J, et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024;12:157–173. DOI: 10.1162/tacl_a_00638.
https://aclanthology.org/2024.tacl-1.9/ -
Wu D, Wang H, Yu W, Zhang Y, Chang K-W, Yu D. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. International Conference on Learning Representations. 2025.
https://openreview.net/forum?id=pZiyCaVuti -
Park JS, O’Brien JC, Cai CJ, Morris MR, Liang P, Bernstein MS. Generative Agents: Interactive Simulacra of Human Behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 2023. DOI: 10.1145/3586183.3606763.
https://dl.acm.org/doi/10.1145/3586183.3606763 -
OpenAI. Conversation State — OpenAI API Documentation.
https://developers.openai.com/api/docs/guides/conversation-state -
OpenAI. Compaction — OpenAI API Documentation.
https://developers.openai.com/api/docs/guides/compaction -
Anthropic. Context Windows — Claude Platform Documentation.
https://platform.claude.com/docs/en/build-with-claude/context-windows