LLM Memory Boundaries

By Published Updated

A technical model of LLM memory boundaries across state, persistence, retrieval, selection, context assembly, compaction, and lifecycle management.

This article is an explanation of LLM memory as an application-architecture problem.
Procedure/checklist: Manage: LLM memory boundaries

Executive summary

An LLM memory system is not a single store, feature, or context window. It is a set of application-level mechanisms that retain state, retrieve relevant records, transform them when necessary, and assemble a bounded context for a model invocation.

The central architectural distinction is between state that exists in the system and state that is visible to the model for a specific request.

A durable record may exist without being included in the current model input. A conversation may be persisted without every prior turn being replayed. A retrieved item may be excluded during selection. A large context window may hold more input, but it does not independently define what should persist across sessions.

This article models LLM memory through five boundaries:

  1. Invocation boundary: what the model can reference during one inference call.
  2. Persistence boundary: what crosses from temporary state into durable storage.
  3. Retrieval boundary: what stored state becomes a candidate for the current task.
  4. Context-assembly boundary: what candidate state is transformed and included in the model input.
  5. Update boundary: how new events revise, supersede, expire, or remove stored state.

These boundaries make it possible to reason about memory behavior without treating conversation history, persistent memory, retrieval, long context, and caching as interchangeable mechanisms.

Scope and evidence boundary

This article focuses on the architecture of memory-enabled LLM applications and agents. It covers state categories, persistence, retrieval, selection, context assembly, compaction, maintenance, observability, and evaluation.

The analysis is limited to state architecture and the path by which retained information becomes model-visible context.

The boundary model presented here is an engineering synthesis, not a formal industry standard. Vendor-specific statements are limited to official documentation, while the broader architecture is grounded in published work on modular agent memory, hierarchical memory management, retrieval-based agent memory, and long-term memory evaluation.

The core distinction: memory state is not model context

A model invocation operates on a bounded input. Depending on the implementation, that input may include instructions, the current user request, prior conversation items, retrieved records, tool results, summaries, and other application-provided state.

The context window defines how much information can participate in one invocation. Anthropic describes the context window as the text the model can reference while generating a response. OpenAI similarly defines it as the maximum token budget for a request, including input, output, and reasoning tokens. Google describes long context as the information passed to a model for a particular generation request.

Persistent memory is different. It is state retained outside the active invocation so that the application can reuse it later.

The relationship is therefore:

stored state
    -> retrieval
    -> candidate records
    -> selection and transformation
    -> invocation context
    -> model output
    -> optional state update

Only the information that crosses the context-assembly boundary is directly available to the model during that invocation.

A state model for LLM applications

The term “memory” is often used for several mechanisms with different lifetimes and responsibilities. A precise architecture separates them.

State category Architectural role Typical lifetime Directly included in a model request?
Model parameters Learned capabilities and representations encoded during training Model-version lifetime No; they are part of the model itself
Invocation context Active instructions and information available for one inference call One invocation Yes
Conversation state Ordered interaction items associated with a thread or conversation Turn, session, or longer, depending on implementation Only when referenced or reconstructed for the request
Workflow state Current task position, intermediate results, pending inputs, and execution metadata Workflow or run lifetime Only when the application includes it
Persistent memory Durable user, project, task, or agent state intended for later reuse Cross-session until updated, expired, or deleted Only after retrieval and selection
External knowledge source Documents, databases, indexes, or corpora maintained independently of the interaction Source-defined Only after retrieval or attachment
Context or prompt cache Reuse of previously processed input prefixes or content Cache-defined It reuses computation; it does not define semantic memory

Model parameters

Model parameters encode patterns learned during training or later model adaptation. They are not an application-controlled record of a particular conversation, user, project, or workflow.

Updating application memory does not update model parameters. The application changes future behavior by changing the state supplied to later invocations.

Invocation context

Invocation context is the active working set for one model call.

It may contain:

  • system or developer instructions;
  • the current user request;
  • selected conversation turns;
  • retrieved records or documents;
  • summaries or compacted state;
  • tool definitions and tool results;
  • structured workflow state.

Context is temporary from the perspective of the model call. Continuity requires the surrounding application or model service to preserve and reintroduce the required state.

Conversation state

Conversation state is the ordered record used to continue a multi-turn interaction.

OpenAI documents two broad implementation patterns: manually resending prior items with each request, or using stateful response and conversation objects that persist items under a durable identifier. In both cases, the architecture still has to manage how accumulated state fits within the context window.

Conversation state is not automatically equivalent to long-term memory. It may preserve a transcript without extracting durable facts, resolving updates, or selecting information by future task relevance.

Workflow state

Workflow state represents the current execution of a task rather than a general history of interaction.

Examples include:

  • the current step in a process;
  • intermediate calculations or model outputs;
  • pending approvals or missing inputs;
  • identifiers created during the run;
  • retry counts and completion status;
  • the current plan or task decomposition.

Workflow state may be short-lived even when the conversation is long-lived. It may also be stored in structured application state rather than natural-language memory.

Persistent memory

Persistent memory is information intentionally retained so that it can influence later interactions or workflows.

Examples include:

  • a stable user preference;
  • a confirmed project constraint;
  • a prior decision and its status;
  • a summary of an earlier session;
  • an event that later tasks may need to recall;
  • a reusable procedural note derived from prior work.

Persistent memory requires more than storage. It requires a lifecycle that defines creation, representation, scope, retrieval, update, and deletion.

The five architectural boundaries

1. Invocation boundary

The invocation boundary separates all state available to the application from the subset available to the model during one inference call.

The model cannot directly use a record merely because it exists in a database, vector index, conversation object, file store, or workflow engine. The record must be represented in the request or made available through an implementation-specific state mechanism that contributes to the request context.

The invocation boundary is constrained by:

  • the model’s context-window capacity;
  • the tokens reserved for output and, where applicable, reasoning;
  • the size of instructions, messages, documents, tools, and tool results;
  • the application’s context-selection and compaction strategy.

A larger context window changes the capacity of this boundary. It does not remove the need to decide what information belongs inside it.

2. Persistence boundary

The persistence boundary determines what temporary information becomes durable application state.

Questions at this boundary include:

  • Which events are eligible to become memory records?
  • Is the original content stored, or is a derived representation created?
  • Is the record associated with a user, project, task, workflow, session, or agent?
  • Does a new record append to history, update an existing record, or supersede it?
  • What retention or expiration rule applies?

The persistence boundary is an architectural decision because retaining every message is not the same as maintaining usable memory. Raw history preserves detail, while structured records and summaries support different retrieval and update operations.

3. Retrieval boundary

The retrieval boundary separates the complete stored state from the candidate records considered for the current task.

Retrieval may use:

  • direct identifiers;
  • structured queries;
  • metadata filters;
  • chronological ranges;
  • keyword search;
  • semantic similarity;
  • graph traversal;
  • hybrid retrieval.

Retrieval should be treated as candidate generation. A returned item is not necessarily relevant enough, current enough, or compact enough to enter the invocation context.

LongMemEval models long-term memory design through indexing, retrieval, and reading stages. That decomposition is useful because retrieval quality depends not only on the query, but also on how information was segmented and represented when it entered the memory system.

4. Context-assembly boundary

The context-assembly boundary determines which candidates become active model input and how they are represented.

This stage may perform:

  • relevance ranking;
  • deduplication;
  • chronological ordering;
  • conflict grouping;
  • truncation;
  • summarization;
  • compaction;
  • conversion into structured fields;
  • allocation of token budgets across source categories.

Context assembly is not a neutral concatenation step. Ordering, representation, and compression affect what evidence remains available to the model.

A memory architecture should therefore distinguish at least three sets:

  1. all persisted records;
  2. records retrieved as candidates;
  3. records actually included in the invocation context.

Collapsing these sets makes memory behavior difficult to explain and evaluate.

5. Update boundary

The update boundary determines how new information changes existing memory state.

An update may:

  • append a new event;
  • revise a mutable record;
  • create a new version;
  • supersede an earlier value;
  • merge duplicate records;
  • invalidate obsolete state;
  • reduce confidence in an uncertain record;
  • expire or delete a record.

This boundary is necessary because long-term memory is not only a recall problem. It is also a state-maintenance problem.

For example, storing both an old and a new project deadline without representing their temporal relationship leaves the retrieval layer to infer which one is current. A stronger state model records whether the new value supplements, contradicts, or supersedes the earlier one.

Memory is a lifecycle, not a database

A memory-enabled system should define the complete path from an observed event to its later use.

1. Capture

The system identifies information that may be useful beyond the current interaction.

Capture can originate from user input, application events, completed workflow steps, tool results, or model-generated summaries. Capture alone does not imply that the information should be persisted.

2. Admission

The system decides whether the captured information should become a memory record.

Admission criteria may depend on:

  • expected future utility;
  • stability over time;
  • repetition across interactions;
  • whether the information is already represented;
  • whether the event belongs to temporary workflow state instead.

This prevents the memory store from becoming an undifferentiated copy of every interaction.

3. Representation

The admitted information is converted into a form suitable for storage and later retrieval.

Common representations include:

  • raw turns or events;
  • structured fields;
  • key-value records;
  • session summaries;
  • timestamped observations;
  • documents or chunks;
  • embeddings plus source text;
  • linked or graph-structured records.

No representation is universally sufficient. A narrative summary preserves different information from an immutable event log or a structured preference record.

4. Persistence

The representation is written to a store with explicit identity, scope, timestamps, and lifecycle metadata.

At minimum, a durable memory record should make it possible to determine:

  • what the record represents;
  • which entity or workflow it belongs to;
  • when it was created and last updated;
  • where it originated;
  • whether it replaces or depends on another record;
  • when it expires or becomes eligible for deletion.

5. Retrieval

A later request produces a candidate set from one or more memory stores.

The retrieval method should match the representation. Structured records support exact filtering and joins. Event histories support temporal reconstruction. Vector indexes support semantic similarity. Summaries support compact recall but may omit details that were not preserved during compression.

6. Selection and transformation

The system chooses which candidates are useful for the current invocation and converts them into an appropriate context representation.

Selection may account for:

  • current-task relevance;
  • temporal validity;
  • record type;
  • duplication;
  • relationship to other selected records;
  • context budget;
  • the level of detail required by the task.

Transformation may preserve the original record, extract fields, group related events, or produce a compact summary.

7. Context assembly

Selected state is combined with instructions, the current request, conversation items, workflow state, and any other required inputs.

The output of this stage is the actual invocation context, not merely a list of retrieved records.

8. Maintenance

After the interaction, the system may create new records or revise existing ones. Maintenance includes versioning, supersession, deduplication, expiration, correction, and deletion.

Without maintenance, retrieval quality can degrade even when the underlying search mechanism remains unchanged.

Common memory architecture patterns

Full-history replay

The application preserves prior interaction items and sends the relevant history again on each turn.

This is the simplest continuity pattern. It preserves verbatim detail while the history fits within the context window. As the interaction grows, the application must truncate, compact, or otherwise reduce the history.

Rolling summary or compaction

The application replaces part of the accumulated context with a shorter representation that carries forward selected state.

OpenAI documents compaction as a method for reducing context size while preserving state needed for later turns. Similar context-management mechanisms exist across model platforms.

Compaction is not equivalent to persistent memory. It is a method for maintaining a bounded active context. The compacted representation may itself be persisted, but its primary role is to reduce the size of continued interaction state.

Retrieval-backed memory

The application stores interaction-derived records outside the active context and retrieves a subset for later requests.

This pattern scales beyond full-history replay because the complete memory store does not have to enter every invocation. Its behavior depends on segmentation, representation, indexing, query construction, ranking, and reading. The Generative Agents architecture illustrates this pattern with a memory stream and retrieval based on relevance, recency, and importance.

Structured state memory

The application stores explicit fields or state objects rather than relying only on free-form conversation text.

This pattern is appropriate when the system needs to preserve values such as task status, project constraints, selected options, dates, identifiers, or current workflow position.

Structured state does not replace narrative memory. The two representations answer different types of future queries.

Layered or hierarchical memory

The application combines multiple state tiers, such as:

  • active invocation context;
  • recent conversation state;
  • compacted session state;
  • durable structured records;
  • retrieval-backed episodic or semantic records.

CoALA models language agents with modular working, episodic, semantic, and procedural memory components. MemGPT demonstrates an operating-system-inspired approach that moves information between memory tiers to work beyond a fixed context window. These are distinct research architectures, but both support the broader principle that memory can be organized into layers with different access patterns and lifetimes.

Mechanisms that should not be conflated with persistent memory

Long context

A long context window increases the amount of information that can be processed in one invocation.

It can reduce the immediate need for truncation or retrieval, but it does not define cross-session persistence, record identity, update semantics, or future selection. Google’s long-context documentation still identifies summarization, filtering, and retrieval as useful techniques and notes that performance can vary with context composition.

Conversation persistence is not long-term memory

Conversation state preserves continuity between turns. It may be durable, but a durable transcript is not automatically an indexed or maintained memory system.

A conversation store answers “what happened in this thread.” A memory system must additionally determine what should be reused, how it should be represented, and whether later information changes earlier state.

Retrieval-augmented generation

RAG is a retrieval pattern for adding external information to a model request.

A memory system may use RAG techniques, but not every retrieved corpus is memory. A product catalog, policy library, or document repository can supply context without representing prior agent or user state.

Memory is defined by retained state and its lifecycle. RAG is defined by retrieval into an inference workflow.

Context or prompt caching

Context caching reuses previously processed input to reduce repeated computation, latency, or cost. OpenAI and Google document caching in those terms.

Caching does not decide what the system should remember. It does not create update semantics, resolve conflicting records, or determine which state is relevant to a future task. It is an execution optimization around repeated context.

Fine-tuning

Fine-tuning changes model parameters. It can adapt model behavior to a task or data distribution, but it does not provide a directly editable, per-user, or per-project memory record.

Application memory should remain conceptually separate from parameter updates because the two mechanisms have different update paths, scopes, and inspection properties.

Why output changes do not prove that memory changed

A different answer is not sufficient evidence that the memory store or retrieval result changed.

Output variation can result from:

  • a different user message;
  • different instructions;
  • a different subset or order of conversation items;
  • different retrieved records;
  • a different summary or compaction result;
  • changes in tool results or external data;
  • a different model or model version;
  • different decoding or reasoning configuration;
  • nondeterministic generation.

OpenAI’s prompt-caching documentation explicitly notes that caching an identical prompt prefix does not make otherwise nondeterministic requests return identical outputs.

Memory-related causality can be established only by inspecting the state path: what was stored, what was retrieved, what was selected, how it was transformed, and what entered the invocation context.

Observability requirements

A memory architecture should make the path from stored state to model-visible context inspectable.

Useful records include:

  • memory creation and update events;
  • record identifiers, types, scopes, and versions;
  • the stores and indexes queried;
  • retrieval queries and filters;
  • candidate identifiers and retrieval scores;
  • selection and exclusion decisions;
  • transformations, summaries, or compaction events;
  • the final context composition by source category;
  • token allocation by context component;
  • the model and configuration used for the invocation.

The objective is to distinguish storage behavior, retrieval behavior, context construction, and model generation rather than inferring the entire system state from the final answer.

Evaluating memory as an end-to-end system

Retrieval accuracy alone does not establish that a memory architecture works correctly.

Evaluation should cover the full lifecycle:

Capture and representation

  • Was the relevant information identified?
  • Was it represented without changing its meaning?
  • Was its temporal and entity scope preserved?
  • Were updates represented as updates rather than unrelated duplicates?

Retrieval and selection

  • Was the necessary record retrieved?
  • Were irrelevant records excluded?
  • Did ranking preserve the records needed for the task?
  • Did the selected set fit within the context budget?

Context assembly

  • Was the selected state represented at the required level of detail?
  • Did summarization or compaction remove necessary information?
  • Were temporal relationships and record updates preserved?
  • Could the model distinguish current state from historical state?

Downstream behavior

  • Did the memory improve task completion or answer quality?
  • Could the system reason across multiple sessions?
  • Could it apply newer information when an earlier record had changed?
  • Could it abstain when the required memory was absent?

LongMemEval operationalizes five long-term memory abilities for chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. These categories are not a complete architecture standard, but they provide a stronger evaluation target than undifferentiated “memory recall.”

Operational efficiency

  • How much storage does the memory representation require?
  • How many records are retrieved and then discarded?
  • How many context tokens are consumed by memory?
  • What latency is introduced by indexing, retrieval, transformation, and assembly?
  • How often are summaries or indexes rebuilt?

A useful memory system must balance retained fidelity, retrieval quality, context efficiency, latency, and maintenance cost.

Design principles

Keep active context and durable state separate

The context window is the model’s active working set. Persistent memory is an external state system. Treating them as separate components makes retention and context behavior explicit.

Define scope and lifetime for every memory class

A memory record should belong to a defined user, project, workflow, task, conversation, or agent scope and have an explicit lifetime.

Choose representations by future access pattern

Use structured records for exact state, event histories for temporal reconstruction, summaries for compact continuity, and semantic indexes for similarity-based retrieval. Do not assume one representation can satisfy every future query.

Treat retrieval and selection as separate stages

Retrieval generates candidates. Selection determines what belongs in the current invocation. Preserving this separation improves debuggability and context control.

Represent updates explicitly

New information should be able to revise, supersede, invalidate, or extend earlier state. Append-only storage without update semantics shifts unresolved contradictions into later retrieval and generation stages.

Make compression visible

Summarization and compaction are lossy transformations unless the complete source remains separately available. Record when a transformation occurred and which source state it represents.

Evaluate the complete path

Measure capture, representation, retrieval, selection, context assembly, update handling, and downstream task performance. A strong vector search score cannot compensate for incorrect admission or stale state.

Key takeaways

  • LLM memory is an application-level state architecture, not a single model feature.
  • Invocation context and persistent memory are different state domains.
  • Conversation state, workflow state, long context, retrieval, caching, and fine-tuning solve different problems.
  • A complete memory architecture defines persistence, retrieval, context assembly, and update boundaries.
  • Retrieval results are candidates; only selected and assembled state becomes model-visible context.
  • Summarization and compaction manage bounded context but may remove information.
  • Output variation alone does not establish that memory changed.
  • Memory evaluation must cover the full lifecycle, including updates, temporal reasoning, abstention, context efficiency, and downstream utility.

Apply the memory-boundary model

Use these resources to translate the architectural model into an implementation workflow and continue into adjacent agent-orchestration patterns.

References

Official documentation:

Research: