LLM Improper Output Handling: Risks, Controls, and Test Cases

By Tamar Peretz Published 2026-08-04

Prevent unsafe use of LLM output with validation, sanitization, encoding, privilege boundaries, approval gates, and regression tests.

An LLM response is not trusted merely because it was generated by the application rather than entered directly by a user.

The response may still be influenced by user input, retrieved documents, tool results, earlier model calls, or a prompt-injection payload. If the application passes that response into a browser, database, shell, code runtime, API, or privileged workflow without enforcing the receiving system’s security rules, natural-language output can become executable input.

OWASP defines LLM05:2025 Improper Output Handling as insufficient validation, sanitization, and handling of model output before it is passed to downstream components. Its documented impacts include cross-site scripting, server-side request forgery, privilege escalation, and remote code execution. The general software weakness is not unique to LLMs: CWE-74 describes improper neutralization of special elements in output used by a downstream component.

The engineering rule is therefore direct:

Treat model output as untrusted data until it has been validated, authorized, and encoded or otherwise constrained for the exact destination that will consume it.

This article explains where improper output handling begins, how it differs from prompt injection, which controls belong at each boundary, and how to test that the controls fail closed.

Scope

This article focuses on application behavior after an LLM produces output. It covers:

free-form text and Markdown rendered in a user interface
structured output parsed as JSON or another schema
model-generated SQL, shell arguments, and code
tool calls and function arguments proposed by a model
state-changing and privileged actions
human approval for high-impact or irreversible operations
security regression tests and acceptance criteria

The article does not treat every inaccurate model response as an output-handling vulnerability. Incorrect or unsupported claims can create misinformation and overreliance risks even when the response is displayed safely as text. Improper output handling is narrower: the application allows model output to cross into a downstream component without the controls required by that component.

Prompt injection and improper output handling are different failure stages

Prompt injection and improper output handling often occur in the same incident, but they are not interchangeable.

OWASP LLM01 defines prompt injection around input that alters model behavior or output in unintended ways. The input may come directly from a user or indirectly from a website, file, retrieved record, image, or another external source. Improper output handling begins later, when the application accepts what the model produced and sends it to another component without sufficient scrutiny.

Question	Prompt injection	Improper output handling
Primary boundary	Untrusted content → model behavior	Model output → downstream component
Immediate failure	Input changes the model’s behavior or output in an unintended way	Output is trusted, interpreted, or executed without destination-specific controls
Typical example	A retrieved document instructs the model to ignore the user’s task and produce a malicious tool argument	The application executes the proposed tool argument without independent authorization and validation
Primary OWASP category	LLM01:2025 Prompt Injection	LLM05:2025 Improper Output Handling
Main control objective	Limit how untrusted content can influence behavior	Prevent model output from becoming unauthorized data, code, or action

The distinction matters because fixing only one side leaves the other exposed. Input filtering cannot prove that every model response is safe. Output encoding cannot stop an injected instruction from changing a non-executable answer. A secure application needs controls on both transitions.

The failure path can be represented as:

untrusted input or retrieved content → model output → parser or renderer → authorization and policy decision → downstream execution or display

Every arrow is a trust transition. The model is not an authorization server, a SQL parser, a shell policy engine, or an HTML sanitizer. Those responsibilities remain with deterministic application controls.

Structured output is a contract, not an authorization decision

Structured output reduces ambiguity. A strict schema can constrain field names, types, enumerated values, string patterns, numeric ranges, required fields, and whether additional properties are allowed. That is materially safer than passing free-form prose into a downstream component.

It is still only one layer.

Consider a model-generated action object:

{
  "action": "delete_document",
  "document_id": "doc_4821",
  "reason": "Requested during account cleanup"
}

The object may be valid against its JSON Schema. Schema validity does not establish that:

the current user owns or may delete doc_4821
deletion is permitted in the current workflow state
the object refers to the same tenant as the authenticated session
the action reflects the user’s actual request
the reason is true
required approval has been granted
the target has not changed since approval

The receiving application must therefore enforce several distinct checks.

1. Parsing and schema validation

Reject malformed output, missing required fields, unexpected properties, invalid types, out-of-range values, and disallowed enum values. Use a strict schema and reject rather than silently coerce ambiguous values on security-sensitive paths.

2. Semantic and domain validation

Validate whether the values make sense for the operation. A date may be syntactically valid but outside the allowed period. A URL may be well formed but point to an unapproved scheme, host, or private network. A file identifier may exist but refer to a protected resource. A monetary amount may be numeric but exceed a transaction limit.

3. Authorization and workflow validation

Bind the proposed action to the authenticated principal, tenant, permissions, workflow state, and current policy. Authorization must be evaluated by trusted application or downstream-system code for every operation. It must not be inferred from the model’s explanation or from a field such as authorized: true.

4. Destination-specific handling

Apply the control required by the sink: output encoding for a browser context, parameterized queries for SQL, a fixed API instead of shell construction, or a sandbox and review gate for generated code. There is no universal escaping function that makes arbitrary output safe for every destination.

5. Side-effect gating

Separate proposing an operation from committing it. A valid proposal may be shown as a preview, diff, or dry run without granting permission to execute it. State-changing actions should pass through a distinct authorization gate, and high-impact actions may require explicit human approval.

Schema enforcement is valuable because it narrows the interface. It should be treated as the first enforceable contract at the boundary, not as proof that the proposed content is safe, accurate, or authorized.

Handle output according to the destination

OWASP’s LLM05 guidance calls for context-aware output handling. This is essential because the same character sequence has different meaning in HTML, JavaScript, SQL, a shell, a file path, and an API request.

HTML, Markdown, and browser rendering

If the product needs to display plain model-generated text, render it through a safe text sink or the framework’s normal escaping mechanism. Do not insert it through raw HTML APIs.

If the product intentionally supports model-generated rich HTML, output encoding alone will remove the markup and break the feature. In that case, sanitize the HTML with a maintained allowlist-based sanitizer, restrict allowed tags, attributes, URL schemes, and protocols, and keep the sanitizer patched. OWASP’s Cross-Site Scripting Prevention Cheat Sheet distinguishes contextual output encoding from HTML sanitization and warns that encoding rules differ across HTML, attribute, JavaScript, CSS, and URL contexts.

Markdown also requires an explicit trust decision. A Markdown parser may allow embedded HTML, links, images, or extensions that create active content. A safe implementation should disable raw HTML unless it is required, validate link and image schemes and destinations, add appropriate link attributes, and sanitize the rendered HTML before it reaches the browser.

Content Security Policy can reduce the impact of some cross-site scripting failures, but OWASP lists it as defense in depth rather than a replacement for correct encoding or sanitization.

Required property: model-generated content is never interpreted as active browser code merely because the model placed markup in its response.

SQL and database operations

Do not execute arbitrary SQL produced by a model in a production database context.

For normal application operations, keep the SQL statement in trusted code and bind model-derived values through prepared statements or parameterized queries. OWASP’s SQL Injection Prevention Cheat Sheet identifies prepared statements as the primary defense because they separate code from data. Escaping an entire model-generated query is not an equivalent control.

When identifiers such as table names, column names, sort directions, or operations cannot be parameterized, map a constrained model value to a server-side allowlist. For example, an enum value such as sort: "newest" can map to a fixed ORDER BY created_at DESC clause. The model should not provide the clause itself.

If a legitimate product feature generates analytical queries, isolate it from transactional systems. Apply a read-only database identity, restrict accessible schemas or views, enforce statement and row limits, set timeouts, reject multiple statements and data-definition or data-modification operations, and validate the parsed query before execution. A preview of the query is not a substitute for database permissions.

Required property: model output can supply validated data values or select among pre-authorized query templates, but it cannot redefine the database operation or exceed the caller’s data access.

Shell commands and operating-system processes

The strongest control is to avoid constructing shell commands from model output. OWASP’s OS Command Injection Defense Cheat Sheet recommends using language APIs or libraries instead of calling operating-system commands directly.

Expose narrow operations such as resize_image(input_id, width, height) or convert_document(source_id, format) rather than a general run_shell(command) capability. Resolve resource identifiers to server-controlled paths, use fixed executable paths, pass arguments as an array without invoking a shell, and validate every argument against a positive allowlist and bounded length.

Escaping is operating-system and shell specific, and argument injection may remain possible even when command separators are quoted. For that reason, shell escaping is a fallback control, not a design goal. If a shell cannot be avoided, combine structured parameterization with strict validation, least privilege, a constrained execution environment, resource limits, and complete logging.

Required property: the model cannot introduce a new executable, option, pipe, redirection, path, or command sequence outside the operation explicitly exposed by trusted code.

Generated code

Generated code is not made safe by escaping it. The security question is whether the code will be interpreted or executed, with which dependencies, permissions, secrets, network access, and filesystem access.

Do not pass model output to eval, a template expression evaluator, a dynamic module loader, or a production runtime merely because the output compiles. Treat generated code as an untrusted artifact. Apply the same controls required for human-authored external code, with additional attention to fabricated dependencies and hidden data flows:

parse and lint before execution
run static analysis and dependency checks appropriate to the language
use an isolated, disposable environment with no production credentials
disable network and host filesystem access unless explicitly required
enforce CPU, memory, execution-time, and output-size limits
require tests and code review before integration
separate a generated patch from the merge or deployment authority

For code-like values that are not meant to execute—for example, a code sample displayed in documentation—encode them for the rendering context and keep them in a non-executable text sink.

Required property: generated code cannot move from model response to trusted execution or deployment without an explicit, independently controlled review and execution path.

Tool calls are untrusted action proposals

A tool call may be syntactically structured, but its fields are still model output. The application should treat the call as a proposal that must be evaluated before dispatch.

OWASP LLM06:2025 Excessive Agency identifies three common root causes of damaging agent actions: excessive functionality, excessive permissions, and excessive autonomy. Its mitigation guidance calls for minimizing available tools and tool permissions, executing in the user’s context, requiring approval for high-impact actions, and enforcing authorization in downstream systems rather than relying on the LLM.

A defensible tool boundary should enforce all of the following:

Tool allowlist: expose only the tools required for the current workflow and user intent. Read-only and write-capable operations should be separate capabilities.
Strict argument schema: reject extra fields, ambiguous types, unbounded strings, invalid identifiers, and values outside defined ranges.
Semantic constraints: validate allowed targets, domains, resource types, counts, amounts, dates, and state transitions.
Caller-bound authorization: evaluate access using the authenticated user’s identity and tenant, not a generic privileged agent identity.
Complete mediation: authorize every tool invocation, including retries and calls generated later in an agent loop.
Least privilege: give each tool the minimum downstream permissions necessary for its single operation.
Side-effect classification: distinguish read, draft, reversible write, external communication, financial action, permission change, destructive action, and other high-impact classes.
Fail-closed behavior: if parsing, policy evaluation, authorization, or approval state is missing or uncertain, do not dispatch the call.

The tool response must also be treated according to its provenance and destination. A successful API response does not authorize the next action, and text returned by a tool must not become a new instruction channel without the same boundary controls.

Human approval before irreversible or high-impact actions

Human approval is appropriate when a model-influenced workflow can create consequences that are difficult to reverse, externally visible, privileged, or materially harmful. Examples include deleting records, changing permissions, transferring funds, publishing content, sending external communications, executing production changes, or disclosing sensitive data.

Approval does not replace authentication, authorization, validation, least privilege, or logging. It is an additional control over a fully specified action.

A meaningful approval step should show the human:

the exact action
the target resource and tenant
the fields or content that will change
the affected recipients or external systems
the current value and proposed value, when applicable
the expected side effects and reversibility
the identity under which the action will execute

The approval must be bound to that exact proposal. A strong implementation stores an immutable action representation or digest with the approver identity, timestamp, scope, and expiry. If the action, target, recipients, parameters, or generated content changes after approval, the prior approval no longer applies.

The approval interface should be produced from trusted application data rather than only from a model-written summary. Otherwise, a misleading summary could hide the actual tool arguments. The commit path should compare the approved action with the action being executed and reject any mismatch.

For lower-risk reversible operations, a preview, undo window, or staged draft may provide a proportionate control. For high-impact operations, step-up authentication, dual control, or a separate downstream approval system may be justified by the organization’s risk model.

Security test cases

Testing should verify the complete path from model output to its final sink. A unit test that only checks whether JSON parses does not establish that the application prevents unsafe execution.

NIST AI 600-1 calls for documented assurance criteria under conditions similar to deployment, purpose-built testing environments, and regular review of security and safety guardrails. The matrix below translates those lifecycle expectations and the cited sink-specific controls into application-level regression cases.

The following matrix defines implementation-level tests. The acceptance criteria are engineering requirements derived from the cited control principles; they are not presented as a formal OWASP certification checklist.

Test	Model output or condition	Expected control	Acceptance criterion
Schema rejection	Missing required field, wrong type, unknown property, or out-of-range value	Strict parser and schema validator	Output is rejected before any tool, renderer, database, or state-changing code receives it; rejection is logged without sensitive content.
Semantic rejection	Valid schema with an unapproved host, resource type, amount, date, or state transition	Domain allowlist and policy validation	The request fails closed with a stable policy reason; no downstream call is attempted.
Cross-tenant reference	Valid resource identifier belonging to another tenant	Caller-bound authorization	The downstream authorization layer denies access even if the model labels the action as authorized.
HTML and Markdown injection	Script tags, event handlers, unsafe URLs, raw HTML, or active Markdown content	Safe text sink or allowlist sanitizer plus contextual encoding	No executable browser behavior occurs; allowed formatting remains functional; blocked content is removed or rendered inert.
SQL injection	Quotes, comments, stacked statements, or a request to change the query operation	Parameterized query or fixed server-side query template	The value remains data; the query structure and authorized dataset do not change; the database identity cannot perform writes outside the operation’s authorized scope.
Shell and argument injection	Command separators, redirection, option injection, path traversal, or an alternate executable	Narrow API, fixed executable, argument array, allowlist, and sandbox	No shell is invoked; only the allowlisted operation runs; invalid arguments are rejected before process creation.
Generated-code execution	Code that accesses the network, filesystem, secrets, dynamic evaluation, or undeclared packages	Isolation, static checks, dependency policy, tests, and review	The artifact cannot access prohibited resources or enter production without the independent review and deployment path.
Unauthorized tool selection	A read request produces a write-capable or administrative tool call	Workflow-scoped tool allowlist and authorization	The router rejects the tool before dispatch; no privileged credential is used.
Argument scope expansion	A tool call changes one item into all items, adds recipients, or targets a broader resource	Semantic constraints and intent-to-action binding	Only the user-authorized scope is allowed; added targets or recipients cause rejection or a new approval request.
Approval mismatch	Target, content, amount, recipient, or parameters change after human approval	Approval bound to the immutable action	Commit is denied and fresh approval is required.
Validator outage or timeout	Schema, policy, authorization, sanitizer, or approval service is unavailable	Fail-closed error handling	No side effect occurs and the workflow enters a visible error or review state rather than a permissive fallback.
Indirect prompt injection chain	Retrieved content induces a valid-looking output that requests a privileged action	All downstream controls combined	The output may parse, but independent policy and authorization still prevent the action.

Regression-test design

A useful regression suite contains more than known attack strings. Include:

valid baseline cases to confirm that controls do not break intended behavior
boundary values for lengths, counts, ranges, dates, and enum transitions
unknown fields and type-confusion cases
cross-user and cross-tenant identifiers
alternate encodings and Unicode normalization cases where relevant
indirect-injection payloads placed in retrieved documents and tool responses
retries, parallel calls, stale approvals, and changed workflow state
validator failures, timeouts, and partial downstream failures
sink-specific payloads for every renderer, query layer, process runner, and code runtime actually used by the application

The test oracle should inspect both the visible response and the side effects. A safe error message is not sufficient if a database write, external request, message send, or file change already occurred.

Minimum acceptance criteria

An LLM application handling output safely should meet the following minimum criteria:

Every model-to-component boundary has an identified owner, schema or input contract, validation rule, and failure mode.
Model output is rejected by default when parsing, validation, authorization, or required approval fails.
Structured output is validated strictly, and semantic checks are performed separately from schema checks.
No authorization decision is accepted from the model. Every protected operation is authorized in trusted code or by the downstream system using the authenticated principal and current resource state.
Browser output uses safe sinks, context-appropriate encoding, and sanitization only where rich content is intentionally supported.
Database operations use parameterized queries or fixed server-side templates; model output is never executed as unrestricted SQL.
Shell access is avoided or reduced to narrow, allowlisted operations with constrained arguments and execution isolation.
Generated code remains an untrusted artifact until it passes the required isolation, analysis, testing, review, and deployment controls.
High-impact actions require approval bound to the exact action, and any material change invalidates the approval.
Logs record validation, policy, authorization, approval, dispatch, and outcome events with correlation identifiers while excluding or redacting secrets and unnecessary sensitive data.
Security tests cover every active sink and verify that prohibited side effects do not occur.
The controls are applied on every retry, loop iteration, and tool call; an earlier successful check is not reused after relevant state changes.

These criteria should be evaluated within the wider eight-checkpoint AI agent security audit, because safe output handling depends on upstream provenance and downstream authorization as well as the output validator itself.

Common control failures

Several patterns appear secure but leave the execution boundary open.

“The system prompt tells the model not to produce dangerous output”

Prompt instructions can reduce undesirable responses, but they are not a deterministic downstream control. OWASP’s prompt-injection guidance explicitly recommends validating expected formats with deterministic code, enforcing least privilege, and requiring human approval for high-risk actions.

“The output is valid JSON”

JSON validity proves that the text can be parsed. A strict JSON Schema can prove additional declared constraints. Neither proves that a resource belongs to the caller, that the action is allowed, or that the factual basis is correct.

“We sanitize everything once”

Sanitization must be defined for a specific data type and use. HTML sanitization is not SQL parameterization, shell argument validation, URL policy, or authorization. Centralized generic filtering can also apply the wrong encoding to the wrong browser context.

“The user approved it”

Approval is ineffective if the user sees only a vague model summary, if parameters can change afterward, or if the system would execute the action under permissions the user does not possess. Approval must be specific and must remain subordinate to authorization policy.

“The tool validates its own inputs”

That is necessary, but the caller must still minimize which tools are exposed, constrain the intended operation, bind the request to the user, and handle tool responses safely. Security cannot depend on a general-purpose tool correctly interpreting arbitrary model output.

Conclusion

Improper output handling is the point where generated language acquires software authority.

The model may produce a string, a JSON object, a query, a code patch, or a tool call. None of those outputs is inherently authorized or safe. The application determines whether the output remains inert data or becomes browser content, database logic, a process invocation, executable code, or a real-world action.

The correct control model is layered:

constrain the output format
validate its declared structure
validate its meaning for the domain
authorize the operation against the current principal and resource
handle the content for the exact destination
separate proposal from execution
require specific approval where impact warrants it
test the complete path and fail closed

Prompt injection explains one way an attacker can influence the model. Improper output handling explains how the application turns that influence into downstream impact. Preventing the second failure requires ordinary secure-software controls applied rigorously at every model-output boundary.