When AI Systems Optimize Against the User’s Real Goal
How optimization targets, instruction authority, resource constraints, context construction, orchestration, tool policies, and completion logic can produce acceptable outputs without completing the user’s real task.
Introduction
Users approach AI systems with a straightforward expectation: the system should help them achieve the result they asked for.
A researcher expects the strongest conclusion supported by the relevant evidence. A developer expects an implementation that works in the target environment. A decision-maker expects a recommendation based on sufficient and current information. A user delegating an operational task expects the intended change to occur in the external system.
The AI system, however, is not governed solely by that outcome.
A deployed system may also be required to follow higher-priority instructions, enforce safety and authorization boundaries, remain within latency and cost budgets, limit tool calls, operate inside a finite context, satisfy product-level quality thresholds, and terminate according to predefined execution rules.
These interests can be compatible with the user’s goal. They can also conflict with it.
When they conflict, the user’s desired outcome does not automatically become the system’s operative definition of success.
The system may produce an acceptable response.
It may produce it quickly.
It may remain within policy and budget.
It may reach a technically valid completion state.
Yet the user’s actual task may remain incomplete.
This article examines that conflict at the level of the complete AI system—not only the foundation model. It explains how optimization targets, instruction authority, operational constraints, context construction, orchestration, tool policies, completion logic, and evaluation criteria can pull system behavior away from the outcome the user needs.
The discussion focuses on legitimate, authorized, and sufficiently specified user objectives. It does not assume that a user request should override safety, legal, authorization, or third-party constraints. The narrower and more important claim is that even a valid user objective can fail to become the system’s effective completion criterion.
Here, “system interests” does not imply human intention or consciousness. It refers to the objectives, constraints, policies, priorities, and success metrics that materially shape system behavior.
1. User Outcome and System Success Are Not the Same
The user usually defines success in terms of an outcome.
Was the research sufficiently supported?
Were the required sources reviewed?
Does the implementation compile and pass its tests?
Was the database actually updated?
Was the external action completed?
Did the system stop when the available evidence was insufficient?
A system can use a different operational definition of success:
- A valid response was generated.
- The output matched the required schema.
- The model returned no additional tool call.
- The workflow reached an exit condition.
- No policy or guardrail was violated.
- The execution remained within its time and cost limits.
- The result passed an evaluation focused on response quality.
- The application returned a successful status.
These conditions are not meaningless. A production system needs them. The problem begins when they are treated as evidence that the user’s underlying objective was achieved.
Three forms of success must therefore be separated.
Response success concerns the generated artifact: its relevance, clarity, factual content, structure, and format.
Execution success concerns the system run: whether components executed, tools returned valid responses, policies were respected, and the workflow reached a recognized terminal state.
Outcome success concerns the task itself: whether the required real-world state was reached and whether the evidence needed to establish that result exists.
A response can succeed while the execution fails.
An execution can succeed while the outcome fails.
For example, an agent may call the correct API and receive an HTTP success response. That establishes that the request was accepted at the protocol level. It does not necessarily establish that the business process completed, that the correct record changed, or that the final state satisfies the user’s requirement.
Similarly, a coding system may generate a coherent patch and explain its logic. That establishes that an implementation was proposed. It does not establish that the patch compiles, passes the relevant tests, works against the deployed schema, or avoids regressions.
The user’s objective becomes operational only when it is translated into explicit acceptance criteria and verifiable postconditions.
Without that translation, the system can complete what it was configured to execute while leaving the task the user intended unfinished.
2. Why This Problem Is Broader Than Hallucination
Hallucination, often described more precisely as confabulation, is one failure mode.
NIST defines confabulation as confidently presented erroneous or false content and treats it as one category within a broader set of generative-AI risks. Its Generative AI Profile also addresses risks involving information integrity, human–AI configuration, and value-chain or component integration. [7]
The conflict examined in this article does not require the model to invent a fact.
A system can fail while every individual statement it produces is technically correct.
A research report can contain accurate claims while omitting evidence that would change the conclusion.
A document can become clearer and more persuasive while preserving an unsupported causal claim.
A code change can be syntactically valid while failing against the current database schema.
A recommendation can rely on real data that is insufficient for the decision.
An automation can invoke the correct endpoint while failing to produce the required business state.
A support agent can accurately explain a policy while leaving the user’s case unresolved.
These are not necessarily hallucinations.
They are failures of task completion, evidence sufficiency, execution control, or outcome verification.
The distinction matters because hallucination-focused interventions are too narrow.
Lowering temperature does not prove that the evidence is complete.
Adding citations does not prove that the cited sources support every material claim.
Providing a tool does not prove that the tool was used.
Calling a tool does not prove that the external action succeeded.
Generating valid JSON does not prove that the content is authorized or correct.
Producing executable-looking code does not prove that it was executed.
A factuality check can improve the output while leaving the underlying workflow incomplete.
The system can be correct about what it says and still fail at what it was asked to accomplish.
3. Where the conflict enters the AI system
The term “AI system” refers here to the deployed combination of model, application logic, instructions, context, tools, policies, infrastructure, and evaluation controls.
The argument applies broadly to LLM-based applications. Some mechanisms discussed in this section—such as iterative tool use, external state changes, and agent-loop termination—apply specifically to tool-enabled and agentic systems.
System behavior is shaped by several technically distinct mechanisms:
- Model-level optimization targets that shape candidate behavior
- Instruction-authority and policy rules that determine which requests take precedence and which actions are permitted
- Resource and runtime constraints that bound execution
- Evidence and context construction that determine what information reaches the model
- Tool-selection and execution controls that determine which actions occur
- Completion logic that determines when a run terminates
- Evaluation criteria that determine what the system owner classifies as successful
These mechanisms should not be collapsed into a single objective function.
A safety policy is not a reward signal. A timeout is not an instruction. An exit condition is not an evaluation metric. A tool permission is not a model preference.
They operate at different points in the system, but each can cause the system to treat a condition other than the user’s required outcome as sufficient.
3.1 Proxy optimization
A general-purpose language model cannot be trained directly against the complete objective of every task it will later encounter.
During pretraining and post-training, model behavior is shaped through optimization targets such as next-token prediction, supervised instruction-following objectives, preference judgments, reward-model scores, and safety-related post-training signals.
Benchmarks and task-specific evaluations are not optimization targets by themselves. They affect system behavior only when they are connected to model selection, routing, release gates, or further optimization.
None of these mechanisms fully encodes the acceptance criteria of every downstream task.
A response can be:
- Fluent but unsupported
- Relevant but incomplete
- Preferred by evaluators but factually weaker
- Well structured but based on insufficient evidence
- Syntactically correct but operationally invalid
- Persuasive while preserving a defective premise
The underlying issue is proxy-objective mismatch.
A proxy is a measurable signal used in place of a target that is more difficult to observe directly. It may correlate strongly with the intended outcome without being equivalent to it.
Controlled research on reward-model overoptimization provides an empirical example of this problem. Gao, Schulman, and Hilton studied systems optimized against an imperfect proxy reward model and evaluated them against a separate gold-standard reward model. In their synthetic setup, stronger optimization of the proxy could increase proxy reward while reducing performance according to the gold-standard model. [1]
This result does not establish that every deployed AI system works against its users. It demonstrates the narrower principle relevant to this article: improvement against a measurable proxy should not be assumed to equal improvement against the intended objective.
At the application level, proxy mismatch appears when response-level metrics influence model selection, routing, release decisions, or product iteration while task-level outcomes remain unmeasured. That mechanism is addressed in Section 3.6 through the distinction between run termination and outcome evaluation.
3.2 Instruction authority and policy constraints
The user’s instruction is not the only instruction governing a production AI system.
Instruction-precedence models differ across platforms. OpenAI’s published Model Spec dated December 18, 2025 provides one documented example. In that specification, the authority order is:
- Root
- System
- Developer
- User
- Guideline
- No authority
Assistant messages, tool outputs, quoted content, attachments, retrieved documents, and other untrusted data do not become additional authority levels by default. They remain data unless a higher-authority instruction explicitly delegates authority to them. [2]
This distinction matters because evidence and instructions play different roles.
A retrieved document may supply a fact. It does not automatically gain authority to redirect the workflow.
The authority structure also establishes that the user’s objective is only one input into the system’s behavior.
A user request may be constrained or overridden by:
- Platform-level safety requirements
- System or developer instructions
- Organizational governance rules
- Authorization boundaries
- Tenant or data-isolation policies
- Legal or regulatory restrictions
- Human-approval requirements
These controls can be necessary and correct. A user’s requested outcome should not override security, law, authorization, or the rights of other stakeholders.
The conflict arises when the system cannot complete the user’s legitimate objective under the applicable constraints but fails to represent that limitation accurately.
A professional system should distinguish among states such as:
- Completed
- Partially completed
- Refused
- Blocked by policy
- Authorization required
- Human approval required
- Required capability unavailable
A policy-constrained response should not be presented as though the original task was completed without limitation.
The relevant issue is not that user instructions must always prevail. It is that the system must accurately expose when another authority or policy has displaced the user’s requested execution path or outcome.
3.3 Resource and runtime constraints
Production AI systems operate under finite computational and operational resources.
Relevant constraints fall into three operational groups:
- Capacity limits, including context windows, token budgets, model availability, and infrastructure capacity
- Execution limits, including maximum turns, tool-call limits, timeouts, and retry budgets
- Service constraints, including latency targets, rate limits, model-routing policies, and cost thresholds
These constraints are not semantic objectives in the same sense as a reward signal or instruction.
They nevertheless determine how much investigation, retrieval, execution, and verification the system can perform.
A deeper search can improve evidence coverage while increasing latency and cost.
A stronger model may improve task performance while consuming more resources.
Additional agent steps may increase the probability of reaching the intended outcome while also increasing the probability of tool failure, compounding error, or timeout.
Production guidance for agent systems explicitly treats performance, cost, latency, and task complexity as engineering trade-offs. OpenAI recommends establishing a performance baseline with capable models and then replacing them with smaller models where the smaller model continues to produce an acceptable result. [3]
This is a rational production strategy.
It also creates a distinction that matters to the user.
“Acceptable” is a threshold established by the system owner.
The user may assume that the returned result represents the strongest defensible outcome available for the task.
Those standards are not automatically equivalent.
A resource constraint can affect the result through:
- Limited search depth
- Restricted tool usage
- Early termination at a turn limit
- Context truncation
- Routing to a less capable model
- Incomplete validation after a timeout
- Suppressed retries after a tool failure
- Return of a partial result to preserve responsiveness
None of these decisions is inherently improper. Every deployed system requires resource boundaries.
The failure occurs when a material resource constraint changes the completeness, certainty, or reliability of the result, but the system presents the output as though the constraint had no effect.
Users do not require disclosure of every internal routing or infrastructure decision.
They do require disclosure when an operational constraint materially affects whether the result can be treated as complete.
3.4 Evidence and context construction
A model does not reason over all potentially relevant information.
It operates on the information made available in its current context.
That context may contain:
- System and developer instructions
- User messages
- Retrieved documents
- Search results
- Tool outputs
- Conversation history
- Stored memory
- Summaries
- Structured application state
- Truncated or filtered evidence
Context is finite and actively constructed. Anthropic describes context as a limited resource and frames context engineering as the repeated selection of which information should be supplied to the model during inference. [5]
This creates two separate evaluation questions:
- Is the generated answer consistent with the evidence supplied to the model?
- Was the evidence supplied to the model sufficient for the user’s task?
The first concerns grounding and generation quality.
The second concerns retrieval, source selection, coverage, and system-level evidence requirements.
Retrieval quality depends on indexing, ranking, search depth, and context allocation. Failure in any of these stages can leave the model with an evidence set that is relevant but insufficient.
A critical source may be absent from the index.
Search may stop before contradictory evidence is found.
A context limit may remove an earlier requirement.
A summary may preserve the main narrative while discarding a decisive exception.
A tool may return only a subset of the available records.
The model can accurately synthesize the evidence it receives while the system fails to establish whether that evidence is adequate.
This is how partial evidence becomes a complete-looking answer.
For closed-domain tasks, evidence sufficiency may be directly testable. The system can verify that it reviewed every record in a defined dataset, every clause in a supplied document, or every test in a known suite.
For open-domain research, exhaustive completeness is generally not provable. Sufficiency must instead be defined relative to the task through requirements such as:
- Mandatory source classes
- Primary-source requirements
- Recency thresholds
- Jurisdictional or domain boundaries
- Contradiction searches
- Minimum coverage criteria
- Claim-level provenance
- Explicit reporting of unresolved evidence
The model should not be the sole authority on whether the evidence set is sufficient.
Evidence sufficiency must be represented as a task-level system requirement.
3.5 Tool selection and execution control
Tool-enabled and agentic systems introduce a separate execution layer between generated reasoning and real-world action.
Giving the model access to a tool does not guarantee that the tool will be used correctly—or used at all.
The workflow must distinguish among:
- Tool availability
- Tool visibility
- Tool selection
- Tool authorization
- Parameter validation
- Tool execution
- Result validation
- Post-execution verification
Anthropic describes tool use as an interface between deterministic software and non-deterministic agent behavior. Given the same task, an agent may call a tool, answer from existing context, choose a different tool, or request clarification. Anthropic also identifies failures in which agents select the wrong tool, supply incorrect parameters, call too few tools, or mishandle the result. [4]
This flexibility is useful in open-ended tasks.
It is insufficient when a specific tool action is required to establish the result.
A research system that must consult an authoritative database should not merely expose that database as an optional tool.
A coding agent that must validate an implementation should not be allowed to complete before executing the applicable tests.
A status system that depends on current external data should not be permitted to substitute model memory for a system-of-record query.
An automation should not treat an accepted API request as proof that the required business state exists.
A tool that is optional cannot function as a mandatory verification boundary.
Where an action is necessary to establish safety, authorization, evidence, or task completion, the application or orchestration layer must enforce it.
The system must also verify more than tool invocation.
A successful tool call can prove that an execution step occurred.
It does not necessarily prove that:
- The correct parameters were used
- The user was authorized
- The result was complete
- The external system reached the required state
- A downstream process succeeded
- A retry did not create a duplicate action
Tool use is part of the execution path.
It is not, by itself, evidence of outcome completion.
3.6 Completion logic and outcome evaluation
Termination logic. A workflow ending and a user task being complete are different events.
The orchestration layer determines when execution stops.
Possible termination conditions include:
- A model-generated final response
- A designated final-output tool
- A maximum number of turns
- A timeout
- An unrecoverable tool error
- A policy block
- Human escalation
- A validated external state
OpenAI’s practitioner guidance describes agent runs as loops that continue until an exit condition is reached. Example exit conditions include a final-output tool, an error, a maximum-turn limit, or a model response that contains no further tool call. [3]
These are execution conditions.
They do not automatically establish that the user’s acceptance criteria have been satisfied.
If a response without a tool call is a valid terminal state, the system can stop before performing verification.
If test execution is optional, a coding workflow can return a patch without validating it.
If a successful API response is treated as final completion, an automation can report success before the intended external state exists.
The completion rule must therefore be separated from the outcome criterion.
Outcome evaluation. Metrics influence system behavior only when they are connected to a control process, such as:
- Model training or fine-tuning
- Model selection
- Routing
- Release gates
- Prompt or tool iteration
- Regression testing
- Product optimization
- Operational decision-making
A metric that is merely observed does not become an optimization target by itself.
When evaluation rewards fluency, relevance, latency, cost, or successful output generation but does not measure task-level outcomes, system improvement can remain confined to those measured dimensions.
Anthropic distinguishes between an agent transcript and its outcome. The transcript records messages, tool calls, intermediate outputs, and execution details. The outcome is the final state of the environment. An agent may state that a reservation was created, but the relevant outcome is whether the reservation exists in the database. [6]
A transcript can show that the system followed a plausible process.
Only outcome verification can establish that the process produced the required result.
Completion logic and evaluation must therefore be tied to:
- Explicit acceptance criteria
- Required tool actions
- Verifiable postconditions
- External-state checks
- Evidence sufficiency
- Valid blocked and abstention states
- Task-level outcome metrics
The system should not be marked successful because execution ended cleanly.
It should be marked successful only when the conditions defining the user’s authorized outcome have been verified.
4. How the Conflict Appears in Professional Workflows
The same architecture produces different visible failures across domains. Each workflow can be analyzed through four questions:
- What outcome does the user need?
- What proxy can the system treat as sufficient?
- What observable failure results from the mismatch?
- Which control prevents false completion?
4.1 Research
User outcome
A conclusion supported by evidence appropriate to the question, including authoritative sources, material counterevidence, relevant recency requirements, and explicit treatment of uncertainty.
System proxy
A coherent report assembled from a set of relevant retrieved sources.
Observable failure
The report may appear rigorous while:
- Primary sources were not prioritized.
- Contradictory evidence was not searched for.
- Source coverage was not evaluated.
- Outdated and current information were combined.
- Material claims were not mapped to evidence.
- Missing evidence was not disclosed.
- Search stopped when a plausible narrative became available.
The system solved a synthesis problem before establishing that the evidence problem had been solved.
Required control
The workflow needs task-relative evidence requirements, source-quality rules, recency boundaries, contradiction searches, provenance checks, and a valid insufficient-evidence state.
For open-ended research, “comprehensive” cannot be reduced to a universal source count. Anthropic’s agent-evaluation guidance notes that research quality is task-dependent and that concepts such as correctness, comprehensiveness, and source quality must be evaluated relative to the specific use case. [6]
4.2 Writing
User outcome
A document that is clear, professionally written, logically valid, and proportionate to the available evidence.
System proxy
Improved fluency, structure, tone, readability, and persuasive force.
Observable failure
The system can make weak reasoning more convincing.
It can:
- Strengthen an unsupported assertion
- Preserve a false premise
- Smooth over a logical gap
- Convert correlation into causal language
- Remove uncertainty that should remain visible
- Present a one-sided source set as a balanced conclusion
- Improve the prose without testing the argument
The writing becomes stronger while the epistemic quality remains unchanged or deteriorates.
Required control
The workflow must separate writing review from claim review.
Claims should be classified, mapped to evidence, tested for contradiction, and assigned an appropriate confidence level before stylistic optimization is treated as completion.
4.3 Software Development
User outcome
A change that works in the target environment, satisfies the requirement, preserves relevant security and data boundaries, and does not introduce unacceptable regressions.
System proxy
A plausible code diff accompanied by a coherent technical explanation.
Observable failure
The code may be returned without verifying that:
- It compiles
- Type checks pass
- Relevant tests pass
- Existing tests still pass
- The database schema matches the application
- Runtime configuration is present
- Authorization boundaries remain intact
- The deployed environment behaves correctly
- No regression was introduced
Code generation is an intermediate artifact.
A verified working state is the outcome.
Required control
The workflow should enforce applicable build, lint, type-check, unit-test, integration-test, migration, security, and environment-validation steps.
Completion should depend on evidence from those checks, not on the model’s explanation of why the implementation should work.
4.4 Decision-Making
User outcome
A decision or recommendation based on sufficient evidence, explicit assumptions, relevant alternatives, stakeholder constraints, and a defensible treatment of uncertainty.
System proxy
A clear, coherent, and decisive recommendation based on the information currently available.
Observable failure
The recommendation may omit:
- Material alternatives
- Unverified assumptions
- Current legal or regulatory constraints
- Full cost implications
- Relevant downside scenarios
- Hidden dependencies
- Stakeholder conflicts
- Evidence that would reverse the conclusion
A coherent recommendation can conceal an incomplete decision process.
Required control
The workflow should require assumption registers, evidence boundaries, alternative analysis, material-risk checks, source recency, stakeholder constraints, and escalation when the decision exceeds the available evidence or the system’s authority.
4.5 Automation and Agentic Operations
User outcome
A verified transition from the initial state to the intended authorized state in an external system.
System proxy
A sequence of apparently successful tool calls and a final confirmation message.
Observable failure
The agent may:
- Call the correct tool with the wrong parameters
- Receive success from an intermediate service while the downstream action fails
- Retry a non-idempotent operation and create duplicates
- Complete only part of a multi-step transaction
- Act outside the user’s authorized resource scope
- Report completion without checking the final state
- Leave the system in an inconsistent state after a partial failure
NIST’s work on tool use in agent systems distinguishes among tool functionality, access patterns and permissions, trusted and untrusted environments, reversibility, stateful effects, reliability, monitoring, and autonomy. These characteristics materially affect both reliability and risk. [8]
Required control
The workflow needs authentication, authorization, scoped permissions, parameter validation, idempotency, transaction or compensation logic, postcondition verification, failure recovery, and an inspectable audit trail.
A tool response is evidence about an execution step.
It is not automatically evidence that the user’s business objective was achieved.
5. Engineering the System Around the User’s Real Goal
The conflict cannot be resolved by adding “be accurate,” “double-check,” or “do not hallucinate” to a prompt.
A prompt can influence model behavior.
It cannot independently guarantee that:
- Required evidence enters the context
- A tool is called
- The user is authorized
- The tool succeeds
- The final state is correct
- A timeout does not terminate execution
- The system stops when evidence is insufficient
- The evaluation measures the actual outcome
The user’s objective must be represented across the architecture.
5.1 Define Task-Level Acceptance Criteria and Postconditions
Before execution, the workflow should define what successful completion means.
Relevant criteria can include:
- The required end state
- Mandatory output properties
- Required evidence
- Authorized sources or systems of record
- Required actions
- Required validation steps
- Acceptable uncertainty
- Invalidating conditions
- Human-approval requirements
- Stop and escalation conditions
- Observable postconditions
Together, these requirements form an outcome contract: a testable representation of the result the user is authorizing the system to pursue.
The contract must be bounded.
“The best possible answer” is not an executable completion criterion. It has no finite stopping rule.
A technically useful target is:
The strongest defensible result that satisfies the user’s acceptance criteria within disclosed and authorized constraints.
5.2 Separate Response Criteria from Task Criteria
Response criteria evaluate the artifact returned to the user:
- Clarity
- Relevance
- Structure
- Tone
- Format
- Citation presentation
- Schema validity
Task criteria evaluate whether the work was completed:
- Required evidence was collected
- Mandatory sources were checked
- Required tools were executed
- Tests passed
- External state was verified
- Authorization was confirmed
- Material uncertainty was disclosed
- Postconditions were satisfied
Both categories matter.
Only task criteria establish task completion.
5.3 Define Evidence and Provenance Requirements
Evidence controls should specify what information the system is required and permitted to use.
Depending on the workflow, this can include:
- Approved source classes
- Required primary sources
- Systems of record
- Recency thresholds
- Jurisdictional scope
- Authority ranking
- Claim-level provenance
- Contradiction-search requirements
- Minimum coverage rules
- Treatment of unavailable evidence
- Conditions requiring human review
The workflow should distinguish between:
- No evidence was found.
- The required source was not searched.
- The source was unavailable.
- Evidence exists but is contradictory.
- The evidence supports only a partial conclusion.
- The evidence meets the defined completion threshold.
These are different system states and should not collapse into the same generated answer.
5.4 Enforce Critical Actions Without Overconstraining Valid Paths
When a specific action is necessary to establish the result, the workflow must enforce it.
Examples include:
- Running tests before closing a coding task
- Querying the system of record before reporting status
- Checking current sources for time-sensitive claims
- Reading the complete supplied document before reviewing it
- Verifying identity before processing a financial action
- Confirming external state after an automated update
Mandatory controls should focus on task-critical invariants, not unnecessarily rigid execution traces.
Anthropic’s agent-evaluation guidance warns that grading an exact sequence of tool calls can be brittle because agents may discover valid paths that designers did not anticipate. It recommends evaluating outcomes while using required tool-call checks where particular actions are genuinely necessary. [6]
The correct principle is therefore:
Enforce the steps that establish safety, authorization, evidence, or completion.
Preserve flexibility where multiple valid execution paths can satisfy the same postconditions.
5.5 Add Validation and Completion Gates
A completion gate prevents the workflow from returning a completed state until required checks pass.
Possible gates include:
- Unit and integration tests
- Static analysis
- Schema validation
- Database-state checks
- Source-grounding checks
- Coverage checks
- Permission checks
- Independent calculations
- Reconciliation against a system of record
- Human approval
- Calibrated model-based grading
Where deterministic validation is available, it should carry more evidentiary weight than model self-assessment.
A second model is not independent verification by default. Two models can share training data, retrieval errors, prompt framing, and blind spots.
Model-based graders can be useful for open-ended qualities such as reasoning, clarity, or source use, but they should be calibrated against deterministic checks or qualified human judgment. Anthropic’s evaluation guidance explicitly distinguishes code-based, model-based, and human graders and notes that model-based graders require human calibration. [6]
5.6 Implement Stop, Abstain, Blocked, and Escalation States
A workflow needs valid alternatives to returning a final answer.
These should be first-class execution states rather than improvised phrases in generated text.
Examples include:
- Completed
- Partially completed
- Blocked
- Insufficient evidence
- Required source unavailable
- Tool failure
- Authorization required
- User clarification required
- Human approval required
- Escalated
- Abstained
- Timed out
- Recovery required
When final-answer generation is the readily available terminal path and non-completion states are missing or weakly enforced, the system is structurally more likely to return an answer than to represent unresolved work accurately.
The correct behavior is not always completion.
In professional systems, stopping can be the successful behavior when the evidence, permission, or execution state does not support continuation.
5.7 Enforce Authorization and Safe State Transitions
Tool use and task completion are separate from authorization.
A system may know how to perform an action without being authorized to perform it for the current user, tenant, resource, or transaction.
Before state-changing actions, application code should enforce:
- Identity
- Authentication
- Authorization
- Tenant and resource scope
- Action-specific permissions
- Approval requirements
- Data-access boundaries
For stateful workflows, reliability also requires:
- Idempotency for retryable actions
- Atomic operations where feasible
- Compensation or rollback for partial failure
- Explicit intermediate states
- Duplicate-action prevention
- Recovery procedures
- Postcondition checks
These controls belong in deterministic application and infrastructure layers.
They should not depend solely on the model’s interpretation of natural-language instructions.
5.8 Evaluate the Outcome, Not Only the Transcript
Agent evaluation should inspect both execution behavior and final state.
Anthropic defines a transcript as the record of messages, tool calls, intermediate outputs, and other interactions. It defines the outcome as the final state of the environment. It recommends combining code-based, model-based, and human graders according to the task. [6]
Depending on the workflow, outcome evaluation can verify:
- Whether the file contains the required change
- Whether tests pass
- Whether the database was updated correctly
- Whether the transaction exists
- Whether the refund was processed
- Whether claims are grounded in the required sources
- Whether mandatory tools were used
- Whether authorization checks occurred
- Whether the system stopped under insufficient evidence
- Whether the intended external state was reached
Non-deterministic agents should also be evaluated across multiple trials. A task that succeeds once may fail on another run. Single-run success is not necessarily a reliable estimate of system performance. [6]
The central evaluation question is not:
Did the system produce a good answer?
It is:
Did the complete system reliably achieve the authorized outcome under the conditions in which it will be deployed?
5.9 Add Observability and Auditability
A system cannot be reliably reviewed if its decisive actions are not inspectable.
Professional workflows should record, subject to privacy and security requirements:
- Model and workflow version
- Applicable instructions and policy version
- Retrieved source identifiers
- Tool calls and validated parameters
- Authorization decisions
- Tool errors and retries
- Completion-gate results
- Human approvals
- Material limitations
- Final external state
- Recovery or compensation actions
Observability supports debugging, evaluation, incident investigation, compliance review, and regression detection.
It also makes conflicts visible.
A team can determine whether a task failed because of model behavior, retrieval failure, missing authorization, a tool defect, a timeout, an incorrect exit condition, or an evaluation gap.
Without that traceability, system failure can be misclassified as user error or dismissed as model unpredictability.
5.10 Disclose Material Limitations
AI systems will continue to operate under safety, product, cost, latency, and infrastructure constraints.
The objective is not to eliminate those constraints.
It is to prevent material constraints from disappearing inside an output presented as complete.
The system should disclose limitations that affect how the result must be interpreted, such as:
- Incomplete source coverage
- Unavailable required tools
- Unresolved contradictory evidence
- Execution timeout
- Partial workflow completion
- Unverified external state
- Provisional conclusions
- Remaining human-review requirements
- Policy or authorization limitations
Not every routing decision or implementation detail requires user-facing disclosure.
The threshold is materiality: would knowledge of the constraint reasonably change the user’s reliance on the result or their belief that the task was complete?
6. Conclusion: From acceptable output to verified outcome
The conflict examined in this article is not limited to hallucination, and it does not originate in a single component. It emerges when a legitimate user objective is mediated by model-level proxies, instruction authority, operational constraints, context selection, tool policies, and termination rules that define success differently from the task itself.
The central distinction is between producing an acceptable output and establishing a completed outcome. A response may be fluent, factual, policy-compliant, and correctly formatted. A run may execute without error and reach a valid terminal state. Neither condition demonstrates that the required evidence was obtained, the necessary actions were performed, or the intended external state was reached.
For the user’s objective to govern completion, it must be represented in the architecture as explicit acceptance criteria, required evidence and actions, enforceable authorization boundaries, verifiable postconditions, valid non-completion states, and outcome-based evaluation. When those conditions cannot be established, the system should report the precise limitation, unresolved condition, or blocked state instead of implying success.
Safety, legal obligations, authorization boundaries, cost, latency, and other system constraints remain legitimate. The professional requirement is not to remove them, but to make their effect on task completion explicit and to prevent a constrained or partial result from being presented as a verified outcome.
The correct standard is therefore not whether the system produced a convincing response or completed a run. It is whether the authorized user objective was achieved and verified.
A convincing response is an output.
A completed task is a verified state.
References
-
Gao, Leo, John Schulman, and Jacob Hilton. “Scaling Laws for Reward Model Overoptimization.” Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023.
-
OpenAI. “Model Spec.” December 18, 2025.
-
Anthropic. “Writing Effective Tools for Agents — With Agents.” September 11, 2025.
-
Anthropic. “Effective Context Engineering for AI Agents.” September 29, 2025.
-
Anthropic. “Demystifying Evals for AI Agents.” January 9, 2026.
-
National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.” NIST AI 600-1, July 2024.
-
National Institute of Standards and Technology. “Lessons Learned from the Consortium: Tool Use in Agent Systems.” August 5, 2025.