The Agent Should Not Decide When It Is Done

A green completion status can prove that the work stopped. It cannot prove that the decision is finished.

TL;DR

An AI agent can complete an assigned task, produce evidence of execution, and report that its run ended without errors. Those facts do not establish that the resulting outcome is acceptable.

The distinction matters because completion and acceptance belong to different parts of the decision system. The agent can operate within defined authority and provide a durable receipt. Acceptance remains a judgment call when the outcome carries ambiguity, tradeoffs, policy implications, or business consequence.

The article argues that trustworthy agent systems need a visible boundary between execution, evidence, acceptance, and escalation. Without that boundary, capability quietly becomes authority.

Completion Was Green

I woke up to a completed agent run.

The status was green. There were no errors. The overnight work had finished, and the system had left a record of what it had done. Nothing had crashed. Nothing was waiting in an obvious retry state. The run looked clean.

I opened the output.

The task had completed. The decision had not.

That distinction was visible almost immediately. The agent had gathered the material, applied the rules it had been given, and produced the expected artifact. It had done what the run was designed to do. It had not established whether the result was acceptable for the decision that depended on it.

This is where most agent conversations become imprecise. A completed run gets treated as a successful outcome because the system has only one visible terminal state. The work is either running or finished. The tile is either green or red. The agent either returns an answer or reports a failure.

That binary is convenient. It is also inadequate.

The green status told me that the execution path had reached its end. It did not tell me whether the output represented the current source of truth, whether the interpretation was sound, whether the tradeoff was acceptable, or whether I was authorized to let the result change what happened next.

Green tiles lie when they collapse those questions into one signal.

The earlier observability work had already exposed one version of this problem. On August 26, overnight work ran and left proof. The canonical report surface was stale. A brief could have been locked on a version of current that looked authoritative and was not.

The dashboard did not save me from work. It saved me from locking a brief on a surface I would have sworn was current.

That was the point of the control surface. It made freshness, execution, and evidence visible before I treated the output as a basis for action. It separated a valid receipt from a confident assumption.

The next boundary was harder.

Even valid proof that the agent executed correctly did not mean the outcome should be accepted.

The Work Finished Before the Decision

The first mistake is to ask whether the agent finished.

The more useful question is what the agent was authorized to finish.

Those are not the same question.

A mechanical task can have a clear terminal condition. A file was created. A report was refreshed. A set of records was reconciled. A calculation passed its validation. A brief was assembled from the approved sources. These outcomes can often be tested with evidence that is independent of the agent’s own statement.

The agent can retry. It can correct a malformed output. It can compare a result against a known constraint. It can attach the relevant receipt. It can show which source version it used and when the work happened.

That is valuable. It is also bounded.

The agent can prove that the task reached its mechanical stop condition. It cannot turn that proof into permission to decide that the business outcome is acceptable.

In my case, the overnight run had done exactly what I had asked it to do. That was part of the problem. I'm like, yes, the task ran. The request had been expressed as a task, but the real work contained a decision. The task had a completion state. The decision had an acceptance state. I had allowed the first to stand in for the second.

The system had completed the work. The system had not completed the decision.

That gap is easy to miss when the output is plausible. Plausibility is especially dangerous here because it feels like competence. The answer is coherent. The evidence exists. The status is green. The agent can explain what it did in a tone that sounds settled.

Chat confidence is not proof. Proof of execution is not acceptance.

The sequence needs more than one terminal state. It needs a visible distinction between a result that has been produced and a result that has been accepted for the consequence attached to it.

The Misdiagnosis Was More Control

The natural response is to add more control.

Add another approval. Require another checkbox. Force the agent to write a longer explanation. Put a person in the loop for every meaningful output. Ask the system to confirm that it is sure.

That response sounds responsible because the underlying risk is real. It also misses the architectural problem.

The issue is not that the system lacks enough visible restraint. The issue is that it does not distinguish different kinds of authority.

Binary governance gives us two familiar states: locked down and trusted. In the first state, the agent cannot do much without permission. In the second, it is assumed to be reliable enough to proceed. Everything important gets pushed toward one side or the other.

That leaves the missing middle unmodeled.

The missing middle is where an agent can execute within deliberate scope, prove what it did, and stop when the next move requires a judgment it does not own.

This is not a weaker version of autonomy. It is a more precise assignment of authority.

The agent might be authorized to collect the current inputs, compare them against a defined rule, identify exceptions, produce a draft, and record the evidence. It might not be authorized to accept an unresolved tradeoff, reinterpret a policy, choose between competing business outcomes, or convert an ambiguous result into a committed decision.

Those boundaries are not all about risk in the abstract. They are about who gets to say that the outcome is acceptable.

That question often gets hidden inside language such as “complete the report,” “finalize the recommendation,” or “resolve the issue.” Each phrase combines work with judgment. The agent can perform the work. The organization still needs to decide whether the resulting state is one it is willing to own.

More controls do not fix a merged boundary. They often make it harder to see.

The system ends up with a large amount of evidence surrounding a decision that still has no explicit owner. Review becomes ritual. Approval becomes a response to a green status rather than a judgment about the result. The human remains technically present while the system starts setting the terms of acceptance.

That is how capability becomes authority without anyone making the decision directly.

The Missing Middle Is Acceptance

The missing middle is not another dashboard tile. It is the space between execution and acceptance.

A trustworthy control surface should make those states visible as different facts.

Execution answers: Did the agent perform the assigned work?

Evidence answers: What receipt allows someone else to verify that claim?

Acceptance answers: Has an accountable person or system judged the outcome acceptable for its intended use?

Escalation answers: What condition prevents acceptance from being automatic?

Each question has a different owner.

The agent can often answer the first. It can help produce the second. The third belongs to whoever carries the consequence of using the result. The fourth belongs in the system wherever ambiguity, exception, or unresolved tradeoff appears. Agents produce noise cheaply. Judgment decides what becomes true and whether the outcome is acceptable.

This is why stop conditions are decision rights. They are not merely instructions for when a process should terminate. They define what the agent is allowed to treat as settled.

A stop condition can say that the required sources were available, the transformation completed, the output passed its mechanical checks, and the evidence was recorded. That gives the agent a real boundary. It does not grant authority to declare the outcome acceptable when acceptance depends on a human reading of context.

The question is not whether the agent should stop. The question is what it is stopping from doing.

An agent may stop gathering data while leaving acceptance open. It may stop iterating after a defined test passes while marking the result for review. It may stop because a required condition is unresolved and escalate instead of filling the gap with its own interpretation.

That is a complete execution state without a completed decision.

The distinction also clarifies what evidence is for. Evidence is not a performance of confidence. It is a durable receipt that lets the next authority inspect what happened, against which version, under which rules, with which unresolved conditions.

The receipt should make it possible to see whether the agent was authorized to complete the task, whether it did so, and what remains outside its authority.

A green result with no acceptance boundary invites overreach. A red result for every unresolved judgment invites unnecessary intervention. The useful state is more specific: execution proved, acceptance pending; or execution blocked, escalation required; or outcome accepted by the accountable authority.

Done Is a Decision Right

The phrase “the agent is done” sounds operational. It is actually a claim about authority.

It says the work has reached a point where no further action is required from the system. Sometimes that is true. Sometimes it means only that the system has reached the end of what it knows how to verify.

Those are different conditions.

If the agent is allowed to decide when it is done, then someone has already given it authority to define the relevant endpoint. That authority may be intentional. More often, it arrives through an underspecified task, a broad success message, or an interface that offers no state between complete and failed.

The agent fills the gap.

It decides that the output is sufficient because the requested fields are present. It decides that the evidence is sufficient because the checks it knows about passed. It decides that no escalation is needed because nothing in its instructions describes the unresolved condition. It stops, and the system interprets the stop as acceptance.

The problem is not that the agent made a bad choice in the ordinary sense. The problem is that the agent was placed in a position where choosing was unavoidable, even though the choice had not been assigned.

An agent can decide how to execute within scope. It can select among permitted mechanical paths. It can retry a failed operation. It can correct an output that violates a stated condition. It can stop when the evidence meets an external acceptance rule.

It should not decide that the outcome is acceptable merely because it has no further mechanical action to take.

That line matters most when the output changes what a team believes, commits to, or does next. A completed recommendation can influence a roadmap. A refreshed report can become the basis for a leadership brief. A resolved record can suppress a signal that should have remained visible. A generated answer can turn an unresolved question into an apparently settled one.

The agent may have completed the production of the artifact. The organization has not necessarily accepted the consequence.

This is the same decision architecture problem that appears when every meaningful question routes back to one founder or executive. The surface looks efficient because the work moves. Underneath, authority remains unclear. In the agent case, the system creates a new version of the same bottleneck by letting the tool absorb judgment that still needs an accountable owner.

The goal is not to keep the human involved in every action. That would confuse judgment with inspection and create a slower version of the same broken system.

The goal is to preserve human judgment where the system cannot honestly substitute for it.

Capability Must Not Become Authority

The practical consequence is not that agents should remain passive.

An agent should be able to do substantial work. It should be able to run overnight, operate across systems, retry mechanical failures, produce durable evidence, and return with a result that is easy to inspect. That is the point of using one.

The constraint is narrower and more important. Capability must not slide into authority.

The agent can prove that it finished the task. The system must still establish whether the outcome is acceptable.

That requires a control surface that does not hide the difference. A useful surface would show the task the agent was authorized to complete, the evidence attached to the completion claim, the acceptance state, and the condition that requires escalation when acceptance cannot be inferred.

It would not treat every unresolved judgment as a system failure. It would not treat every green status as permission to proceed. It would show where execution ends and where accountability begins.

This also changes how shared rules should be written. A rule cannot simply say “complete the task.” It needs to make clear what complete is allowed to mean in that context.

Does complete mean the artifact exists? Does it mean the source data was fresh? Does it mean the output passed a mechanical test? Does it mean an accountable leader accepted a tradeoff? Does it mean a downstream system can act without another review?

Those are different terminal conditions. If they are left merged, the agent will choose one.

The answer will usually be the condition easiest to observe, not the condition most important to the business.

That is why the previous layers matter. Shared rules prevent the standard from changing when the model changes. Durable receipts show what actually happened. Observability prevents a stale surface from impersonating current truth. Acceptance defines what the evidence is allowed to support.

Together, they create a system in which the agent can be fast without being the final judge of its own work.

The Boundary Holds

I still want the overnight run to finish without me watching it.

I still want the system to retry what can be retried, correct what can be corrected, and leave a receipt that lets me understand the result in the morning. I do not want a person manually confirming every mechanical step simply because the system cannot distinguish execution from acceptance.

But I also do not want a green status to settle a question that carries a decision.

The morning test is simple in principle. I should be able to see what the agent completed, what evidence proves it, what remains open, and who has the right to accept the result. If those states collapse into one green answer, the system is asking the agent to supply judgment that was never actually delegated.

That is not autonomy. It is authority by omission.

The agent should be able to say, “The work is complete and here is the proof.”

The system, and the accountable human inside it, must still decide whether that is enough.

Key Takeaways

An AI agent can establish that an assigned task reached its mechanical completion condition. It cannot establish by itself that the outcome is acceptable when acceptance depends on context, tradeoffs, or accountable judgment.

A trustworthy control surface separates execution, evidence, acceptance, and escalation. These states should not be collapsed into a single green or red status.

Stop conditions define decision rights. They determine what the agent may treat as settled and what must remain open for human review.

The missing middle between locked-down automation and fully trusted autonomy is deliberate scope. In that space, the agent can do meaningful work while remaining outside the authority to accept its own consequential outcome.

The central risk is not only incorrect execution. It is the quiet conversion of capability into authority.

Frequently asked questions about AI agent completion

What is the difference between task completion and outcome acceptance?

Task completion means that the agent performed the assigned work and reached the defined execution condition. Outcome acceptance means that an accountable person or system judged the result suitable for its intended consequence. The first can often be supported by mechanical evidence. The second may require context and judgment.

Can an AI agent decide when to stop working?

An AI agent can stop when externally defined execution conditions are satisfied, such as a successful validation, a completed transformation, or a recorded artifact. It should not use its own stop signal to decide that a consequential outcome is acceptable unless that authority has been explicitly assigned and the acceptance condition is independently verifiable.

Why is a green status not enough?

A green status usually proves that a run completed without a reported error. It does not necessarily prove that the source was current, the interpretation was correct, the scope was authorized, or the outcome was acceptable for the next decision. Green indicates execution status, not complete accountability.

When should an AI agent escalate?

An AI agent should escalate when a required condition is unresolved, when the result falls outside its authorized scope, when evidence is stale or incomplete, or when acceptance depends on a judgment that has not been delegated. Escalation is a valid terminal state for execution when the next decision belongs elsewhere.

Does separating acceptance from execution reduce agent autonomy?

Separating acceptance from execution does not require the agent to be passive. It allows the agent to operate independently within a defined scope while keeping consequential judgment with the accountable authority. The result is narrower authority, not necessarily less useful work.

I help product leaders at complex product organizations unblock execution when their decision architecture starts breaking down, so that they can ship the roadmap they committed to without another quarter of explanation.

If this sounds familiar, you're not alone.

The work is not about moving faster. It is about preserving judgment as systems scale.

If you are navigating agent systems where green status is standing in for acceptance, book a Relevance Check. We will walk through what you can move first.

No pitch. Just the read.

Clinton Pracher | CP Product Advisory