Failure as a Process: A New Way to Debug CLI Coding Agents

What the paper actually studies

Most reliability write-ups about coding agents stop at a single label: the run passed or it failed. That tells you how often to worry, but nothing about *when* things went sideways or whether a nudge a few steps earlier would have rescued the task.

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories, an arXiv preprint from Xiangxin Zhao and six co-authors, treats failure as a timeline instead of a verdict. The authors describe it as the first large-scale empirical study of CLI coding-agent failure trajectories, using a framework that breaks a run into three phases: the onset where things first go wrong, the evolution where the error compounds, and the point where recovery is no longer possible.

The scale is worth noting. They collected 3,843 execution trajectories from seven frontier models running across three agent scaffolds — OpenHands, MiniSWE, and Terminus2 — on Terminal-Bench. After filtering for complete, valid runs, they manually annotated 1,794 trajectories covering more than 63,000 execution steps, and derived 14 findings spanning failure occurrence, root causes, recovery, and consistency across systems.

Manual annotation at that step count is the interesting part. It is slow, expensive work, and it is what lets the study talk about the moment of failure rather than just the outcome.

The findings that matter for builders

Three headline claims come out of the analysis, and they line up with what many people already suspect from watching agents work.

First, failures are predominantly driven by epistemic errors — the agent believing something false about the task, the codebase, or the environment, rather than, say, running out of tokens or hitting a tool bug. That reframes reliability as a reasoning-and-grounding problem more than an infrastructure one.

Second, failures typically begin within the first few execution steps. The agent misreads the task or forms a wrong plan early, and much of what follows is elaboration on a bad premise.

Third, and most useful, those early errors often stay hidden until recovery is no longer possible. The run looks fine for a while. By the time the failure is visible in the output, the trajectory has already drifted past the point where the agent can correct itself.

The authors' own conclusion follows directly: improving reliability requires earlier validation and intervention, not just scoring the final outcome. If the fatal mistake happens in step three but only surfaces at step forty, a pass/fail metric measured at step forty tells you nothing about where to place a guardrail.

A fair caveat: this is a preprint, the findings come from Terminal-Bench tasks rather than arbitrary production workloads, and "epistemic error" is a category that depends on how the annotators drew lines. The specific step-count numbers should be read as evidence for a pattern, not as a universal constant. But the direction of the argument is hard to dismiss.

Why this is useful if you build or supervise agents

If you run a coding agent in an automation — say, a pipeline that dispatches a fix to a repo, or an agent triggered from an issue tracker through n8n or Make — the practical takeaway is about *where* to spend your supervision budget.

A pass rate tells you how often to babysit. A failure anatomy tells you where a checkpoint would earn its keep. If most fatal errors form in the opening steps, then a validation gate after the agent states its plan is worth far more than one after it has written forty commands. You are checking the premise, not the aftermath.

Concretely, that suggests a few patterns worth trying with agents built on the same models this study used — the OpenAI and Claude families both appear in the frontier-model bracket:

• Force the agent to restate the task and its intended plan before it touches the file system, and validate *that* against the actual requirements.
• Add an early sanity check on the agent's understanding of the environment — its read of the repo layout, the failing test, the target file — since a wrong mental model here is exactly the epistemic error class.
• Treat the first few steps as the high-leverage window for a human-in-the-loop approval, rather than reviewing a long transcript after the fact.

The study also checks cross-system consistency across three scaffolds, which matters because it suggests these patterns are not an artifact of one tool's quirks. That is reassuring if you are choosing between agent frameworks and worried the failure modes are wildly different across each.

What the paper does not give you is a drop-in detector — it is an anatomy, not a product. The work of turning "errors start early and hide" into a concrete validation step in your own pipeline is left to you. That is the gap between a reliability metric and a reliability practice.

If you want help designing early-checkpoint guardrails for a coding agent you are running in production, browse the provider directory to find people who can put this into practice.