Coding-Agent Planning: Accuracy and Cost as Separate Outcomes

Does asking AI to plan first produce better work? Enabling a planning feature might raise its success rate. It might also lower cost while slightly lowering success. Which outcome counts as an improvement depends on what failure costs in the assigned task.

Released on September 17, “An Empirical Study of Harness Design for Coding Agents” separately examines planning, tools, and context management across four models. This article focuses on planning. Completing more tasks successfully and completing attempts more cheaply are separate outcomes. Paper and publication history

A persistent progress list, rather than a one-off plan

Planning in this study is not simply an initial long explanation. The model updates a plan through a tool and receives that plan again as execution continues. Disabling planning removes the instructions, reminders, tool, and repeated plan input together. The result concerns this implementation of persistent planning. It does not directly measure a particular product’s planning mode or the phrase “think step by step.” Planning implementation

A progress list offers a useful analogy. Reading a task list once at a meeting differs from revisiting completed, active, and unfinished items before every next action. The latter can affect judgments about omissions and completion. This analogy explains the difference in mechanism; it does not establish that adding such a list must improve performance.

Changing planning while holding other conditions fixed

The following comparison comes from software-issue tasks in SWE-Bench Verified. The context budget remains 128k, using the paper’s T4 context management and predefined tool set. T4 first reduces old tool observations, summarizes when needed, and retains recoverable copies of omitted records.

Model	Success rate: off → on	Change	Mean cost per task: off → on
Nemotron-3 30B	13.6% → 25.2%	+11.6 pp	$0.02 → $0.09
Nemotron-3 120B	46.6% → 44.0%	−2.6 pp	$0.25 → $0.34
Nemotron-3 550B	67.8% → 65.8%	−2.0 pp	$3.31 → $2.33
Mistral Medium 3.5	69.0% → 68.6%	−0.4 pp	$4.65 → $3.14

These values come from Table 3. Cost applies the researchers’ chosen token prices to usage; it is not API spending measured for this article. “pp” means percentage points, the difference between two rates. Small differences in this table should not all be treated as statistically established improvements or regressions.

The 30B model succeeded more often at higher cost. Under this condition, 120B improved on neither measure. The 550B and Mistral models cost less without raising success rates. These outcomes do not support a blanket claim that planning makes agents both more accurate and cheaper.

Why the same feature shortened different parts of the work

The researchers examined execution traces as well as scores. They interpret planning as reducing termination before any edit for the smallest model, while primarily shortening post-edit verification for the larger two. Execution analysis

In the first case, continuing longer was necessary. An attempt that ends without starting the required modification is not necessarily efficient because it used fewer tokens. In the second case, deciding when enough verification has been done becomes important.

Shorter verification alone does not establish that every removed check was unnecessary. The table does not show a simultaneous rise in success rate. To apply this to my work, I would separate repetitions of the same check from different checks aimed at different failures.

Checking a calculated total and checking saved state after reload both use a browser, but they catch different errors. Capturing the same screen repeatedly with unchanged code and inputs may add no new evidence. In the app-recreation example, the initial-screen check missed both calculation and persistence faults.

Recalculating cost per successful result

Cheap attempts can still leave many tasks unfinished. The published averages support a further calculation:

Allocated cost per successful result
= cost of all attempts / number of successes
= mean cost per task / success rate

Using the rounded table values, this changes from approximately $0.15 to $0.36 for 30B and from $4.88 to $3.54 for 550B. It allocates the cost of failed attempts across the number of successful results. It is not a prediction of how much repeated attempts on one task would cost before success.

The higher ratio for 30B does not automatically justify disabling planning. Resolving more tasks may be worth the added cost. Nor does the lower ratio for 550B justify immediate adoption: it does not identify which tasks failed or account for additional human review.

This calculation does not produce a model ranking. It provides a recording rule: track attempt cost, successful deliverables, and recovery costs together. Fewer reported tokens do not, by themselves, establish a cheaper overall workflow.

Keeping planning, context, and tools separate

The paper’s “harness” is the software connecting the model to tools, work history, and its execution loop. Changing that surrounding setup can change how the same model works.

If I enable planning while also summarizing history and removing tools, a different result will not reveal which change caused it. For a comparison in my own workflow, I would first keep the model and task copies fixed and change only the planning condition. This is a proposed comparison derived from the review, not a model experiment already performed.

Hold fixed	Record
Model version, initial files, request, tools	Success and failed checks
Context limit and maximum execution budget	Tokens and cost per task
Predefined final checks	Termination before editing or verification
External service and data state	Human intervention, corrections, and rerun time

Passing the conversation from the planning-enabled run into the next run would change the conditions. Each needs a separate copy of the initial state. One success or failure is insufficient; the effect should also be examined across task types and repeated runs.

Evidence for marking a plan complete

In practice, I would ask for more than headings such as “analyze, implement, test.” Each stage should include evidence needed to mark it complete. The following brief was composed after reading the paper.

List each task with its completion condition.
Connect each implementation item to inputs and expected outcomes.
Keep failed checks and rerun them after corrections.
Before repeating a check, identify any change to code, input, or environment.
Stop once the checks required after the final edit have passed.
Keep unexecuted checks separate from completed work.

A longer plan is not the objective. The plan should help address premature termination and repeated actions after verification is complete, while preserving necessary checks.

The research is a preprint covering four models and two evaluation environments. Its findings cannot be applied to every coding agent. The narrower criterion I take from it is this: evaluate planning through where execution stopped and what was verified, rather than how much planning text was produced.

Sources checked and published figures recalculated on September 21, 2026, using the v1 paper. No model runs or performance remeasurement were performed. Cost ratios use rounded published values.