Blog

9 min read

Agentic workflow v2: why I rebuilt it and how many errors it stopped

An audit of my own agent usage showed where the workflow was failing. V2 reviews the plan and the code on every task and runs in waves. Here are the errors it stopped, how I counted them and what it cost.

  • agents
  • workflow
  • claude
  • gpt
  • code-review
  • pi

The first post about my workflow came out on November 9. Five days later, I decided to measure whether that workflow actually worked. It did, but the filter that found the most problems only ran on some of the tasks.

This is v2. The core idea is the same: Claude Opus 5.5 plans with me, GPT agents execute in isolated worktrees, and nothing lands on the branch without review. Almost everything else changed, and now I have numbers on what that stopped.

344

tasks delegated in v2, Nov 14 to 22

646

errors that would have gotten through, at most

≈274

would show up in normal use (range: 176 to 374)

514

spec errors stopped before any code existed

What motivated the change

On November 14, I audited 30 days of my Claude Code usage: 188 sessions, 356 cumulative active hours and the records of every delegation. Three things stood out.

The filter that found the most problems only ran on some of the tasks. The reviewer requested changes in 56 of 98 reviews (57%), with a median of 32 seconds each. Even so, the rule kept it for large diffs or sensitive areas.

The spec was blocking the executor. In 79 of 493 reports (16%), the executor came back with UNSURE: because of a contradiction or gap in the spec itself.

The same kinds of failure kept coming back. In one large project, there were 61 continue runs per 100 tasks, against 40 on average in the others. About 47% fixed a reviewer finding, nearly always edge-case logic: races, remounting components, stale data, queue limits. In several cases, the tests passed without catching the bug. Another 20% were tasks abandoned halfway, and 30% were decisions the spec hadn't covered.

So the problem wasn't only the model. It was the model, weak tests and holes in the spec.

The second motivation came a week later, and that one was on me. Measuring the sessions, from November 14 to 17 I wrote 100 to 360 words per design, with decisions that changed what got built. From the 19th to the 21st, it was 5 to 18 words: "let's go with A", "go ahead", "I'll take the suggestion". The orchestrator arrived with a recommendation and I confirmed it. This after I had said, days earlier, that my main focus is the planning stage.

On top of that, the orchestrator was spending its context reading whole diffs, when its job is something else. And we weren't parallelizing much: of 76 recorded rounds, 45 had a single task.

What changed

First post (Nov 9)v2 (Nov 22)
ExecutorGPT-6.1 Terra, xhighGPT-6.2 Sol, high
InvestigationGPT-6.1 Sol, highGPT-6.2 Astra, high
Front-endthe executorfrontend-engineer: Sol 6.2 with a UI prompt
Queriesthe executorscout: Terra 6.1, read-only
Code reviewGPT-6 Astra, relevant diffs onlyGPT-6.2 Astra, every task, three layers
Plan reviewdidn't existClaude Opus 5.5, xhigh, every round
Specno edge casesrequired Edge cases and a Bite check
Executionparallel when it was obviousgated waves, up to 4 concurrent tasks
Orchestratorreads the diff and fixes small thingsreads reports and writes no code
Entry pointI drove each step/implement, with three stops for me

How many errors would have gotten through

This is the question I wanted to answer: if I had accepted what the executor delivered as done, how many errors would have reached the code?

None of these errors were reproduced. The numbers come from the reviewers' reports in the sessions and in each delegation's artifacts, 666 reviews in v2 alone. I split them by workflow phase:

PhaseTasksFlaggedWould passNormal useConfirmedSpec
Nov 1–7: no routine reviewer1731n/an/a00
Nov 7–14: the first post's workflow18488833581
Nov 14–17: automatic and plan review1612471757499318
Nov 17–21: GPT-6.2, three layers, bite check130489375159131157
Nov 21–22: waves, gates and reports5313596415039
v2 (last three phases)344871646274280514
Total701960729309288515

Each column counts something different (n/a means not measured):

  • Flagged: blocker or major findings about logic or an unmet criterion, with repeats grouped. Style, scope and test quality are left out.
  • Would pass: the flagged errors already present in the first review, meaning in the delivery the executor called done. Those that appeared during fixes wouldn't exist in direct development.
  • Normal use: I read a sample of 40 errors and classified each one. 17 would show up in normal use, 21 depend on a rare condition and 2 are cosmetic or unlikely. The column applies that 17-in-40 ratio.
  • Confirmed: first-review errors that a re-review of the same delegation did not flag again. This is the most solid floor.
  • Spec: serious plan-review findings, before any code existed. They stay out of "would pass", because not all of them would have become bugs.

The columns are not a funnel. "Normal use" is a sample-based estimate and "confirmed" is a direct count, which is why the two land close together without one containing the other.

Errors per v2 phase
  • Flagged
  • Would have passed
  • Confirmed
Nov 14–1724717599
Nov 17–21489375131
Nov 21–221359650

Number of errors

The last phase has about one day of use. The jump in Nov 17–21 comes largely from the new reviewer, which finds more problems in the same diffs.

To make it concrete, here are three errors from the sample that would show up in normal use, all fixed before landing:

  • The edit that vanishes. You save a form and start another edit while the refresh is still pending. When it returns, the second edit disappears and the screen says there's nothing to save.
  • One Esc that closes two things. Typing @ in the channel input and pressing Esc closed the mention menu and also the members panel, and saved the panel as closed.
  • A judge outage counted as a model error. In a benchmark, a 503 from the service that grades answers became a negative judgment and dragged down the tested model's success rate.

What these numbers don't say

  • 646 is likely a ceiling. In direct development, CI, tests or I would catch some of these errors, and whoever wrote the code might make different ones.
  • The phases don't compare quality with each other. The reviewer changed along the way. On the same diffs, Astra 6.2 found more problems than Astra 6, which explains much of the jump in the Nov 17–21 phase.
  • The sample is small. It's 40 items classified in one read, with a margin of about 15 percentage points. Hence the 176 to 374 range.
  • The grouping is approximate. The same error reworded can count twice, and two similar errors can become one.
  • The first phase is undercounted. There was almost no reviewer, so the errors in those 173 tasks weren't measured.

The cost

Reviewing every task shows up in the same data, on the other side.

Fix rounds per task
Nov 1–70.31
Nov 7–140.46
Nov 14–170.89
Nov 17–211.09
Nov 21–221.43

continue runs per run.sh new, across all projects

Part of this is the workflow doing its job: each finding now goes back to the executor as a continue. In the current phase, 83 of 91 reviews requested changes. But my goal was for this number to drop, and so far it has gone up. I can't yet separate how much is a stricter reviewer and how much is the executor delivering worse work.

How it works now

Everything starts with /implement and a card. It keeps the task's state, resumes where it left off and only calls me at three points: design, plan alignment and delivery.

card ─► triage ─┬─ fix ─────► short design
                └─ feature ─► I describe it, then the design
                                     │  stop 1
                                     ▼
                  wave plan ─► plan review
                                     │  contradiction? stop 2
                                     ▼
       wave 0 (contract) ─► gate ─► wave 1 (up to 4) ─► gate
                                     │
                                     ▼
                  final review ─► delivery (stop 3)

I went back to planning

The card now gets classified first. A fix keeps the short design and a single approval. On a feature, the orchestrator can't show up with the answer: it first asks me open questions, with no options attached, about the problem, what success looks like and what would be a failure even if it worked. Only then do the approaches come.

The plan can also come back to me, but only in three cases: a contradiction that forces a choice between two behaviors, a design gap that changes what the user sees or the architecture, or a new contract the design didn't describe. Otherwise, it moves on. An alignment that only asks for an "ok" counts as a workflow failure.

Review before the code

Before any delegation, a Claude Opus 5.5 at xhigh reads the design, the plan and every spec in the round, looking for contradictions, open behavior, verification that can't run and parallel tasks touching the same file. That's where the 514 spec errors come from.

The spec got stricter too. Edge cases is required, with input and expected result, and run.sh refuses a spec without it.

Waves

A round is split into waves. The next wave only starts once the previous one is integrated and verified.

Wave 0 · contract
  team-channel-contract  fast-executor      core/src/types.rs, ui/src/types.ts
Wave 1 · depends on wave 0
  channel-backend        fast-executor      core/src/channel/
  channel-ui             frontend-engineer  ui/src/channel/
  mention-parser         fast-executor      core/src/mention.rs
Wave 2 · depends on wave 1
  channel-e2e            frontend-engineer  ui/e2e/channel.spec.ts

The rules are few:

  • One owner per file. Two tasks in the same wave never touch the same path.
  • Hotspot files (routes, registries, config, lockfiles, i18n catalogs) only change in wave 0 or in an integration task.
  • Wave 0 only exists when there's a contract: types, interfaces, schema and routes, with no logic.
  • At most 4 active tasks per session. run.sh refuses the fifth.
  • A gate between waves. Tests, build and typecheck on the integrated branch, plus a review of the wave to catch the seams between tasks that no individual reviewer sees.

Three-layer review and the test that bites

Every diff is reviewed automatically, in a single pass:

  1. Completeness: each acceptance criterion becomes met, not met or unverifiable, with evidence.
  2. Logic and edge cases: races, remounts, stale data, limits, error paths.
  3. Tests and scope: would any test pass without the change? Does each part of the diff map to an item in the spec?

Before the reviewer, run.sh runs mechanical checks: disabled tests, new TODOs, files outside the scope, files from a neighboring task. And when the spec has a Bite check, it copies the worktree, reverts only the production code and runs the tests. If they still pass, they prove nothing, and that becomes a finding.

The orchestrator stopped reading diffs

Reading diffs polluted the context of the one who needs to plan, and that gets worse with four tasks per wave. Now the orchestrator reads the verdict, the findings, the checks and the executor's report. Code findings go straight back to the executor. If the executor disagrees, it answers with DISPUTE: and Astra arbitrates.

The risk is the reviewer becoming the only barrier. In the comparison test, Astra 6 approved a task with a real cancellation bug, which 6.2 caught. That's why there are three exceptions where the orchestrator opens that task's diff: a sensitive area, a red gate with everything approved, and the same finding coming back after two continue runs.

What I tested on the models

Before settling the roles, I ran four real tasks that had findings at the time, from the same commit and with the original spec:

ExecutorBlocker/majorUnmet criteriaAbandoned items
GPT-6.1 Terra, xhigh14174
GPT-6.2 Sol, medium1190
GPT-6.2 Sol, high12120

Terra abandoned items halfway and Sol didn't. Between medium and high, the difference was within the noise, and I stayed with high because quality comes first. It's one run per combination, so treat it as an indication rather than a benchmark.

I also tested Claude Sonnet 5.5 on the front-end. On the two UI tasks, it did worse on one and tied on the other, took about twice as long and produced larger diffs. Today the front-end runs on Sol 6.2, with its own UI prompt.

What's still open

  • Waves have about one day of use. They're still being validated, and the next real feature is the true test.
  • continue runs per task need to drop. If they don't, the next step is a second reviewer from another model family.
  • It needs remeasuring with more time on v2. The current phase is still running, and its numbers will change.