Testing the agents that review engineering plans.
An AI reviewer that writes convincing feedback is easy to get. One that catches the specific thing wrong with a plan is not. This is about testing the difference — turning review failures into cases that fail until the reviewer actually improves.
Problem
When a plan is written with AI help, the fastest way to check it is to have AI review it. The reviews come back fluent, organised and confident. That is the problem: fluency is not the same as catching what is wrong, and a confident review of a flawed plan is worse than no review, because it ends the conversation.
The failures were specific and they repeated. A review would accept a git object that did not exist. It would accept a stated cause with nothing establishing it. It would pass a plan whose steps contradicted each other. It would report a step as done when nothing had confirmed it.
Each time, the tempting fix is to add a line to the prompt — "verify that referenced commits exist". That produces an untested rule, and a set of untested rules that nobody can tell apart from prompt decoration. What was missing was not more instruction. It was a way to know whether a reviewer catches a given class of mistake, and to notice when it stops catching it.
My role
This was my own tooling for my own workflow, so the split is between building and judging.
- I implementedthe review protocol: the passes a plan goes through, and what each pass has to produce.
- I implementedthe reusable skills that carry those passes, so a review is repeatable instead of improvised per plan.
- I implementedthe evaluation cases, built from real review failures, each pairing a plan with the finding a reviewer is supposed to raise.
- I proposed and ledthe definition of expected behaviour, case by case — what counts as catching the issue is the judgement the harness cannot make for me.
- I proposed and ledthe decision to accept or reject each change to a skill, and the rule that a change is not done until the case that motivated it is re-run.
Key decision
Turn recurring review failures into regression cases instead of adding untested prompt rules.
A prompt rule is cheap to write and impossible to evaluate. Once there are a dozen, nobody knows which ones still matter, which ones conflict, and which were never doing anything. Worse, the rule usually gets added right after the failure it was written for — so it looks effective precisely because nobody tests it.
A case is more expensive. It needs a plan, an expected finding, and a judgement about what counts as catching it. In exchange it can fail. When a skill changes, the cases say whether the thing that was broken is now caught, and whether something that used to be caught no longer is.
The operating rule that comes with it has two halves, and the second one is the one that does the work: if a plan needs more than two review cycles to converge, add it as a case — and trace its failure back to a specific rule in a specific skill. Without the second half you accumulate cases without ever learning why the reviewer missed anything. It is a rule about when to invest in a case, not a claim that plans now converge in two cycles.
How it works
The loop below is the one that turns a failure into a case. The example in it is a real recurring failure class from my own reviews — a reviewer accepting a reference to a git object that does not exist. It is not a fabricated production incident.
- A plan with a known defect
A plan that references a specific commit as the basis for a change.
- Expected flag
The reviewer should report that the reference cannot be confirmed — the object is not there to check against.
- Actual reviewer response
The review accepted the reference and assessed the rest of the plan on top of it, which read as a clean review.
- The gap, named
The reviewer had no step that required confirming a referenced object against the repository before reasoning from it.
- Skill change
The validation pass was changed to require that referenced objects be confirmed, and to report an unconfirmable reference as a finding.
- Re-check
The case is run again. It only counts as fixed when the expected finding is raised — and the other cases are run to see what the change cost.
Plan → expected flag → actual response → named gap → skill change → re-check. The human steps are 02 and 04: deciding what should have been caught, and naming why it was not. The harness cannot do those.
The point of writing the expected flag before looking at the response is that it is the only order in which the test means anything. Reading the review first and then deciding whether it was good enough is not a check — it is agreement.
Step 06 is the one that makes it a regression case rather than a bug fix. A change that fixes one case and breaks two is a common outcome with prompt-driven tooling, and the only way to see it is to re-run the rest.
The five reviewer roles, and what each one has to produce
Codebase Validator — verifies every file, line and claim in the plan against the actual checkout. Expected output: each reference either confirmed against the code or reported as unconfirmable. This is the pass the git-object example belongs to.
Architecture & Risk Reviewer — patterns, scope, memory and retain-cycle risks, whether the change fits the schema. Expected output: named risks and affected boundaries, not a general approval.
Modern Practices Research — whether the approach matches current platform and SDK guidance rather than remembered guidance. Expected output: a practice tied to what the project actually depends on.
Premise Adversarial Validator — attacks the plan's stated reason for existing: is the thing already implemented, is the cause unverified, is the evidence cherry-picked. Expected output: premises marked supported or unsupported, with the unsupported ones named.
Execution Validator — whether the plan can actually be carried out step by step as written. Expected output: steps that are executable as described, or the point at which they stop being.
Five different questions, so a plan is not reviewed five times from the same angle, run for up to five cycles and treated as finished only when a fresh pass comes back with nothing. This is a process I run, not an autonomous system that runs itself.
Cross-model review, and what it does not give you
One of the five — the Premise Adversarial Validator, the pass that attacks the reason for the work — deliberately runs on a different model from the others. Different training and different habits mean it sometimes objects where the first reviewer did not, which is the point.
Two honest caveats. First, I have no measurement showing that this role in particular benefits from a different model; it is a reasoned choice, not a demonstrated one, and I am not going to dress it up as the latter. Second, a different model does not guarantee independence: models trained on overlapping data share blind spots, so two reviewers can miss the same thing for the same reason and produce agreement that reads like confirmation.
Which is the whole argument for the cases. Agreement between agents is the absence of one kind of disagreement, not evidence of correctness. A case that fails tells you something; two agents nodding does not.
Verification
The eval set is 11 cases. The run below is the recorded result I have, reported as it came out — not as a pass rate.
- Documented11 regression cases, each from a plan that really did go wrong
Every case holds the earliest version of a real plan — ones that had taken between nine and eighteen review iterations to settle — together with the findings a reviewer must raise, a minimum severity for each, and the properties the review output has to have. It is the size of a set: 11 cases is not 11 passing checks.
Source inspected, not independently verified · Step Experience Master Document Appendix G4 · 2026-09-11
- DocumentedTwelve documented anti-patterns and fourteen logged rule changes
The anti-patterns are the specific mistakes a review is checked against — the unconfirmable git reference above is one of them. The rule changes are the edits logged as coming out of running the cases. Both are counts of written material: a logged rule change is not evidence that reviews got better.
Source inspected, not independently verified · Step Experience Master Document Appendix G4 (quality gate and logged rule additions) · 2026-09-11
- Historical runRecorded run, 2026-05-28: 6 pass, 2 fail, 2 partial, 1 fixture defect
The outcomes of one execution of the set on that date. 'Partial' means the reviewer raised the issue but missed part of the expected finding. 'Fixture defect' means the case itself was mis-specified, so its result said nothing about the reviewer at all — worth reporting rather than quietly excluding, because a bad case is its own kind of failure. Two reviewer roles were patched afterwards and the broken case was fixed.
Source inspected, not independently verified · Step Experience Master Document Appendix G4 (run log for 2026-05-28) · 2026-05-28
- Not presentedThe re-run that would show whether those fixes worked
Missing. The policy attached to those fixes was explicitly 'fail before the fix, pass after', but no post-fix run is recorded anywhere I can point to — not in my notes, not as a report. So the 2026-05-28 result stands as the last one I can show, and the two failures and two partials should be treated as open. This is the weakest point in this case study and I would rather name it than round it up.
- DocumentedOperating rule: more than two review cycles means add a case and find the rule that missed it
A rule about when to invest in a new case, not a measurement. It is not evidence that plans now converge within two cycles; no cycle-count measurement is published here, and the 'converges in two cycles' phrasing that appears in my own notes is a statement of the rule, not of an observed rate.
Outcome & limitations
What the work produced
- Review became a defined protocol with five passes, each with its own question, instead of one general request for feedback.
- Recurring failures became cases that can fail, so a change to a skill can be checked rather than believed, and the recurring mistakes got written down instead of being rediscovered.
- One concrete example is documented end to end above: a reviewer accepting an unconfirmable git reference, the missing validation step, the skill change and the re-check.
- The recorded run is published as it came out — 6 pass, 2 fail, 2 partial, 1 fixture defect — including the case that turned out to be wrong itself.
What this does not show
- This is an engineering process I run, not an autonomous review runtime. The judgement about what should have been caught is mine, case by case.
- The last run I can point at is 2026-05-28 and it was not green. The fixes that followed it have no recorded re-run, so their effect is unestablished and the open cases stay open here.
- Cross-model review adds criticism, not independence, and I have no measurement that the one role running on a different model catches more because of it.
- The skills, cases and run logs are my own working files and are not published, so the run result is my report of it rather than an artifact you can open.
Sources & available artifacts
This tooling is my own working setup. What each claim rests on:
- My own accountThe review skills and their revision historyNot public
My own working files. Not published here; they contain project-specific paths and operational detail.
- Test runEval run log, 2026-05-28Not public
The source of the 6 / 2 / 2 / 1 breakdown. Reported here, not published as a file.
- Technical write-upMy own experience document for this work (§3.K, Appendix G)Not public
Where the protocol, the five roles, the eval set and the 2026-05-28 run are recorded. Inspected while writing this page.
- Source codeThis portfolio's own evidence toolingNot public
The same idea at a smaller scale: this site validates its approved content and runs grounding evals as part of its own checks.
Every figure on this page, with what it measures and what it does not prove (4)
The review skills are regression-tested against 11 cases built from real plans that had needed between 9 and 18 review iterations.
- Verification
- Source inspected, not independently verified
- Source
- Step Experience Master Document Appendix G4 (2026-09-11)
- Environment
- local
- What is measured
- The number of cases in the evaluation set. Each case holds the earliest version of a real plan, the findings a reviewer must raise with a minimum severity for each, and the properties the review output must have.
- Limits
- The size of a set. It says nothing on its own about how many of them pass.
Twelve recurring review anti-patterns are written down, and fourteen rule changes are logged as coming out of the eval cases.
- Verification
- Source inspected, not independently verified
- Source
- Step Experience Master Document Appendix G4 (quality gate and logged rule additions) (2026-09-11)
- Environment
- local
- What is measured
- Two counts of my own working material: the documented anti-patterns a review is checked against, and the rule changes recorded as having come out of running the cases.
- Limits
- Counts of written rules and logged edits. A logged rule change is not evidence that reviews improved — the only recorded run of the cases is still the one below, and it was not green.
The recorded run on 2026-05-28 returned 6 pass, 2 fail, 2 partial and 1 fixture defect across the 11 cases, followed by patches to two of the reviewer roles and a fix to the broken case.
- Verification
- Source inspected, not independently verified
- Source
- Step Experience Master Document Appendix G4 (run log for 2026-05-28) (2026-05-28)
- Environment
- local
- What is measured
- Per-case outcomes from one execution of the eval set on that date, judged against each case's expected findings. 'Partial' means the reviewer raised the issue but missed part of the expected finding; 'fixture defect' means the case itself was mis-specified, so its result said nothing about the reviewer.
- Limits
- A single historical run, and not 11 passing checks. The run was followed by fixes under a 'fail before the fix, pass after' policy, but no post-fix re-run is recorded anywhere in the source — so the outcome of those fixes is not established and the failing cases should be treated as open.
The review protocol uses five reviewer roles — Codebase Validator, Architecture & Risk Reviewer, Modern Practices Research, Premise Adversarial Validator and Execution Validator — of which the Premise Adversarial Validator runs on a different model as a cross-model check.
- Verification
- Source inspected, not independently verified
- Source
- Step Experience Master Document Appendix G5 (2026-09-11)
- Environment
- local
- What is measured
- A description of an engineering process: five review passes, each with its own question, run for up to five cycles and treated as finished only when a fresh pass returns no warnings.
- Limits
- A process, not an autonomous runtime. The source states that one role runs on a different model as a cross-model check but gives no measured reason why that role in particular benefits, so no such rationale is claimed here. Five passes agreeing is not proof that a plan is correct.