What Grading 10 AI Specs Taught Us About Decomposition

Ten specs, ten forecasts, ten graded outcomes. That is the whole pilot. We wrote down what we expected an AI-decomposed spec to cost before any code existed, then checked the forecast against what actually happened, and we are publishing the numbers as they came out, including the one hypothesis that did not survive contact with the data.

This is the sequel to our earlier post on spec-driven development. That post argued for writing specs before letting an agent touch code. This one asks the harder question: if you write the spec first, can you also predict how it will go, and were we right?

The method: forecast at shaping, grade at verify

A spec in Monday Morning is written in two passes. Shaping turns a brief into numbered requirements and a task list. For this pilot, shaping also wrote a small forecast block into the spec's frontmatter: how many tasks the work would take, a short list of preconditions stated as things you could grep for and check, a named risk, and a confidence number.

That forecast is frozen the moment it is written. Nobody touches it again until verify. At verify time — after the code exists, after tasks are marked done — we compare the forecast against reality and record what we find, in the same file. Predicted three tasks, spec closed with three tasks: recorded as a match. Predicted a precondition and it was still true when we rechecked it: recorded as held. Named a risk and it actually bit during implementation: recorded as it happened.

The idea traces back to a hand-retro we ran on 2026-08-15, going back over a sweep of ten earlier specs that had no forecast at all. Zero of ten had a written verify verdict on file, and three of five specs that were not clean failures had failed on a precondition that would have been grep-checkable before anyone wrote a line of code. That retro is the reason this pilot exists: rather than adopt an external rubric for grading spec quality, we decided to grade our own forecasts against our own outcomes and see what the numbers actually said.

The mechanism, only as far as you need it

You do not need the whole frontmatter schema to read the results below, so here is the minimum that matters.

Every requirement in a Monday Morning spec gets a stable id — R1, R2, and so on. Every task in that spec has to cite the ids it satisfies in its own title, something like (R3) or (R1, R3), with (R0) reserved for pure scaffolding work that does not map to a numbered requirement. This is enforced at creation, not at review: a task that cites nothing, or cites an id that does not exist in the spec, gets the whole batch rejected before it is written. That is the task citation gate.

The same ids anchor verification. Rather than re-reading the whole repository and trying to reconstruct what a spec was supposed to do, the verifier reads the actual diff and checks it against the specific requirements the spec claimed to satisfy. We call this diff-anchored verify. It is the reason a verify verdict can be trusted as a record of what the code does, not a summary of what the spec says it should do.

Those three pieces — R-ids, the task citation gate, and diff-anchored verify — are the whole mechanism this post needs. The rest of the frontmatter schema (the exact fields, when each one gets written) is implementation detail we are not going to walk through field by field here.

The numbers

Ten specs went through the full cycle: four from an earlier wave on a companion project, six from dogfooding the pilot on Monday Morning itself. Here is what the graded forecasts showed, combined across all ten.

Task count matched on all ten. Every spec's predicted task count equaled its actual task count at close. Zero delta, ten for ten.

Precondition claims held 58 out of 58. Across all ten specs, the forecasts listed 58 grep-checkable preconditions total — claims like "this function reads its config from a directory, so no registry file needs editing." When we rechecked those claims against the code that actually existed, none of them turned out to be false. Zero misses.

Confidence did not order outcomes. Size did. Each forecast carried a confidence score, something like 0.65 or 0.75. If confidence were doing real work, the low-confidence specs should have been the rougher ones. They were not — confidence scores clustered in a narrow band regardless of how cleanly a spec actually went. What did track outcomes better was simply how big the spec was: more tasks, more surface area for something small to go sideways.

The core hypothesis was not supported. Going in, we wanted to test whether grep-checkable preconditions would predict which specs ran into failure — the theory being that a precondition you can verify with a single grep call is exactly the kind of thing that quietly goes stale and causes trouble downstream. At n=10, we cannot say that, because there were no precondition failures for the hypothesis to explain. We are calling H1 not supported at this sample size, not confirmed and not refuted. That distinction matters more than it sounds like it should.

What actually bit

If preconditions did not cause the friction, something did. Across the ten specs, two failure modes showed up more than once, and neither is a precondition problem.

The first is a seam mismatch between parallel implementers. On one spec, two agents worked simultaneously on the same feature from opposite ends — one in Rust, one in TypeScript — and the Rust side emitted camelCase fields where the TypeScript side expected snake_case. The root cause traced back to a wrong claim in the prompt that had coordinated the two agents' work, not to anything either implementer did locally. We now tag this failure kind as seam-semantics-wrong in our defect taxonomy, because it is specifically about the contract between two writers, not a bug inside either one.

The second is unstated environment or gate state on the main branch. One spec ran into a design-token check that was already failing on main before the spec's own work started — a pre-existing red gate the spec had no way to know about from reading the spec alone. A separate case: running a formatter against a named list of files reformatted the entire crate anyway, because the formatter follows module declarations rather than respecting the file list you gave it. Neither is a defect in the spec's own reasoning. Both are facts about the state of the repository that the spec had no mechanism to check. We tag this kind write-shape-unspecified, and we have since added an explicit precondition kind for main-branch environment and gate state, specifically so a future forecast can check for this instead of discovering it mid-run.

We are not generalizing past these two. Ten specs is not enough data to claim these are the only two failure modes that matter, only that they are the two that showed up more than once in this sample.

The caveats, stated plainly

Three things keep us from overselling this.

Ten is a small number. We are reporting what happened on ten specs, not making a claim that would hold at a hundred.

All ten specs came from two projects and one author. There is no cross-team or cross-organization signal in this data at all. A different team, writing different specs, might see a completely different pattern of what bites.

The zero task-count delta is flattering in a way that needs an asterisk. The same pipeline that writes the forecasted task count is the one that later registers the actual tasks, using the same task citation gate described above. Some of that zero delta is a consistency result — the system agreeing with itself — rather than a forecasting triumph where an independent measurement confirmed an independent prediction. We think it is still meaningful, because the citation gate makes it hard to quietly pad or shrink a task list to match a forecast after the fact, but it is not the same claim as "we predicted an unknown outcome and nailed it."

No claim here rises to statistical significance, and none of it is meant to generalize beyond this pilot.

Where this leaves us

The task-count and precondition numbers are good news, with the caveat above attached. The confidence number is a real miss — a field we are recording that is not yet earning its keep, and we would rather say that plainly than keep publishing it as if it worked. The hypothesis about preconditions predicting failure is the most interesting non-result: we built a specific mechanism to catch a specific kind of quiet failure, and at this sample size that failure mode simply did not show up, while two other modes we had not specifically forecasted for did.

None of that changes the argument from the earlier post: write the spec before the code, give it numbered requirements, and gate tasks on citing them. What this pilot adds is the discipline of writing down what you expect before you know the answer, and publishing the miss along with the hit. We are going to keep doing that as the sample grows, and we will report back honestly when it does.