I tried to disprove the TDD benchmark. I reproduced it.

Kun Chen published a ProgramBench result arguing that a TDD skill hurt GPT-5.5 in Codex. I expected the result to depend on the specific skill he tested.

His setup was simple: same benchmark, same model, same Codex defaults. The control arm used no mandated workflow. The TDD arm added the Superpowers test-driven-development skill. Kun reported lower test pass-rate and higher cost for the TDD arm across easy, medium, and hard tasks.

I tested a different TDD prompt: Matt Pocock's version, with four small edits to remove instructions that ask the human for confirmation, since an automated harness cannot answer. The prompt emphasizes behavior through public interfaces, one red-green-refactor cycle at a time, and vertical slices rather than writing a large test suite up front. I thought that difference might explain the original result.

I reran the comparison with that skill.

It did not change the conclusion.

Result

I used the same n = 192 denominator as Kun's write-up. The structurally broken and unscoreable tasks stayed blocklisted. The alternate TDD arm completed all 192 non-blocklisted tasks, and I refreshed stale evals before scoring.

My TDD skill lost to the no-TDD control by 4.6 mean pass-rate points. It won 67 tasks, lost 124, and tied 1. The paired Wilcoxon p-value was 1.2e-6.

Against the original TDD arm, it was effectively tied: -1.0 mean points, 93 wins, 95 losses, 4 ties, p = 0.74.

The difficulty split matched the original pattern.

Failure modes

I do not have full transcripts for Kun's original TDD arm, so this section uses the available artifacts: per-task scores and submitted tarballs. That limits what I can claim about the agents' internal process.

The two TDD arms failed in similar places. Original TDD lost to control on 136 tasks. My TDD skill lost on 124. Both lost on 106 of the same tasks. Their per-task deltas against control had a moderate correlation of about r = 0.48.

The shared failure mode appears to be this: the agent's own tests became a narrower definition of done than the hidden grader used. TDD increased the amount of work spent on local tests, but those tests did not reliably discover the hidden behavior ProgramBench scores.

The two skills did not fail identically.

The original TDD arm looked more prone to writing a larger guessed local spec. On simeg__eureka, control scored 67.4, original TDD scored 38.4, and my TDD arm scored 41.9. The original TDD submission kept a large behavior test file and a smaller implementation than control. My version did slightly better, but still missed enough hidden behavior to lose badly.

My skill did help on some tasks. On eradman__entr, original TDD scored 42.7 while my TDD skill scored 69.0, close to control's 72.4. On hairyhenderson__gomplate, original TDD scored 30.9 while my skill scored 64.1, slightly above control's 59.6.

It also produced a different kind of bad result. On bootandy__dust, my arm kept reasonable public-interface tests for help/version output, apparent sizes, JSON output, filters, and files-from behavior. The final score was still 2.4, compared with 63.9 for control and 59.2 for original TDD. On bellard__quickjs, control and original TDD both scored about 60, while my arm scored 2.1 after choosing a Python mini-JavaScript interpreter architecture.

My interpretation: the original skill more often failed by writing a guessed local spec in bulk. My skill produced cleaner behavior slices and fixed some of those failures. But both shared the deeper ProgramBench problem: self-written tests are an unreliable proxy for hidden requirements, and the extra TDD loop competes with exploration and implementation time.

What I take from this

My prior was that a better TDD prompt would recover the gap. It did not.

ProgramBench is not a normal maintenance task. The agent is not changing a known codebase with an agreed specification. It is reconstructing CLI behavior from a reference binary, docs, and probes, then facing a hidden grader. The hard part is discovering behavior the agent does not yet know.

A behavior-first TDD skill can improve the tests the agent writes. It cannot make those tests cover behavior the agent has not discovered.

That makes Kun's proposed failure mode plausible to me. The problem is not only that agents write weak tests. The problem is that passing self-written tests can become a stopping signal. The agent can satisfy its local spec while still missing the benchmark's spec.

What this does not prove

This does not prove TDD is bad for humans.

It does not prove agents should never write tests.

It does not prove TDD hurts every model, benchmark, task, or codebase.

It does show that I failed to disprove Kun's result with the stronger TDD variant I expected to help. On this benchmark, with this model and this hidden-spec setup, the no-TDD control still won.

I sent the reproduction data upstream as a PR: per-task rows, submissions, a difficulty breakdown chart, and a short reproduction note.