The TDD (test-driven development) workflow can be used with AI-augmented coding in multiple ways:
- Human writes the tests: A human defines the test scenarios in some form, be it in natural language, in BDD style, or directly in code. Then AI writes the implementation to make those tests pass (with maybe a first step that transforms the human’s scenarios into code).
- Review checkpoint for the human: AI writes a failing test, human looks at it to review that the test is testing the wanted behavior, then AI writes the implementation
- Fully inside the agentic loop: Prompt an agent to write failing tests first, one by one, and then write the implementation and check that the previously failing test is green.
At this stage, that last usage is by far the most common one. But does it really make a difference, asking an agent to follow a TDD workflow fully inside its own loop? Does it really provide value, or is it one of the rare examples where what’s good for the human might be irrelevant or bad for a coding agent?
I created an exploratory evaluation setup to scratch the surface of this question and see what I would find. It is far from a comprehensive and structured eval result, but it did create some hypotheses to think about if you are working hard to get your agent to use TDD.
TLDR; Based on Opus’s judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
The setup
- Tasks: I created a small, medium and a larger task with the help of Claude, all green field implementations of a bit of business logic. I had it make a bunch of suggestions, asking for idiosyncratic and specific logic to increase the probability that there will be variance between solutions, and not just a repetition of something that is already dominant in the training data.
- Instructions: In all runs, I included instructions to achieve at least 80% code coverage.
- Model: I used Sonnet 4.6 to generate the solutions.
- Judgment of TDD adherence: Evaluation of adherence to TDD was also done by Sonnet 4.6.
- Judgment of solutions: Opus 4.8 compared the quality of both solutions and their tests, without knowledge of how the solutions were created. I didn’t give very specific inputs on what I consider to be good quality, as this was a very open exploration. And in my experience, the more specific I would have gotten, the more the model could have over-indexed unnecessarily on the quality criteria I list. Opus has shown to be quite a capable model in terms of judgment of code quality. For its ranking of the solutions, it created a rubric on the fly to pass to all subagents that were evaluating the individual solutions.
When you draw your own conclusions from my results, the main caveats to consider are:
- This is obviously a very small sample size, so take it with a grain of salt
- Judgment of what “quality” means was almost fully left to Opus (with only a few pointers about test quality)
- None of the runs ever followed TDD perfectly, but pretty well
- The coding tasks given to the agents were all greenfield and relatively small, purely about business logic
How good are agents even at TDD?
Before I even started, I needed to make sure the TDD instructions were actually followed. Historically that hasn’t gone well for me: agents often write the implementation first and generate tests after, skip confirming the red step, or over-implement ahead of the current test so the next one passes without ever going red.
The prompt I ended up using worked well enough with Sonnet to use for the comparison, though all sessions showed some of these failures to an extent. For each TDD run, I had an independent agent judge how well the workflow was followed, based on the session transcript, so that I wouldn’t accidentally take into account a run that didn’t meaningfully do it.
Results
I created 5 batches of solutions, with two non-TDD and two TDD solutions each. In one batch, I also added two runs that were instructed to write the tests first, without full TDD discipline (no incremental red/green).
Across the small (1 batch) and medium (3 batches) tasks there was a bit of a pattern: Opus ranked the two non-TDD solutions #1 and #2, and the two TDD solutions #3 and #4. Only once – after I strengthened the TDD prompt with a more explicit refactor-and-design-review step – did a TDD solution rank #1. In that same batch, the other TDD solution, run with the identical prompt, ranked last though… For the larger task, TDD landed in the middle, while the two non-TDD runs took both the best and the worst spot.
(Details in the appendix)
Hypotheses
So in summary, both TDD and non-TDD scored both as a best and a worst solution across the batches, with TDD overall performing slightly worse.
Asked to look at the session traces to hypothesize about the results with knowledge of which workflow was used for which, Opus found that the non-TDD and test-first runs always created the full design (architecture, data types, edge cases, contracts) before writing any code or tests, rather than working through it one requirement/test at a time. That seemed to be the thing that moved the needle slightly towards comparatively better data models, more cross-cutting edge cases, and better completeness of the functionality.
The TDD instructions actively work against such an up front design step. The design in those runs emerged from the sum of many locally-minimal decisions and was rarely revisited, so it tended to land on whatever shape the first test happened to lock in. Behaviour the agent didn’t think to write a test for didn’t get implemented at all.
When I chatted to Ivett Ördög about this, she had this theory: “The way AI agents were trained is that they have seen completed functions and descriptions of those functions. The number of actual step-by-step TDD examples they have seen is a tiny part of the training data. That means that the LLM has an internal representation of code that is a direct translation of requirements to code, and not a process of how to get to that representation.”
Goals of TDD – still achieved in the agent loop?
The following are my general reflections about using TDD in the agent loop, not only based on this experiment. I’m going through the ultimate goals I personally have when I use TDD, skipping some of the ones that are about having tests in the first place, and unit tests in particular (like refactoring safety net, living documentation, test coverage), focussing on the ones that are specific to the TDD workflow.
Test first >> Avoiding tautology
Test-first makes it easier to assert the output I want, rather than restating the implementation. Such a test can never fail when the implementation is wrong as it was derived from the same logic it’s supposedly checking. When the assertions are decoupled from the specific implementation path, the test can actually catch when the behaviour is not what I intended.
Still achieved in the agent loop?
In my experiment, some TDD sessions had this problem anyway, in spite of writing the test first. In one particularly obvious example, tests checked the implementation’s output against itself, re-running the same code to produce the “expected” answer (see 4. on this list of observations). Writing the test first doesn’t reliably prevent this – it might make it less probable, which is all we can ever hope for anyway with LLMs, but from this small data set I can’t draw any conclusions about that probability.
Test first >> Testability
Test-first ensures the code is designed to be testable from the start, rather than retrofitting tests that are more complex and brittle than necessary.
Still achieved in the agent loop?
The results didn’t give me any clear cut signals either way. For what it’s worth, the size and nature of the tasks I chose didn’t require a lot of design complexity that could have surfaced this. To an extent though, testability is a corollary to driving design (see below).
Red-green >> Test effectiveness
Observing a test fail first, then succeed (red-green), proves it will actually catch a regression.
Still achieved in the agent loop?
How much sense does this really make when the human is removed? Watching a test go red is only proof of anything if someone is checking why it went red. When the agent both writes the test and confirms it failed, a red test tells you the agent ran it and saw failure, not that the failure was for the right reason. The evaluations of TDD adherence in my experiment also show this: agents still sometimes skipped or faked the red step, or implemented ahead of the test so that it passed immediately. Regression effectiveness can be monitored and improved with mutation testing (as I wrote about here). Mutation scores across the solutions didn’t show any signals that TDD runs produced meaningfully better mutation scores than non-TDD runs. I don’t really care how regression quality was achieved, as long as I have a mechanism to see how good it is.
Test first, red-green-refactor >> Driving better design
Writing the test first forces us to specify usage before implementation, pushing toward better interfaces and more modular code. The refactoring step in the TDD loop further pushes us to improve the design step by step.
Still achieved in the agent loop?
The experiment at least hasn’t demonstrated superior design in the TDD runs at all. I now even wonder if TDD makes it worse, based on Opus’s scoring, as the non-TDD solutions more often than not were ranked higher, and the design flaws it listed made sense to me. But the data set is of course too small to definitively conclude anything. (If anybody has time and tokens to run a larger experiment, that would be very interesting!)
When humans write a test first, it forces us to think about usage before implementation, we have to sit with the friction of specifying behaviour and expectations before knowing how to build it. An agent doesn’t experience that and can write a test the same instant it plans an implementation. Without a human checkpoint between the two, is there really any purpose left to writing the test first?
Small steps >> YAGNI
Writing only enough code to pass the next test is about restraint. It’s supposed to stop us from building abstractions or handling cases nobody has asked for yet.
Still achieved in the agent loop?
This is a very human-centered benefit that gets lost when an agent does TDD by itself. We don’t get to sit in that friction anymore where we really have to think about all the intricacies of what we’re building. That is theoretically shifting to when we are writing the specs to give to an agent, but we don’t have a TDD-like mechanism there that lets us think the spec through in small steps.
Couldn’t an agent work in those small steps though and ask us questions whenever it finds something that might be unnecessary? In my general experience, they’re not very good at that. And in the experiment as well, minimal-implementation instructions didn’t reliably stop them from building more. They frequently overshot and implemented more than the current test demanded, because they had the full requirement available. We usually don’t spoon-feed the spec one by one, that would be very inefficient.
Small steps >> Fast, localized feedback
Taking one small step at a time means that when a test fails, I know almost exactly what caused it, as the only thing that changed since the last green state is the one thing you just wrote.
Still achieved in the agent loop?
The setup didn’t show if agents got stuck debugging more frequently with versus without TDD. But in my general experience, agents are usually reasonably good at figuring out why a test is red, even without having taken small, deliberate steps to get there. I’m still doubtful if the times when they do get stuck could be meaningfully mitigated with small TDD steps, and if the overall cost/benefit comparison would hold up.
Small steps >> Confidence and learning
In Kent Beck’s preface to “Test-driven Development by example”, his biggest rationale for TDD is “managing fear”. He says that the legitimate fear of hard problems makes developers tentative, less communicative, and avoidant of feedback. With TDD, each passing test shows us progress, so we can relax knowing that progress is locked in. The tests are a psychological mechanism that helps us keep going.
Still achieved in the agent loop?
This is very much about managing a human’s fear and giving a human permission to relax. That doesn’t transfer when the agent is doing TDD inside of the loop, as it doesn’t give me the same control and trust as when I do it myself, step by step.
Costs
At least 3x the tokens
See detailed numbers in the appendix.
Naturally, as a TDD workflow requires many more turns and tool calls, more tokens will be used. However, many of those will be cache hits, so note that the 3x or more factor of tokens aren’t a direct representation of how much more costly it is. (Unfortunately, I didn’t track cache hits during the experiment.)
Prompt maintenance and testing
TDD is a process that doesn’t seem to “come natural” to models. It’s like an uphill battle against the training data, and takes a lot of iterations on a prompt to get it to follow the process most of the time. For example, when I realised after my first batches that the agent didn’t do much refactoring in the red-green-refactor loop, I changed the prompt to put more emphasis on that step, as it’s of course crucial to TDD. I later asked Opus to look at those sessions and see if it found an improvement in refactoring efforts. It did report an increase in refactoring steps – however, it also listed some cases in which the agent set out to refactor, but decided the design was good enough even in cases where Opus thought it clearly wasn’t (e.g. when everything was implemented in one big module, but could have clearly been split up into multiple responsibilities).
TDD is a comparatively complex set of instructions with lots of variables, and consequently lots of variations in how agents interpret it. So I imagine this type of prompt to be even more volatile across models than simpler instructions are, meaning it takes effort to keep the prompt working across models and model releases.
My conclusions
I think at this point there is generally more and more evidence that being overly specific about how we want a model to do something is not a sustainable approach. Instead, we should find as many ways as we can to monitor the outcomes and give feedback. That feedback should be automated wherever possible, and we need to carefully think about where we insert ourselves as arbiters of what is good and correct.
Even though I am aware that my little eval is far from representing a broad perspective on the effectiveness of TDD, it definitely hasn’t given me any new indications that all this effort is worth it. Especially not if we can find other ways to achieve the majority of TDD benefits.
I personally have stopped telling my coding agents to write tests first, let alone do TDD (which I never did, to be honest), until I see evals or other strong arguments that convince me otherwise. I’m trying to focus instead on the benefits of TDD when I use it outside of the agent loop, and exploring alternative ways to achieve them.
How to get good regression tests?
…so that the agent and me get signals when existing functionality breaks
I still care about solid regression tests, because even though an agent can of course fix red tests the wrong way around, at least the red test gives it a feedback signal to double check pre-existing requirements that might have broken. I monitor and improve regression quality with the help of mutation testing, instead of giving elaborate TDD instructions and hoping for the best.
How to build regular refactoring into the process?
…so that the codebase remains easy to change
Refactoring remains crucial, but the small steps of traditional TDD don’t seem to be an efficient or effective way to do it in the agent loop. A few examples of triggers for refactorings: Give the agent access to static code analysis; run regular reviews of structure and modularity; develop team rituals to maintain a good understanding of the codebase and catch drift early; keep an eye on the trend of number of files touched per change, and number of tokens are for a change.
How to get confidence?
…so that I am not afraid to push to production
The hardest question remains, how do we get that confidence that TDD was giving us, how do we manage fear, how do we lock in progress? I don’t have a clear answer to that, but I’ll just mention one of the things that seems like a good building block for that: I have recently tried out the Approved Scenarios approach that Ivett Ördög is advocating for. In my words (don’t hold her to it), it’s a form of semi-manual testing that is supported by a bespoke test runner for each application. That runner shows me functional test scenarios in an easy to think about way, and allows me to “freeze” expectations (scenarios / fixtures) in that runner after I have thoroughly confirmed them. Whenever those frozen expectations are violated in the future, I have to approve them again. My colleague Matteo Vaccari gave a great overview of his experiences with that approach here.
Whatever ends up giving us trust and confidence in our software in the future – I think the role of TDD as we’ve known it is significantly smaller than pre-GenAI.
