When AI Writes the Code, Specifications Need an Exit Strategy – O’Reilly

The following article has been extended and rewritten by Markus Eisele from The Main Thread and is being republished here with the author’s permission.

Open a repository after six months of spec-driven agent work and you may find a second system sitting next to the code. Requirements, research notes, high-level designs, low-level designs, implementation plans, task lists, review reports, and a growing stack of Markdown files that explain what the code is supposed to mean. Even if the code changed significantly last Tuesday, the last documentation update was weeks ago.

I understand how teams get there. And it’s not a really new effect after all. We had software evolving parallel to documentation since I can remember. Now that agents produce code so  quickly, we try to control the drift and the code generation by moving more thought in front of implementation. Instead of documenting code, we try to drive code generation with it, making Markdown files with requirements, decision records, design approaches, and acceptance criteria the center of gravity and turning them into our workflow drivers.

What effectively is becoming a very large prompt can easily fill a significant portion of the context window even of modern agents before any relevant source code gets added to it. Natural language specification is a weak system for agents to synchronize a codebase with. Without additional attention and diligence, most agents I work with slowly shift attention away from it quickly and focus on the stronger signals in the codebase, forgetting to update the specification eventually.

Even if it sounds like it, I am not advocating for one-shot prompting or vibe coding here. We still need some specifications to build successful software. The mistake is treating a specification as a permanent natural-language copy of the software. A useful spec describes the next change, documents the decisions that drive the change, sets boundaries, and gives us and the agents enough verification surface. But as soon as the change ships, most of it should be removed.

What remains should move into the artifacts software teams already know how to maintain. First and foremost, obviously, the code. But I also count schemas, configuration, and policies as relevant artifacts. They carry meaning about domain knowledge and system configuration. Two categories that I value highly get easily forgotten: tests as the stable verification layer and runtime telemetry. In fact, I do let my agents look at evidence from all these places not only to hunt for errors but also to continuously optimize existing codebases. Oh, and I do keep decision records. But only a small number and only when their content really has no other place in any of the mentioned artifacts. They can even look like Javadoc, but that will be another article someday.

Code is the fact

Code is actual behavior. Once code is deployed to production, users and connected systems are depending on it. Even a mistake can become an observed contract because it has behaved the same way for three years. The runtime behavior takes precedence in this contract because nobody checks the specification anymore, even if it defines a very different behavior. This is the strongest signal for me to start with the actual code in the production system. Reading a natural-language summary instead of the implemented truth cannot accurately reflect runtime behavior. Code to me is the ultimate, executable specification. Just written in a very specific and deterministic language.

What production code cannot drive though is the next version or iteration of a feature. While agents can infer technical patterns from well-structured codebases, there’s no way they could predict policy changes or future feature requests. Neither can they know about regulatory requirements like retention periods or other specific exceptions, such as why one export runs every night for only one customer. That specific context has to come from somewhere else. But it does not require us to keep a permanent prose description of the whole system. We need just enough context to decide the delta: the difference between what exists and what should exist next.

Written words are for the delta

A change specification should exist when it helps a team decide and review that delta. It should name the outcome, non-goals, constraints that differ from current behavior, and the evidence required for acceptance. It might even contain technical design elements when new features cross architectural boundaries or introduce new patterns that are not present in the code yet. Sometimes it is also worth thinking about how expensive reversing the change is, especially if the existing system has various implementations for a certain pattern and the risk is high that an agent might invent another new version.

The list necessary for changes is very short:

• The intended outcome and non-goals (where necessary)
• Known unknowns and decisions that need human judgment
• Affected system boundaries and authoritative interface artifacts
• Functional and nonfunctional constraints that differ from today
• Acceptance criteria/test scenarios covering the risky path

I prefer calling this a “change brief” instead of a “specification.” Specification carries too much negativity. It sounds heavyweight and reminds me of times long past. It also pretends to be complete. And this completeness is making it very expensive.

We have tried exhaustive specifications before and produced requirement documents and other  high- and low-level designs, followed by architecture decision records for everything. I remember reading folders full of paper over the weekend to get started on a new project on Monday. Way before AI even entered all our lives and codebases. We called this waterfall back in the day, and the approach still has the same negative side effects today. The documentation was complete in an administrative sense and was mostly useless in the engineering sense. We all have seen this happening. Agents easily recreate the same erratic results from overflowing documentation, like we did back in the day.

One particular risk I am seeing with many teams is that they let agents generate the initial version of the spec. A long workflow run produces not only the research but directly derives the requirements, design, and planning, and reviews artifacts on top. While the completeness makes everything look very controlled and defined, it also generates a lot more material to be reviewed and approved. Even if models and harnesses continue to evolve at breathtaking speed, it is still challenging for them to generate real cohesiveness out of chaos. The chance they put the wrong attention on some tempting repetitive words is high. This results in an even higher burden on the human reviewer and makes it endlessly harder to keep the various documents aligned.

I think that additional prose like research notes, prototypes, and design records should only be added to a software project when uncertainty justifies them. They resolve a specific problem. Or help navigate the terrain. I wrote about this before. They should absolutely not become required stages for every pull request.

The map will always be incomplete

A prompt, ticket, or change brief captures what we know before the work starts. The codebase, runtime information, configuration, connected systems, and years of accumulated decisions glued into code hold the rest. Some of those decisions were never written down.

When agents get to work they expose the missing information. Reading a module reveals an unexpected dependency. A prototype shows that a specific user-interaction is awkward. A test uncovers an edge case. Production data contradicts an assumption in the design. This field guide on finding unknowns in agent work describes the problem well. We can identify some unknowns at the start. Others appear only after we inspect the references, build a prototype, or review a result using judgment that was difficult to write down in advance.

Discovery happens and continues during the work:

• Before implementation, inspect the current system and identify decisions that could change the architecture or user experience. When preferences are difficult to describe, build a cheap prototype.
• During implementation, record meaningful deviations. Stop and reassess when a new unknown changes the risk or direction.
• After implementation, read the code, run the checks, and compare the result with the original intent.

The change brief remains part of this loop. It provides the starting point and records the intent, while the work supplies the information needed to complete it. Only promote durable constraints.

Keep durable facts in their native form

When I say “promote durable constraints,” I do not mean turning every decision into permanent Markdown. That gives us the same stale documentation problem in a different way. Software engineering already provides better versions for most of the necessary, durable facts:

• API shape and compatibility belong in OpenAPI, AsyncAPI, protocol schemas, types, and compatibility tests.
• Data invariants belong in types, database constraints, validation, and migration checks.
• Security rules belong in access policies, static analysis, dependency policies, and runtime enforcement.
• Architecture boundaries belong in module structure, dependency rules, and focused architecture tests.
• Reliability requirements belong in load tests, service objectives, telemetry, and alerts.
• Release rules belong in continuous integration and deployment policies.

These artifacts are already part of delivery. A failed schema check or alert needs to be fixed and handled while the corresponding paragraph in an old design folder does not.

Natural language and specification still have a place in software. Specific domain knowledge like business policy, trade-offs, and even architectural rationale do not always fit into an executable artifact or annotation. I keep that prose short and close to the thing it explains. A small architecture decision record is worth keeping when a future team might otherwise repeat an expensive investigation and a code comment cannot justify the implementation. Recording every local choice just hides the few decisions that matter and confuses the agents that are supposed to build the software. Ask which fact must survive and what its authoritative form should be.

Judgment belongs in the workflow

Heavyweight specification methods try to control quality by prescribing the path. Every change goes through the same documents, reviews, and test categories. That approach creates a lot of attention on low-risk work while avoiding the deep technical judgment needed for harder changes. A copyedit and a payment-flow change should not have to follow the same process or testing strategy.

Simon Willison describes a simpler approach: give the coding agent the outcome and let it judge how much process the task requires. His examples include deciding whether a change warrants automated tests and whether routine implementation can be delegated to a cheaper model while keeping judgment-heavy work in the main loop. This replaces a growing list of procedural branches with one expectation: Choose tactics that fit the work. That matches how I want these systems to operate. And I think it extends to specification and how we document intent.

Agentic changes still require clear boundaries. The team defines the outcome, safety constraints, ownership, and who has authority to accept the result. Within those boundaries, the agent can choose its tactics. When uncertainty introduces consequences beyond its authority, it should surface the problem and ask for a decision.

The workflow then starts matching the risk introduced:

• A small, familiar change can move from a short brief to implementation and review. Almost a one-shot prompt change.
• Unfamiliar code requires factual research before design. Explore codebases, identify implementation details. Preload intent and agent knowledge.
• An unclear user experience calls for prototypes and comparison. And might even require user research after all.
• An architectural change requires explicit human alignment.
• High-consequence behavior requires stronger independent evidence and approval.

I would rather add processes and additional artifacts when the work becomes risky or unfamiliar. Starting every change with the full ceremony just burns time and context.

Context is an engineering budget

Large specifications cost more than the time required to write and maintain them. They also  compete with the code and evidence the agent needs for the current decision. Every requirement, design note, repository instruction, and tool definition consumes part of a limited working context. Extra material burns expensive tokens, but the much bigger cost is lost attention. Important rules become harder to follow when they are surrounded by stale or duplicated material. A spec that leaves too little room for the repository defeats its own purpose.

Progressive disclosure is a better fit. Give the agent a small map, a few stable rules that apply broadly, and pointers to deeper material. A concise AGENTS.md can document build commands, repository layout, and architectural boundaries. It should not narrate every class or repeat API documentation. The file helps humans for the same reason: It tells them where to look without pretending to replace what we will find.

Experience with Research-Plan-Implement shows what happens when the context grows too large. The original workflow moved human review before implementation, but teams ended up with large prompts and plans that could reach 1,000 lines. Engineers reviewed those plans while treating generated code almost like compiler output. The implementation could still drift from the approved plan, which meant that eventually someone had to reconstruct the decision from the code. That problem becomes worse in brownfield systems, while greenfield systems might even survive large plans because they inherited no hidden constraints. Complex changes, in contrast, often inherit behavior that plans may miss.

In “Everything We Got Wrong About Research-Plan-Implement,” Dexter Horthy revisits the original position. Teams shipped more code and then spent much of the gain time cleaning up earlier low-quality output. The implementation could also diverge from the reviewed plan, which forced engineers to reconstruct what happened from the code anyway. The revised workflow uses smaller contexts for factual research, design alignment, structure, implementation, and review. I take a simple lesson from this: Research and design give me leverage, but I still need to understand and own the code that is generated.

Modernization makes this obvious

A mature application contains several kinds of behavior in the same codebase. Some logic represents durable business logic or implements a published interface. Some code exists because an old platform imposed a technical constraint. An incident fix remains long after its  context is gone. And even defects can survive to the point where they almost look intentional when undiscovered.

An agent that treats every code variant as a new target specification can translate those layers faithfully into a new language or architecture. The translation may be technically accurate but also preserves defects and old architecture approaches in newer and cleaner code.

I design changes to brownfield projects similar to the way I did modernizations before the agentic age. Classification and observation are central aspects that I put first. The goals are:

• Preserve durable business invariants and externally required behavior
• Verify behavior that appears active but lacks clear ownership or evidence
• Redesign logic tied to obsolete architectural constraints
• Remove dead paths, duplicated logic, and confirmed defects

You can read a lot about static source code analysis when it comes to brownfield assessments or modernization. You can inspect dependencies and current behavior by executing tests and maybe even adding test cases to secure behavior. What I do recommend is to also embrace mutation testing approaches (e.g., PIT) to find hidden assumptions and failure behavior. Code coverage is also seeing a renaissance because it aids in identifying dead code paths.

On top of that we still ignore operational context and telemetry data. Both are vital elements to not only control but also to help judge existing behavior. All this together helps you judge which elements belong in the system going forward and which don’t. It all starts from code. It is the foundation of the behavior we have. The original and leading specification. A change brief will always be temporary and its sole job is to describe the delta between existing and future functionality. The new implementation and its native checks become the next durable state.

Small specs still need real evidence

Keeping specifications small does not mean returning to a loose prompt followed by hopeful review or even vibe-coding approaches. An agent can turn an underspecified request into a coherent implementation before the missing decisions become visible to anyone. The result may compile, pass the available tests, and look internally consistent. That coherent appearance is part of the risk now. Unapproved business decisions disappear into something very ordinary-looking because they got resolved plausibly.

And this behavior is backed by research. If we look at repairing ambiguous natural-language requirements, for example, we can see that directly asking models to resolve ambiguity often leads to inconsistent or even irrelevant results. Choosing a more targeted repair approach around the identified defects (change brief) improved the results by roughly 31%. SWT-Bench found that generated tests could filter proposed fixes and double the precision of a software repair agent. They used one agent to generate a proposed change and gave another the task to produce evidence to reject it. Lastly, the topic of formal specification generation: One interesting study I found gave 30 models the task to translate natural language into TLA+ (Temporal Logic of Actions, a specification language created by Turing Award-winner Leslie Lamport). The best results only reached about 27% syntactic correctness and 9% semantic correctness. The formal notation helped to detect mistakes, but it did not guarantee correctness or that the translation preserved the original meaning.

These results support focused clarification and independent checks. Clarify the uncertainties that can change the outcome, then verify the implementation with evidence that does not come entirely from the same reasoning path. Generating a longer specification does not solve that problem at all.

I want the strength and independence of the evidence to match the consequence of being wrong. A small internal refactor may need ordinary tests and code review. A change that involves security or financial aspects, or that even touches regulated data, needs a much stronger separation coupled with adversarial review and explicit human approval. For those changes, the agent proposing the implementation should not also be the only source of its requirements and tests.

A lighter operating model

In practice, I want a workflow that I can explain without a complex flow diagram. It starts with the evidence already in the system and makes the intended change explicit. Everything else is added only when the potential risk of the change justifies it. Ideally, this is a simple five-step process:

1. Start from the code and operational evidence that describe the current system
2. Define the intended delta, important boundaries, and known unknowns
3. Add research, prototypes, design alignment, or stronger verification where risk requires them
4. Read and review the implementation, not just the plan
5. At release, discard temporary reasoning and preserve each surviving fact in its native authoritative artifact

That is enough structure to guide the work without building a natural-language replica of the software.

Before implementation, the change brief describes the intended delta, and during implementation it helps people and agents align while new information changes the plan. But after the release the code and production behavior become the primary evidence of what the system does. Not separate documentation in any form that potentially drifts over time.

Durable obligations remain in the artifacts we already know how to maintain: schemas, tests, policies, configuration, telemetry, and short records for rationale that cannot be encoded elsewhere. Most planning details have completed their job by then and should expire.

I expect teams to get the most from coding agents when they are selective: specify what must be decided, discover what the system can answer, verify what carries risk, and let temporary planning go.

Sources

• A Field Guide to Finding Your Unknowns, Thariq Shihipar, 2026.
• Judgement, Simon Willison, 2026.
• Can LLMs Write Correct TLA+ Specifications?, Bisharat et al., 2026.
• Automated Repair of Ambiguous Natural Language Requirements, Jia et al., 2025.
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents, Mündler et al., 2024.
• Large Language Models for Requirements Engineering: A Systematic Literature Review, Zadenoori et al., 2025.
• Advanced Context Engineering for Coding Agents, HumanLayer, 2025.
• Everything We Got Wrong About Research-Plan-Implement, Dexter Horthy, 2026.
• Usage, Effects and Requirements for AI Coding Assistants in the Enterprise, IBM Research, 2026.
• From AI Governance to AI Assurance, IBM, 2026.