Prove The Bug Before You Fix It
Never let AI fix a bug it has not proven exists. When a bug report arrives, the tempting prompt is "here's the error, fix it" — and the agent will happily produce a plausible-looking patch. But without a reproduction you have two unknowns instead of one: you don't know the patch addresses the real cause, and you won't know if the bug quietly returns next month. One extra sentence in your prompt removes both.
Why "just fix it" is guessing
A coding agent given only an error message pattern-matches to the most likely cause. Often that's right. When it's wrong, the failure mode is nasty: the code changes, the error message disappears (or moves), and everyone assumes the bug is dead. You have no signal separating fixed from masked.
A failing test flips the situation. Before any fix, the agent must write a test that reproduces the reported behavior and run it to watch it fail. That failing run is evidence: the bug is real, it's captured in executable form, and the test's assertion documents what correct behavior looks like. Only then does the fix begin — and "done" means the exact same test passes, not "the diff looks reasonable."
This is the classic test-first loop from test-driven development applied to bug fixing, and it's the standard reflex on teams with self-testing code: first write a test that exposes the bug, only then try to fix it. Agents make the discipline nearly free — the test costs you one sentence of prompt instead of twenty minutes of your time.
The prompt pattern
Instead of:
users report login fails after session timeout. fix itwrite:
users report login fails after session timeout. check the auth flow
in src/auth/, especially token refresh. write a failing test that
reproduces the issue, run it and confirm it fails, then fix the code
until that test passes. show me both test runs.Three load-bearing pieces:
| Piece | What it buys you |
|---|---|
| "write a failing test that reproduces the issue" | Forces a concrete reproduction before any code changes |
| "run it and confirm it fails" | Blocks the agent from writing a test that accidentally passes (proving nothing) |
| "show me both test runs" | You review evidence — a red run then a green run — instead of trusting a summary |
The middle row matters most. A reproduction test that passes on first run means the agent misunderstood the bug — better to find that out before the "fix."
Why this suits agents especially well
Agents work best with a check they can run: a signal that returns pass or fail closes the loop, so the agent iterates against the test instead of stopping when the code "looks done." The failing test is exactly that check — the agent fixes, runs, reads the result, and keeps going until it's green. You've turned a judgment call ("does this patch look right?") into a machine-verifiable condition.
The same principle appears in vendor guidance across the board: give the agent verification criteria and demand evidence, not assertions of success.
Power-user moves
- Keep the test forever. The reproduction test becomes a regression test. If the bug ever comes back, CI catches it in minutes instead of a user catching it in production.
- Name it after the bug.
test_login_fails_after_session_timeout(or the issue ID) turns your test suite into a searchable history of every bug you've killed. - Park known-unfixed bugs with an expected failure. In pytest, mark the reproduction with
@pytest.mark.xfail— the docs call out "a bug not yet fixed" as a canonical use. The bug stays visible in every test run without breaking the build; when someone fixes it, the mark comes off. - Split writer and reviewer. Have one agent session write the failing test, and a fresh session write the fix. The fixer can't game a test it didn't write, and a fresh context won't be biased toward the code that caused the bug.
- Demand root cause, not symptom suppression. Add "address the root cause, don't suppress the error" — otherwise a determined agent can make a test pass by special-casing the exact input.
Resources
- Self-Testing Code — Martin Fowler — "first write a test that exposes the bug, and only then try to fix it"
- Test-Driven Development — Martin Fowler — the red → green → refactor loop this tip borrows
- Claude Code best practices — provide verification — "write a failing test that reproduces the issue, then fix it"
- pytest: skip and xfail — marking a reproduction for "a bug not yet fixed"
- GitHub Copilot coding agent — get the best results — scoping tasks so the agent can verify its own work
Building an AI feature? Yeda AI designs, audits, and ships production LLM systems.