Español

A Test That Passes Against Empty Code Tests Nothing

Would your tests still pass if you deleted the production code? Then they are testing nothing. AI assistants write tests fast — dozens in a minute — and a green suite feels like safety. But green only means "no assertion failed." A test that can't fail against an empty implementation is a test that can't catch the bug that empty implementation represents. It's theater.

The false-green problem

Line coverage tells you which code your tests executed, not whether they would notice a fault in it. The PIT project puts it bluntly: "Traditional test coverage measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code." A test can walk every branch, assert something vacuous like assert.ok(true), and report 100% coverage while verifying nothing.

AI-generated tests make this worse, not because the model is careless, but because it optimizes for a passing suite. If the fastest way to green is a tautology — asserting the mock was called with the value the test itself set up, or catching-and-ignoring the exception under test — you'll get a tautology. You asked for tests; you got a green checkmark.

The five-minute audit

Prove your tests can fail. Pick the module the AI just "covered" and break it on purpose:

# 1. Gut the implementation (stub every function to return nothing)
# 2. Rerun the suite
pytest tests/test_orders.py

# Real tests: go red immediately.
# Theater: still green. Delete or rewrite them.

Stub the function body to return None (or pass, or throw new Error()), rerun, and sort tests into two piles. Red means the test earns its keep. Green means it never depended on the behavior at all. This is the manual version of mutation testing, and it takes five minutes per module.

Smells that predict a fake green

That last row is the cheapest habit with the biggest payoff: every bug fix ships with a test that failed before the fix and passes after. If you never saw it red, you don't know it works.

Automate it: mutation testing

The delete-the-code trick, industrialized. Mutation tools seed small faults — flip < to <=, add 1 to a literal, swap break and continue — then run your suite. If tests fail, the mutant is killed; if they pass, it survived, and a survivor marks a spot where a real bug would sail through.

• Python — mutmut run mutates your code and drives pytest; mutmut browse shows survivors in a terminal UI.
• JavaScript/TypeScript, C#, Scala — Stryker reports a mutation score: the percentage of mutants your tests killed.
• Java/JVM — PIT integrates with Maven and Gradle and overlays mutation results on line coverage.
• Rust — cargo mutants finds "places where bugs can be inserted without causing any tests to fail."

This isn't academic. Google wired mutation testing into mandatory code review: its system has been used by ~6,000 engineers, surfacing surviving mutants on the diffs they author and review.

Power moves

• Run the audit on AI output by default. Treat "the tests pass" as the start of review, not the end. Ask the assistant itself to stub the implementation and rerun — it's a one-prompt check.
• Mutate only the diff. Full-codebase mutation runs are slow; Google's approach analyzes changed lines only. Run mutation tools on the files touched in the PR.
• Ban the placeholders mechanically. A lint rule or grep in CI for assert.ok(true)-style tautologies costs one line and never sleeps.
• Red first, then green. For any new test — human or AI — watch it fail once before you trust its pass.

Resources

• mutmut — mutation testing for Python
• Stryker — mutation testing for JS/TS, C#, and Scala
• PIT — mutation testing for Java and the JVM
• cargo-mutants — mutation testing for Rust
• State of Mutation Testing at Google (research paper)