Never Make a Flaky Test a Required Gate
Re-running a red build is training your team to ignore CI. The first time a required check goes red and someone clicks "re-run" and it turns green, they've learned a lesson: red doesn't mean broken, it means try again. Do that a few dozen times and the whole suite becomes background noise — nobody reads the failure, they just retry until the dice come up green.
A pass-on-retry is a defect, not a hiccup
A test that passes on the second run with no code change is, by definition, non-deterministic. Something raced: a timing assumption, a shared fixture, an order-dependent global, an unmocked network call. The flake is a signal — it's often the only symptom of a real race condition you also have in production. When you paper over it with automatic retries, you throw away the signal and keep the bug.
Martin Fowler put the cost bluntly: non-deterministic tests are "a virulent infection that can completely ruin your entire test suite." His arithmetic is the scary part — a suite of 100 tests with just 10 flaky ones will fail most runs. And "once that discipline is lost, then a failure in the healthy deterministic tests will get ignored too." One flake doesn't cost you one test. It costs you the credibility of every test.
Quarantine, don't require
The fix is boring and it works: keep the flaky test out of the required gate. Three moves, in order:
| Step | What you do | What you must NOT do |
|---|---|---|
| 1. Quarantine | Move the test to a non-blocking suite that still runs and still reports | Don't delete or skip it — you'll lose the signal |
| 2. File the fix | Open a ticket with an owner and a deadline | Don't leave it ownerless — quarantine becomes a graveyard |
| 3. Report, don't block | Let it show red on the PR without failing the merge | Don't add retries: 3 to the required job |
The test keeps running. It keeps reporting. It just can't hold another engineer's PR hostage while someone chases the race. Now green means green: every required check that passes actually passed, deterministically, and nobody is learning to reflexively hit retry.
Power tricks
- Cap the quarantine. Fowler's rule from the Mingle team: allow at most 8 tests in quarantine, and none for longer than a week. Hit the cap and you stop feature work to clear it. A cap is what stops quarantine from becoming a permanent junk drawer.
- Run quarantined tests last. Put the flaky suite in the pipeline after the healthy deterministic tests pass. You still get the coverage, and the ordering pressures the team to fix flakes rather than accept them.
- Never wire retries into the required job. Auto-retry on the blocking gate is the anti-pattern this whole tip is about — it's institutionalized "hit retry until green." If you retry at all, retry only in the quarantined, non-blocking lane.
- Track flake rate as a metric. A test that fails 1 run in 50 is quarantine material long before a human notices the pattern. Log pass/fail per test and quarantine on a threshold, not on vibes.
Resources
- Eradicating Non-Determinism in Tests — Martin Fowler (the quarantine mechanism, the 8-test cap, the "virulent infection" argument)
- Flaky Tests at Google and How We Mitigate Them — Google Testing Blog
- About protected branches — GitHub Docs (what a required status check is and how it blocks merges)
- Test Quarantine — Mergify
Shipping with AI agents? Yeda AI helps teams build CI and test discipline that agents (and humans) can trust.