Español

Flaky Tests Trace To Time, IO, Or Randomness

Your test isn't flaky. Your clock is. When a test passes on one run and fails on the next — same code, same input — the test almost never contains a mystery. It contains a hidden dependency on something the run doesn't control: the wall clock, the file system, or a random number generator. This is doubly true for AI-generated tests, which happily reach for datetime.now(), hard-coded paths, and unseeded randomness because that's what most training-set code does. The fix is the same in every language: find the nondeterministic input and inject it.

The three usual suspects

Martin Fowler's classic catalog of non-determinism lists time, isolation, async waits, remote services, and resource leaks; in day-to-day unit tests, three of those dominate:

1. Time. now() returns a different value every run. Any assertion within a hair of a boundary — midnight, month end, a 30-day expiry — fails on the runs that land on the wrong side. Fowler's rule: "Always wrap the system clock, so it can be easily substituted for testing."
2. File system and shared state. Tests that write to a real, shared path collide with each other, with leftovers from previous runs, and with parallel workers. Google's testing blog names shared state and environment as core flake sources.
3. Randomness. An unseeded RNG means every run exercises a different input. The test isn't checking your code — it's sampling it.

Inject all three

The mechanism is dependency injection, applied to the boring stuff:

from freezegun import freeze_time
import random

@freeze_time("2026-01-14")
def test_expiry(tmp_path):
    rng = random.Random(42)              # same sequence, every run
    cache = Cache(dir=tmp_path,          # unique dir per test
                  rng=rng,
                  ttl_days=30)
    cache.put("k", "v")
    assert not cache.expired("k")        # clock is frozen — no midnight roulette

Same input, same result, every single run. Note the pattern: the production code takes dir, rng, and (implicitly, via freezegun) the clock as inputs instead of reaching for globals. That's the whole trick.

Why AI-generated tests flake more

Code assistants generate the statistically common version of a test, and the statistically common version calls datetime.now() directly, writes to /tmp/test.txt, and calls random.random() bare. None of that fails on the first run — it fails on the 40th, in CI, at 11:59 PM. So review AI-generated tests for exactly these three calls before merging. A 10-second grep — now(, random(, hard-coded paths — catches most of it.

Power-user moves

• Make determinism the default, not the patch. Design constructors and functions to accept a clock, an RNG, and a base path with production defaults (now=datetime.now, rng=random, dir=None). Then tests inject; production doesn't even notice.
• Seed per test, not per suite. A module-level random.seed(42) breaks the moment tests run in a different order or in parallel. Use random.Random(42) instances — the docs are explicit that they don't share state with the global generator.
• Freeze, then travel. freezegun freezes datetime.now(), time.time(), date.today() and friends in one decorator; you can also take the frozen object as a parameter and tick it forward to test the expiry actually firing — the failure path, not just the happy path.
• Quarantine, don't delete. Fowler's advice for an already-flaky suite: move flaky tests to a quarantined set so they can't poison trust in the green build, then burn the quarantine down using the three-suspect checklist.
• Reproduce a flake by reversing the trick. Can't reproduce a flaky failure? Force the conditions: set the clock to 23:59:59, run tests in random order, run them in parallel. A flake you can reproduce on demand is just a bug.

Resources

• Eradicating Non-Determinism in Tests — Martin Fowler
• Flaky Tests at Google and How We Mitigate Them — Google Testing Blog
• How to use temporary directories in tests — pytest tmp_path docs
• freezegun — freeze datetime in Python tests
• random — seed() and Random(seed) instances — Python docs