Español

Shuffle Your Test Suite to Expose Order-Dependent Tests

Shuffle your tests, then watch them break. If a suite only passes in one specific order, it isn't testing your code — it's testing an accident. Hidden shared state between tests is a top cause of flaky builds, and it stays invisible until something changes the run order: a new file, a parallel worker, a CI machine. Randomizing the order on purpose surfaces those failures on your machine, today, instead of as a random red build next month.

Why order dependence hides so well

In a fixed order, test B always runs after test A — so B silently inherits whatever A left behind: a cached singleton, a mutated module-level object, a row still sitting in the test database, an environment variable nobody unset. Every local run replays the same order, so every local run is green. Then CI splits the suite across workers, or someone adds a file that sorts earlier, and the dependency snaps. The failure looks random; the cause never was.

The fix for detection is one flag away: make random order the default, so any new coupling breaks immediately, while the guilty commit is still on top.

The shuffle flag in your runner

Every one of these prints or accepts a seed. That's the whole workflow: shuffle finds the failure, the seed replays it deterministically while you debug.

# Vitest: shuffle tests, then replay the exact failing order
vitest run --sequence.shuffle.tests
vitest run --sequence.shuffle.tests --sequence.seed 1734

# pytest with pytest-randomly: rerun the last recorded seed
pytest --randomly-seed=last

Fix the state, not the order

When a shuffled run fails, resist the urge to pin the order back. The failure is telling you two tests share state. Typical fixes:

• Reset in setup, not teardown. A crashed test skips its teardown; setup that starts from a known-clean state is immune.
• Kill module-level mutables. Anything created at import time and mutated by tests is a shared-state bug waiting for a reorder.
• Fresh fixtures per test. New DB transaction (rolled back), new temp dir, new client instance — cheap insurance against coupling.

Power-user notes

• Make it permanent in CI. Run shuffled on every build; log the seed in the job output so any red build is replayable.
• pytest-randomly does more than reorder. Before each test it also resets random.seed() to a fixed value (and the seeds of NumPy, Faker, and factory boy), so tests that lean on randomness become deterministic too. Temporarily disable everything with -p no:randomly while bisecting.
• Jest seed introspection. --showSeed prints the seed in the report summary, and jest.getSeed() exposes it inside a test file — handy for seeding your own data generators from the same value.
• Go reports the seed either way. -shuffle=on seeds from the system clock but always prints the seed, so a flaky CI run is reproducible after the fact.

Make order independence a rule, not a hope — a suite that passes in any order is a suite you can parallelize, split, and trust.

Resources

• Vitest CLI — --sequence.shuffle and --sequence.seed
• Jest CLI — --randomize, --seed, --showSeed
• pytest-randomly — shuffle + seed reset plugin for pytest
• Go testing flags — go test -shuffle
• RSpec — --order rand and --seed