Make a Flaky Test Fail Ten Times on Purpose
Flaky test? Make it fail ten times on purpose. A test that fails once in CI and passes on retry can waste a week of re-runs while the team argues about whether it's real. The fix starts with one command: run the suspect spec repeatedly on your own machine until the failure is boring and reproducible — then debug it like any other bug.
Why retries hide real bugs
A retry that turns red into green doesn't fix anything — it reclassifies it. Playwright even reports these as a separate category: a test that fails, then passes on retry, is marked flaky, not passed. Every flaky pass is a real race, timeout, or ordering bug that shipped anyway. And because CI failures are slow to iterate on (push, queue, wait 15 minutes, read logs), teams rationalize instead of reproducing: "infra was slow", "re-run it". Ten local runs in under a minute end that argument with data.
The command
Point Playwright at the one suspect file and repeat it:
npx playwright test tests/checkout.spec.ts --repeat-each=10--repeat-each runs each matched test N times (default 1). If it's flaky, an intermittent failure that shows up 1-in-20 in CI will usually surface within seconds on your machine — now you have a deterministic loop instead of a debate.
Tighten the loop further:
| Flag | What it does | When to add it |
|---|---|---|
--repeat-each=10 | Runs each test N times (default 1) | Always — this is the tip |
-g "adds to cart" | Only runs tests matching a regex | Narrow to the one test, not the file |
--workers=1 | Single worker, no parallelism | Rule parallel interference in or out |
--retries=0 | Zero retries (the default) | Make sure nothing masks the failure |
--trace=retain-on-failure | Keeps a full trace for failed runs | Get a timeline of the failing run |
--headed | Runs in headed browsers | Watch the race happen |
Compare two runs: --repeat-each=10 --workers=1 versus --repeat-each=10 at the default worker count. Fails only with parallelism? You have shared state between tests. Fails either way? The bug is inside the test or the app — usually a missing wait or a real race.
Fix the race, not the symptom
Once it fails on demand, the failure mode is usually one of three: the test asserts before the app settles (fix the wait condition, don't add a sleep), two tests share state (isolate the fixture), or the app itself has a race (congratulations — the flaky test just did its job). --trace=retain-on-failure gives you a step-by-step trace of exactly the runs that failed.
Quarantine is the last resort, not the first. If you must park a test, Playwright's test.fixme() marks it as failing and skips execution — visible in reports, unlike a deleted test — and test.fail() runs it while asserting that it does fail, so you're notified the moment the bug disappears. File the issue when you quarantine; a fixme without an owner is a deletion with extra steps.
Power user: the same move in every stack
- pytest — the
pytest-repeatplugin adds--count:pytest --count=10 test_checkout.py, plus@pytest.mark.repeat(10)for a single test and--repeat-scope(function/class/module/session) to control ordering. - Go —
go test -count=10 -run TestCheckout ./...runs the test ten times;-count=1alone is also the idiomatic way to bypass Go's test-result cache. Add-raceand you'll often get the data race pinpointed for free. - CI gate — run
--repeat-eachon new test files in CI before they merge. A test that can't pass 10 consecutive runs on day one will not get less flaky with age. - AI-coding angle — agent-generated tests are a common flake source (generated waits are optimistic). Make "passes
--repeat-each=10" part of your acceptance loop before committing anything an agent wrote.
Resources
- Playwright test CLI reference —
--repeat-each,--workers,--trace - Playwright test retries — how flaky is detected and reported
- Playwright annotations —
test.fixme(),test.fail() - pytest-repeat —
--countand--repeat-scope - go command testing flags —
-count
Building an AI feature? Yeda AI designs, audits, and ships production LLM systems.