An Untested Rollback Is Not a Rollback
Your rollback plan has never been tested. That is not a plan — it is a hope with a runbook attached. The night your launch goes sideways is the worst possible moment to discover the off switch was wired to nothing.
Why untested rollbacks fail
The failure is not the bug you shipped. The failure is that recovery took 40 minutes instead of 40 seconds because nobody had ever flipped the switch under load. Rollbacks rot silently: the flag gets renamed, the config path moves, a dependency now reads the feature at startup so turning it off does nothing until a restart. None of that shows up until you need it — and by then you are debugging your escape hatch in front of angry users.
Google's SRE practice is blunt about this: the ability to revert to a known good state is fundamental, and every automated change is watched so it can be "quickly reverted" the moment it misbehaves. The reversion path is not an afterthought — it is designed and exercised first.
Prove the kill switch before launch
A kill switch is a feature flag whose only job is to make the new code path stop executing — no deploy, no rebuild, no waiting on CI. The point of decoupling it from deploy is speed: flip the flag and the feature is gone in seconds.
Test it the way you will use it:
- Turn the feature on in a staging or canary environment carrying real traffic.
- Flip the flag off and confirm the old, known-good path is serving again — measured, not assumed.
- Time it. Seconds is the bar. If it needs a redeploy or a restart, it is not a kill switch yet.
- Write the trigger down. Decide the rollback threshold in advance — e.g. "error rate up 0.5% vs. control → flip off" — so the call is arithmetic at 2 a.m., not judgment.
Rules of thumb
| Situation | Do this |
|---|---|
| New risky feature | Ship it behind a flag defaulted off |
| Deciding when to roll back | Write the numeric threshold before launch |
| Incident starts | Flip flag off, restore known-good, then diagnose |
| "I can just hotfix it" | No — a rushed fix under pressure is how you cause the second incident |
| Flag turned off | Confirm the old path actually serves before you exhale |
Flag off beats heroic hotfix
When it breaks, the calm move is boring: flip the flag off, restore the known-good state, and only then diagnose — with the pressure gone because users are already served. The heroic move — patching forward at speed while everything is on fire — is how teams turn one incident into two. Recovery first, root cause second.
Power user: make the switch trustworthy
- Kill the read path, not just the write path. If any consumer caches the flag or reads it once at startup, "off" is a lie until restart. Read the flag at evaluation time.
- Default new flags to off. A flag that fails open defeats the purpose; if the flag service is unreachable, the safe answer is the old behavior.
- Rehearse rollback as a game day. Trigger it deliberately in a low-stakes window. A kill switch that has been pulled once on purpose is one you trust when it counts.
- Clean up after. A permanent flag is permanent tech debt and a new failure mode. Remove it once the feature is trusted.
Resources
- Canarying Releases — Google SRE Workbook — revert to a known good state when a canary misbehaves.
- Reliable Product Launches — Google SRE Book — plan the off switch as part of the launch.
- Feature Flags 101 — LaunchDarkly — the kill-switch pattern explained.
- Feature flags as part of your rollback plan — Harness — flags as the fast revert path.
- What is a feature flag — Honeycomb — best practices and lifecycle.
Shipping features behind flags — and want the off switch to actually work? Yeda AI designs, audits, and ships production systems with recovery built in.