Field note
Rollback drills that fail on purpose
Why a failed rollback rehearsal is often more valuable than a clean run—and how we structure drills inside DevOps enablement sprints.
Teams often schedule a rollback drill hoping for a reassuring green result. We prefer the opposite posture: design the drill so something awkward is likely to appear—stale image tags, missing runbook steps, or an on-call alias nobody monitors.
What we script into the hour
- Nominate a driver who did not write the original deploy job.
- Freeze “ask the senior in chat” as an escape hatch for the first twenty minutes.
- Require the rollback target to be a specific digest, not “whatever we ran last week.”
- Capture every pause longer than two minutes as a note, not a personal failing.
After the drill
The write-up should name owners for each friction point. A DevOps enablement sprint is a good container for that follow-through; a one-off meeting without owners becomes folklore again by the next quarter.
If your last drill was flawless, ask whether it tested the path you would actually use at 02:00—or the path that works at 14:00 with half the platform team on the call.