When the Trains Stop, the Runbook Starts: Rehearsing Incidents on Disrupted Days
Photo by Photo by Robert Likovszki on Unsplash on Unsplash
At 08:12 on a Tuesday in February, the departure board at Leeds goes all orange. Everything to London is cancelled — points failure near Doncaster. Your on-call engineer is already on a rail replacement bus, tethering a laptop to a phone with one bar of signal. The incident channel is quiet, not because nothing is wrong, but because nobody can get to a keyboard.
That is the day you want to rehearse for. Not the calm Wednesday when everyone is at their desk with fibre and a full battery.
Why drill on a disrupted day?
Most drills are run in comfortable conditions. Everyone logs in from the office, the VPN behaves, and the runbook is open on a second monitor. That tests the runbook's grammar, not your ability to respond when the team is scattered across the country. A disrupted-day drill tests the bits that actually fail: comms, access, decision-making, and the small matter of whether anyone can find the runbook without the corporate SSO.
You are not trying to ambush people. Announce a window in advance, keep the scenario short, and make it clear that the goal is finding friction, not catching anyone out. The weather does the chaos for you; you just add the scenario.
Out-of-band comms that actually reach people
Assume Slack, Teams and email are unavailable or unread. What then? A phone tree, an SMS bridge, a conference bridge with a dial-in number, a status page that lives outside your primary provider. Test that people can receive the message and act on it. Ask for a read-back: reply with the incident reference and your role. If half the team replies 'what?' you have found a fix.
Keep a one-page comms card in the runbook and somewhere offline — a printed copy in the on-call bag, a pinned note in the password manager. Yes, it feels old-fashioned. So does a rail replacement bus.
Access reviews: can you get in from the kitchen table?
The drill will expose the gap between 'has a work laptop' and 'can actually work'. Check VPN, MFA, hardware tokens, password manager access, and break-glass accounts. Test the break-glass path with a time-box and an audit trail; do not leave it as a rumour in a wiki. Make sure the runbook itself is reachable without the primary identity provider. If your runbook lives behind the thing that is broken, it is not a runbook, it is a wish.
Also check the human stuff: who has a laptop at home, who is on a phone, who needs a dongle, who is the backup for the backup. The drill is a good time to find out that the only person with the database credentials is on a ferry.
Decision logs: write it down while it is happening
During a real incident, decisions vanish into chat. In a drill, you can practise the habit. Use a shared doc with four columns: time, decision, owner, rationale. It does not need to be beautiful. It needs to be true and fast. The log is for the post-incident review, for handover, and for the regulator if one turns up. It also slows the room down just enough to stop people from making the same decision twice.
Assign a scribe. Rotate the role. The scribe is not the note-taker for everything; they capture decisions and actions, not the colour of the incident channel.
Turn findings into runbook edits, not a slide deck
Within 48 hours, run a blameless debrief. Every finding gets an owner and a date. Not 'improve comms' — 'add SMS bridge number to runbook page 2 and test it with three responders by 30 November'. Update the runbook, the access list, the comms card, and the decision log template. Then re-test the top three fixes next quarter. If you do not re-test, you have written fiction.
Measure a few things: time to acknowledge, time to first out-of-band contact, time to gain access, time to start the decision log. Trend them across quarters. Do not turn them into a league table for individuals.
The box-ticking trap
A drill can become theatre. People perform the runbook they wish they had, the facilitator nods, and everyone goes back to the desk. Avoid that by making the scenario mildly unpleasant but safe: one inject where the VPN fails, one where a key person is unreachable, one where the status page needs an update. Keep it to 60–90 minutes. Observe, do not rescue. If the facilitator keeps stepping in, you are testing the facilitator, not the system.
A simple quarterly format
Pick a disruption window — a weather warning, a strike day, engineering works on your line. Write a one-page scenario. Assign a facilitator, a scribe, two or three responders, and an observer. Run it. Debrief. Ticket the fixes. Put the next one in the calendar.
The point is not to have a perfect drill. The point is to make the real day less surprising. The rail replacement bus is coming. Make sure your runbook can get on it.
