I’ve sat on a lot of major incident bridges. The bad ones are never bad because the wrong people were on the call. They’re bad because everyone on the call assumed someone else was in charge.
Forty people join. Somebody starts debugging out loud. Someone else joins late and asks for a recap, so the debugging stops. An executive joins and asks for an ETA, so the debugging stops again. Ninety minutes in, the only artifact is a very long call.
The fix isn’t technical. It’s deciding, before the incident, who does what.
Three roles, and they’re different people
- Incident commander. Runs the call. Does not debug. Their entire job is deciding what happens next, who is doing it, and when we check back. If your best engineer is commanding, you’ve taken your best engineer off the keyboard.
- Communications. Owns updates to everyone not on the bridge — the business, the service desk, the executives. This exists so the commander is never interrupted to answer “what do I tell the customer.”
- Technical lead(s). Actually fixing it. Reports findings to the commander. Doesn’t manage the call.
One person can hold two of these on a small incident. Nobody should hold all three on a big one.
The commander’s script
I’ve never needed more than four questions, on a loop:
- What do we know? Facts only. “The database is slow” is a fact. “It’s probably the storage array” is not.
- What are we trying? One thing at a time, with an owner and a time box. “Ravi is rolling back the config, we check back in ten.”
- What’s the impact right now? Who can’t work, what can’t process. This is what comms needs.
- What’s our next decision point? If the current attempt fails, what do we try, and when do we call it.
Then say all four out loud every fifteen minutes, even when nothing has changed. Especially when nothing has changed. Silence on a bridge is how you get six people independently deciding to try something.
Stop debugging and restore
The hardest call a commander makes is telling a good engineer to stop.
There’s a moment — usually thirty or forty minutes in — where the team is close to understanding the cause, and restoring service would destroy the evidence. Restore anyway. Take whatever you can capture on the way past, and put the rest in a problem record.
Service comes first. Curiosity is a problem-management activity, and it has a much more forgiving clock.
After the call
Two things, and neither is a blame conversation:
- A timeline while people still remember it. Not a polished document — just what happened and when. Do it the same day or you’ll be reconstructing it from chat logs in a fortnight.
- A problem record, if it’s the sort of thing that can recur. Most things can.
The bridge call that ends with “great work everyone” and no artifact is the bridge call you’ll run again.