Every organization I’ve worked in had a disaster recovery plan. Most of them had never been executed by the people who would actually have to execute it.
That’s the gap. Not the absence of a plan — the absence of anyone who has done it. A plan nobody has run is a set of assumptions written down neatly, and you find out which assumptions were wrong at the exact moment you can least afford to.
What the paper exercise misses
Tabletop walkthroughs are useful for sequencing and for finding gaps in who calls whom. They’re worth doing. They just don’t test the things that fail.
The things that fail are boring and physical:
- Credentials. The recovery account expired. The person who had it left. The password vault is in the environment you’re recovering from — that one genuinely happens, and it’s fatal.
- Dependencies nobody mapped. The application comes up and can’t authenticate because the directory it depends on is in the failed site, and nobody listed it because everybody knows about it.
- Capacity. The recovery environment is sized for the application but not for the batch load, or for everyone reconnecting at once.
- Data currency. Replication was three hours behind. Documented as fifteen minutes. Nobody had checked in a year.
- Time. The runbook says four hours. Nobody has ever timed it. It’s eleven.
None of these show up in a walkthrough, because in a walkthrough everyone says “and then we’d fail over” and moves on.
Test what the business actually needs
The unit of recovery isn’t a server, it’s a business process. “The database is up” is not recovery. “Payments can be processed” is recovery.
That reframing changes what you test, because it drags in everything the process touches — the interfaces, the file transfers, the third party you don’t control, the printer in the branch. Recovering the systems you own and discovering the process still doesn’t work is a very common and very expensive outcome.
For revenue-critical systems I’d rather test one full business process end to end, badly, than fifty components cleanly.
Test in pieces, often
The annual full-scale failover is a big enough event that it gets deferred, and deferred, and then done under such controlled conditions that it proves very little.
Smaller and more frequent beats bigger and rarer:
- Fail over one component with the real runbook and the real on-call person.
- Restore one database from backup to a scratch environment. Actually open it.
- Have someone who has never done it walk the first thirty minutes.
Every one of these finds something, and finding it on a Tuesday afternoon costs you nothing.
Write down what failed
The output of a DR test isn’t a pass. It’s a list.
I get suspicious of a clean test result — it usually means the scope was narrow enough to guarantee success, which is the opposite of the point. A test that finds five problems has just bought you five problems you don’t have during a real event.
The only genuinely bad outcome is a test that finds nothing and teaches nobody anything. That’s not readiness. That’s a rehearsal of the parts you were already confident about.