There is a failure mode I have watched play out more than once across financial services and large enterprise environments: a change goes through every gate, collects every signature, clears the change advisory board, and then breaks production in a way that the approval process had every opportunity to catch.
The post-incident review surfaces the same finding each time. The change was approved. The implementation plan was attached. The backout procedure was documented. And nobody had actually run any of it in an environment that resembled production before the maintenance window opened.
Approval is a governance activity. Testing is a risk-reduction activity. They are not substitutes for each other, and treating the first as evidence that the second happened is how you get outages that were fully authorized.
The distinction matters because it changes what you ask during review.
Most change review processes — formal CAB or otherwise — are built around documentation completeness. Is there an implementation plan? Is there a backout? Has the requestor identified the risk tier? Those are legitimate questions. They tell you whether the person submitting the change thought carefully about it. They tell you almost nothing about whether the change will work.
What tells you whether the change will work is evidence from a pre-production execution: what the deployment log actually showed, what the smoke test actually covered, what the validation steps confirmed. Not what the plan says will happen — what happened when someone ran it.
When I was in large-scale infrastructure and operations roles, the changes that caused the worst incidents were rarely the ones that skipped the process. They were the ones that completed the process correctly and skipped the test. The paperwork was immaculate. The environment behavior was a surprise.
The practical problem is that organizations conflate test environment parity with test execution.
Having a staging environment is not the same as having run the change in staging. Having a UAT sign-off on the application does not mean the deployment procedure for the infrastructure supporting it was exercised. Having a rollback script in the ticket does not mean the rollback script was invoked against anything resembling a real database before someone invoked it against the real one at 2 a.m.
This is not a process design failure in the abstract. It is a resource and timeline failure in the specific. Testing takes time. Pre-production environments are often stale, undersized, or shared with other work in ways that make a clean test inconvenient. The change window is scheduled. The pressure is to move. The approval exists. Close enough.
Close enough is where the outage lives.
The fix is not more gates. It is making test evidence a submission requirement rather than a courtesy.
That means defining, for each risk tier, what evidence of pre-production execution is required before a change enters final review. Not a checkbox that says testing was completed. Actual output: a deployment log, a test case result, a record of the rollback being validated. Something that could be reviewed after an incident and confirm that the test happened and what it showed.
It also means being honest about what your pre-production environments can actually support. If staging is six months behind production on schema, a successful staging deployment is weak evidence. That does not mean you skip it — it means you document the gap explicitly as a residual risk the approver is accepting, not a gap nobody mentioned.
Ownership matters here the same way it does everywhere else in service management. Someone has to be accountable for confirming that test evidence exists and is sufficient before the change moves. If that accountability sits with the requestor alone, with no independent check, it will be rationalized away under schedule pressure every time.
The change process exists to reduce the probability that a change causes an incident. Approval is one input to that. It is not the output.
If your review process produces approvals but not test evidence, you have a documentation process dressed up as a risk control. The production environment will eventually tell you the difference.