Post-Incident Playbook for Building Resilience
The outage is finally resolved and the incident dust has settled. So you do a postmortem, craft a plan to address the issues and deploy it. Now you can finally move on and get back to making progress on your roadmap.
At least until another outage takes your system down and the whole circus starts again.
Sound familiar?
It's an all-too-common loop that's painful for engineers and companies. Unreliable systems mean engineers are constantly pulled off important work, delaying releases and contributing to burnout, while outages lose revenue, damage brands, and even lead to regulatory fines.
Theoretically, we should be building more resilience with every incident, but in practice, the rapid deployment of code means we're always behind—and it's only gotten worse with AI-driven development.
It's time to break the cycle with a proactive reliability management practice built around preventative testing and risk resolution with provable results.
And the conversation needs to happen now as part of the postmortem process. Not next quarter or next year when your organization is "more mature," but when everyone has just felt the impact of poor resilience and the outage is fresh in your minds. Because if you don't, then it's only a matter of time until the next failure.
Following this approach and these best practices, hundreds of leading companies have improved their availability, decreased outages, and broken the endless cycle of disruptive incidents.
Resilience has to be verified with real failures
Every piece of code undergoes some testing before it's deployed. But it's impossible to do fully exhaustive QA testing, and there are many failure modes that don't show up when you test code in isolation. The only way to really know whether your service will be resilient is for it to face failure in a production-like environment with real traffic levels.
Which leaves you with two options:
You can wait until an actual failure mode occurs in production and causes an incident, then react as fast as you can.
You can safely simulate those failure conditions and verify resilience before it causes an outage.
The first option is, sadly, more common, but the second option is the more prudent one. It's also the only way to break the firefighting cycle. By using fault injection testing, you can create the conditions of failure, such as a dependency being unavailable, increased latency, or a region outage, see how your systems react, then safely restore back to the previous state.
And by integrating the tests with your observability tool's existing monitors and alerts, you can create a pass/fail gate based on signals you care about. If metrics don't hit the alert threshold, then the code passes and is resilient. If it does, then you've identified the point of failure and have an opportunity to fix the issue before it causes an outage.
When applied correctly, this kind of testing can help you avoid outages outright. And if you test regularly with automation, a natural cultural shift will start to happen where engineers proactively build better resilience and are more efficient.
How to determine which failures to test
The first thing you should test is the failure that just occurred. Yes, you patched it, and it's tempting to assume this means it won't happen again…but you can't be 100% sure. And that means you can't prove resilience to leadership and the rest of the company.
As an example, say a dependency outage of your buyer rewards service caused cascading failures with your checkout service. After resolving the incident, you built circuit breakers to prevent the situation from happening again.
It's logical to assume that a circuit breaker will do the job, but if you want to prove it, then you should run a test that simulates the same outage by cutting off all traffic to the buyer rewards service. If the circuit breaker engages and the purchase can still go through, then your fix was successful. If it doesn't, or if it engages and the service still goes down, then it's far better to find out now than when a customer tries to check out.
Once you're done testing that failure, you should also branch out to similar failures. After all, just because the checkout service can withstand the failure of one dependency doesn't mean it's resilient to all dependency failures.
| Outage Cause | % of companies affected | How to test for resilience |
|---|---|---|
| Network failure | 35% | Verify service redundancy and failover by blocking network traffic |
| Third-party or cloud provider failure | 28% | Verify resource/dependency redundancy by blocking network traffic |
| Deploying software changes | 28% | Run a standardized group of tests in staging to verify resilience before deployment |
| Someone making a change to the environment | 26% | Automate test suites to run in production and verify continued resilience |
| Security failure | 24% | Follow standard security practices |
| Hardware failure | 23% | Verify resource redundancy by blocking network traffic to a node |
| Power failure | 22% | Verify resource redundancy by blocking network traffic to an availability zone or region |
| Capacity constraint | 20% | Verify service resilience by blocking traffic to a percentage of nodes in a cluster |
| Unexpected traffic surge | 18% | Verify scalability by consuming resources (CPU, memory, disk I/O, etc.) |
| DNS issue | 18% | Verify DNS redundancy by blocking network traffic to your primary DNS provider |
| Certificate expiration | 13% | Ensure no SSL/TLS certificates are expiring soon with a passive check, including third-party dependencies |
SOURCE: New Relic 2025 Observability Forecast
The failures in the table above match the most common outage causes with tests to prove resilience. Obviously, there's an entire other practice for 24% of security outages, but the majority of the other causes can be tested using resilience testing. To truly improve the reliability of your application and its resilience to failure, you should test all of these other causes to eliminate as many reliability risks as possible.
Measuring reliability and automating at scale
No individual service exists in a vacuum. Every application and system out there is a web of interconnected dependencies, and one that's almost constantly changing. You might increase the reliability of the checkout service, but that can only go so far if the payment processing service isn't also resilient.
Improving reliability has to happen across the whole application, which means you need a systematic way to track changes in the reliability of its individual pieces. And for that, you need a single metric that can apply to all services.
Unfortunately, it's impossible to measure reliability with incident response metrics like MTTR, uptime, or incident counts. All of these are backward-looking: they'll show you what already happened, but not what will happen.
Instead, you can use the test results to create a forward-looking reliability metric. After all, the whole point of the test is to show you how your service will react in the future.
Start by setting policies around a standard group of tests. The most common failures listed above are a great place to start, and give you a broad range of coverage. Once you have results from those tests, you can use them to produce a reliability percentage.
Say you had a group of ten tests you ran. Passing all of them would indicate that the service is 100% resilient against those failures, which means a passed test would be worth 10 percentage points. However, a failed test is worth zero, as it means there's no resilience against that failure mode. So if your service passed seven tests and failed three, then it's 70% resilient against those particular failures. If you fix something and now it passes eight tests, it becomes 80% resilient. (Note: For this scoring, an inconclusive test, such as one that didn't run correctly, should also be scored zero because you can't be sure your service would've passed.)
As you track these results over time, you get a metric that shows you when the reliability of the service improves or worsens. And if you automate the same group of tests to run across all of your services, now you have a consistent reliability metric that shows you, and your leadership team, the resilience of your application and which outages you're actively avoiding.
That's how you really break the loop: systematic, standardized reliability measurement across your organization. Because when you can predict your system's response to failure, you're no longer just reacting to outages—you're preventing them.
