How to Run a Blameless Post-Incident Review, Step by Step

How to Run a Blameless Post-Incident Review, Step by Step

It was a quiet Tuesday when the checkout page started returning errors. The on-call engineer rolled back within minutes, the pages recovered, and then the real question arrived: what do we do now? Some teams reach for a name. A better team reaches for a notebook. This how-to follows one small, invented incident from first alert to final fix, so you can see how a blameless post-incident review works step by step and borrow the method for your own pipeline.

The incident in brief

A routine release went out in the afternoon. It contained a harmless-looking configuration change: a database connection setting copied from a staging file into the production template. The setting pointed at a pool size that suited a test environment and starved the real one. Within a short time, requests queued, timeouts rose and the checkout path failed. Monitoring caught it, the engineer on call rolled the release back, and service returned. Nobody was hurt, but customers had seen errors, and the team wanted to be sure it would not happen again.

Step one: stabilise first, review second

Do not start the review while the fire is still burning. Restore service, confirm it with a few real requests, and tell stakeholders in two lines what happened and when the next update will come. Once things are calm, schedule the review within a day or two, while memories are fresh, but after people have slept. A tired review produces tired conclusions.

Step two: set the ground rules aloud

The facilitator opens with three sentences. We are here to understand how the system allowed this. We will describe actions, not judge people. We will leave with one change. Saying this at the start sounds ceremonial, yet it is the difference between a candid room and a defensive one. The facilitator should not be the person who shipped the change, which keeps the tone neutral.

Step three: build the timeline from records

Gather facts from version control, deployment logs, alert history and chat. List events in order, each with a time and a neutral description: “release prepared”, “template updated”, “pipeline passed”, “traffic shifted”, “alert fired”, “rollback started”. Then add a second column for what the person involved could see at that moment. In our example, the pipeline reported success on every stage, which meant the engineer who approved the release had every reason to feel confident. That single observation reshapes the conversation immediately, because it shows that the checks were green and the system still failed.

Step four: ask how, not who

Now work through questions that probe conditions instead of character.

  • How did a staging value end up in a production template?
  • What did the pipeline check, and what did it not check?
  • How quickly could we tell that the change was harmful, and from which signal?
  • What made rollback easy or hard?
  • Was the review of the change shaped by the size of the diff or the time of day?

In the story, the answers formed a tidy picture. Template files for different environments lived side by side with near-identical names. Review of configuration changes was lighter than review of code. And the pipeline validated that the file was well formed, but not that its values were within sensible ranges for the target environment.

That last point is where careful engineering lives. A deeper look at how requirements are captured up front, as in this deep dive on drawing up web project requirements, often reveals that operating limits were never written down anywhere a machine could check them.

Step five: choose exactly one change

The team listed candidates: rename the files, add a reviewer rule for configuration, add a range check in the pipeline, retrain everyone on environments. All reasonable, but a review that ships four changes ships none. They chose the one that removes the possibility of the failure rather than relying on attention: a validation step in the pipeline that fails the build whenever the production pool size falls outside an agreed range. Teams that have already invested in automation will recognise the pattern; the argument in streamlining a workflow so that automation trims slips and saves hours follows the same logic of turning a human memory task into a machine check.

Step six: give the change an owner and a test

Write the change as a single sentence with an owner by role, a due date and a way to verify it. “The platform engineer adds a range check for connection pool settings to the release pipeline by the end of next week; we will confirm by deliberately submitting an out-of-range value in a test branch and watching the build fail.” That final clause matters, because a safeguard that has never been seen failing is only a hope.

Step six and a half: tidy the alerting while you are here

Reviews often expose a second, smaller gap in the alerting itself. In the story, the first alert fired only after errors had been climbing for several minutes, because the threshold was tuned for slow drifts rather than sudden stalls. The team noted it as a follow-up, deliberately separate from the headline change, so the main fix would not get buried. Keep a short parking list of such side findings; schedule them for later sprints individually, and do not stretch today’s review to cover them.

The communication side deserves a glance too. Did customer support learn about the problem from the dashboard or from an angry email? If the latter, add a line to the incident checklist naming who posts the first internal notice. It is a tiny change that spares everyone a lot of confusion on the next occasion.

Step seven: share the story without names

Publish a short summary where the whole engineering group can read it. Use roles, not names. Include the timeline, the contributing conditions and the change. The point is to give the next person a head start. When a colleague later proposes a similar configuration tweak, the summary tells them to check the range, and the review quietly pays for itself.

A note on keeping perspective

Engineers also have lives beyond the pipeline, and team debriefs often end with a meal out. A restaurant discovery website such as FindMyResto, where diners can search, browse menus, find directions and book a table and owners can list their place, might help the group pick somewhere in a few minutes. The relevant habit carries over, though: if the table is lost, ask how the plan allowed it, not who to scold.

Step eight: look back

Two or three weeks later, return to the note. Did the check run? Did it catch anything? Did it cause new friction, such as false alarms? Adjust the range, the wording or the owner if needed. A blameless review is a loop, and the loop is closed only when someone checks that the fix survived real use.

The charm of this method is how small it is. One incident, one conversation, one change. Repeat that pattern a few dozen times and a pipeline that once relied on vigilance starts to rely on design. That is a quieter, calmer way to run production, and it is one that whoever carries the pager next quarter will quietly appreciate.