How to Write a Postmortem After an Outage in Real Scripts

An outage is expensive, and the postmortem is how the cost buys something. The aim is not to find who pressed the wrong key but to find why the system let a wrong key cause an outage. Written well, it becomes a runbook and a list of fixes. This post gives a template and the evidence-gathering steps, using shell tools for the timeline. It leads into runbooks.

Gather the evidence first

journalctl --since "2026-11-03 09:00" --until "2026-11-03 10:30" -o short-iso --no-pager > incident-journal.txt
git log --since="2026-11-03 08:00" --until="2026-11-03 10:30" --format='%ad %h %an %s' --date=iso
grep -E ' (5[0-9]{2}) ' /var/log/nginx/access.log | awk '{print substr($4,2,17)}' | sort | uniq -c | head

Pull the journal, the deploy and commit history, the error-rate curve and the chat transcript for the window. Convert everything to one time zone, preferably UTC. The third command counts 5xx responses per minute by time bucket, which pins the start and end of customer impact better than memory. Do this before the discussion, since people's recollection drifts quickly, and use git log filters for the change history.

The template

# Checkout unavailable for 47 minutes

## Summary
A config change removed the database pool limit, exhausting connections. Checkout failed from 09:12 to 09:59 UTC.

## Impact
About 1,900 failed checkouts. No data loss. Support received 41 tickets.

## Timeline (UTC)
- 09:05 Deploy 8f2d1c4 reaches production
- 09:12 Error rate alert fires
- 09:20 On-call restarts app servers, no change
- 09:41 Database connection count found at max
- 09:52 Config reverted
- 09:59 Error rate back to baseline

## Causes
Trigger: pool setting removed during a cleanup.
Contributing: no staging load test; alert pointed at symptoms only; no runbook for connection exhaustion.

## What went well
Rollback took 7 minutes once the cause was clear.

## What went badly
It took 29 minutes to look at the database; the first guess was the application.

## Action items
- Add a connection count alert (owner: Ada, due Nov 10)
- Write the runbook for pool exhaustion (owner: Grace, due Nov 14)
- Add a config check to CI for the pool limit (owner: Ada, due Nov 21)

The summary is two sentences a stranger could read. Impact has numbers. The timeline holds facts with timestamps, not judgments. "Contributing" lists what made the failure possible or longer, which is where most fixes live.

Keep it blameless

Write "the deploy removed the setting", not "Sam removed the setting". Ask "what made this easy to do and hard to notice?" The answers are code review gaps, missing tests, unclear alerts. People who expect blame hide details, and details are the data. A single "root cause" is often a comfortable simplification; the useful list is longer. Avoid "human error" as a conclusion, because it stops the inquiry exactly where the learning begins.

Close the loop

Review the action items weekly until done. Link the postmortem from the new runbook and alert. If nothing in the document changes how the team works, it was theater, and the way to find out is to check the list next month.

Where this does not apply

Not every blip needs a document. Set a threshold, such as customer-visible impact over 15 minutes, data loss or any security event, and write a short note for the rest. Security incidents follow their own process, and their reports may need restricted circulation. See responding to a compromised server.

What the document is for

A postmortem has one job: to make the next incident less likely or less damaging. It is not a report to management, not a ruling on who was at fault and not a ritual to be completed and filed. The useful output is a clear account of what happened, why the system allowed it, and a short list of changes with owners and dates. It also serves as a shared memory. Six months later, when something similar happens, the document tells the new on-call person what was tried, what worked and what did not. For that reason it should be written in plain language, with enough detail that someone who was not there can follow, and should be easy to find. Keep the set in one place, tagged by system and by cause, so patterns show up over time.

Not every incident needs one. A threshold such as customer impact, data loss, an incident that needed more than one person or a response that took longer than a set time keeps the process from becoming a burden.

Building the timeline from evidence

Write the timeline from logs, alerts, chat transcripts and commit history, not from memory, since memory of a stressful night is unreliable and tends to rearrange events into a tidier story than the real one. Record times in one time zone, UTC, and note the source of each entry. Include what people believed at the time, not only what turned out to be true: the theory that the database was at fault and the time spent pursuing it are part of the learning. Mark the points that mattered: when the problem started, when it was detected, when someone responded, when it was mitigated and when it was resolved. The gaps between those points are the measures to improve, since a long gap between start and detection means monitoring is the weakness, and a long gap between detection and mitigation means the response is.

Keep metrics to a few that you can compare between incidents: time to detect, time to mitigate, time to resolve and the user impact in minutes or requests failed.

Finding causes without a villain

Ask why in several steps, and stop when the answer becomes a change you can make. "The deploy broke the site" is a start. Why did the deploy reach production? Why did the checks not catch it? Why did it take twenty minutes to notice? Each answer reveals a gap in the system and not in a person. People make mistakes in every team, and a system that depends on no one ever making one will fail. Phrase findings about systems and processes, and describe actions in terms of what was reasonable given what the person knew at the time. A document that assigns blame teaches people to hide problems, and the next incident takes longer to understand because details are withheld. Aim for contributing factors in the plural, because incidents nearly always have several.

Finish with action items that are specific, owned and dated, and track them to completion. An item such as "improve monitoring" will not happen. "Add an alert when queue depth exceeds 1000 for 5 minutes, owner named, due next Friday" will.

Running the meeting and sharing the result

Hold the review within a few days, while memory is fresh and before the urgency fades, with the people who responded and someone who can make changes. Circulate the draft timeline beforehand so the time goes to discussion. A facilitator who was not involved keeps the conversation on causes and prevents it from turning to blame. Agree on the actions in the meeting, assign owners and dates, and record them in the same tracker the team uses for other work, so they compete honestly with other priorities. Publish the document where the whole engineering team can read it, and where relevant, summarize the cause and the fix for customers in plain language without hiding what happened.

Revisit the document after a month and check which actions were done. A postmortem whose actions were never completed has taught the team that the process is theatre.

See the Google SRE book on postmortem culture and Atlassian's incident handbook.