Most runbooks are written by the person who least needs them, for a situation they understand well, and are read by someone tired, at night, who has never seen this system before.
That mismatch explains nearly every runbook failure. The author knows which environment to run the command in, so they do not say. They know the output is normally noisy, so they do not mention it. They know the third step fails harmlessly the first time. The reader knows none of this and has no way to tell a normal failure from a real one.
This post is how to write one that survives contact with the reader it was written for. For whoever owns an on-call rotation.
The reader you are writing for
Assume: they did not build this, they have not read this document before, it is the middle of the night, something is broken, and they are anxious about making it worse.
That last point drives the design. An anxious reader does not improvise. They do exactly what the document says and stop at the first thing that does not match. So every step needs an expected result and an instruction for when the result differs.
What a usable runbook contains
When to use this, in one line at the top. The reader arrived from an alert and needs to know within five seconds whether this is the right document. Name the symptom, not the cause: "API returning 502 from the edge", not "notification service OOM recovery".
What this will do, and what it will affect. Before any command, tell them the consequence. Restarting the service drops in-flight requests. Failing over takes four minutes during which writes fail. Someone about to run a step deserves to know its blast radius before they run it, not after.
Preconditions and access. What they need before starting: which VPN, which role, which credential, where the key lives. Discovering halfway through that you lack access, at 03:00, with no one awake to grant it, is the most common way a runbook stalls.
Exact commands, copy-pasteable, with the expected output.
# 1. Confirm the container is actually restarting, not booting slowly
docker inspect notification --format 'restarts={{.RestartCount}} status={{.State.Status}}'
# EXPECTED: restarts=0 and status=running -> this is a slow boot, wait
# IF restarts > 0 -> crash loop, go to step 4
# IF command errors -> you are on the wrong host, see Preconditions
That structure is the whole technique. Command, expected result, and a branch for each way it can differ. Without the expected output the reader cannot tell whether it worked.
Where to stop and escalate. Give explicit permission to stop. "If step 6 does not resolve it within ten minutes, wake [role] via [mechanism]. Do not continue past step 7 alone." An anxious engineer will keep trying things for an hour rather than wake someone, and that hour is usually more expensive than the call.
How to undo it. Every destructive step needs its reverse, written before the reader needs it.
When it is over. How to confirm recovery, what to communicate, and to whom.
What to leave out
Background and architecture. Link it. Nobody reads the system design during an incident, and it pushes the first command below the fold.
Alternatives and judgement calls. "You could also try X" invites deliberation from someone with no basis to choose. Pick one path. If the judgement genuinely matters, that is an escalation point, not a menu.
Anything you have not verified this quarter. A stale command in a runbook is worse than an absent one, because it gets run.
Test it against someone who did not write it
This is the part that is always skipped and it is the only real test.
Take someone unfamiliar with the system, give them the runbook and a non-production environment, and have them execute it while you watch in silence. Every question they ask is a defect. Every pause is a missing expected-output line. Do not help them; write down where they got stuck.
Half an hour, once a quarter, and it finds more than any amount of careful authoring, because the author cannot see their own assumptions.
Store it where the outage is not
A runbook in the wiki that is down, or in the cloud console you cannot reach, or behind the SSO that is the thing broken, is not available when it matters.
Keep the critical ones in the repository, so they are on every engineer's laptop by default. For the true disaster cases, offline: printed, or in a password manager, or in a different provider entirely. The test is simple and worth actually running: if your primary systems were unreachable right now, could the on-call engineer read this document.
Keep them from rotting
Update after every use. The moment immediately after an incident is the only time anyone has both the motivation and the context. Make it part of the incident close-out rather than a follow-up ticket.
Date them, visibly, at the top. A reader deciding whether to trust a step needs to know if it was written two years ago.
Delete the ones you no longer use. A directory of forty runbooks where six are current is worse than six runbooks, because the reader cannot tell which is which.
Track how often each is used. A runbook for a scenario that happens monthly should be automated. One that has never been used may be for a scenario that no longer exists.
The concession
The honest position is that a runbook is a workaround. Any procedure written down in enough detail to be followed mechanically is, by definition, a candidate for automation, and a script that does the same thing is faster, more reliable, and does not get stale silently.
So the runbooks worth investing in are the ones requiring judgement: deciding whether to fail over, deciding whether to declare an incident, deciding whether the data is safe to restore. Write those carefully. For the mechanical ones, write the script, and let the runbook be one line that says which script to run and what to check afterwards.
The implication
The measure of a runbook is not whether it is correct. It is whether a tired stranger can follow it without asking anyone, and there is exactly one way to find out.
Hand it to someone who has never seen the system, watch them try, and fix everything they stumble on. That is the whole practice.
