{"id":1316,"date":"2026-11-08T23:40:00","date_gmt":"2026-11-08T23:40:00","guid":{"rendered":"https:\/\/cotocus.cn\/blog\/?p=1316"},"modified":"2026-11-08T23:40:00","modified_gmt":"2026-11-08T23:40:00","slug":"incident-runbook-diagnosis","status":"publish","type":"post","link":"https:\/\/cotocus.cn\/blog\/incident-runbook-diagnosis\/","title":{"rendered":"The 3 a.m. Problem: How an Incident Runbook Frees Your Team"},"content":{"rendered":"<p>The page came in at ten past three on a Sunday morning. The checkout on a client&#8217;s web application had stopped responding, orders were failing, and the on-call engineer, three weeks into the job, was staring at a dashboard full of red. The only person who knew how to bring the service back, a senior engineer named Wei, was on a flight with no connectivity. Nothing about the failure was exotic. A queue had backed up and a worker process needed restarting in the right order. But the knowledge lived in Wei&#8217;s head, and Wei was at thirty-five thousand feet.<\/p>\n<p>Every operations team has a version of this story. What follows is a diagnosis of why it happens and a practical cure: the incident runbook, written so that the next person on call can fix the problem without waiting for a hero.<\/p>\n<h2>The symptom: a team that depends on one person<\/h2>\n<p>The signs are easy to spot once you look. Questions in the team chat always go to the same name. Handover notes say &#8220;ask Wei&#8221;. Certain tasks, like restarting a worker pool, rotating a credential or draining a queue, are never done by anyone else. Outages last longer when that person is away, and everyone quietly plans holidays around theirs.<\/p>\n<p>It is tempting to call this a staffing problem. Usually it is a documentation problem. The skill is not rare, but the sequence of steps was never captured, so it cannot be taught except by apprenticeship.<\/p>\n<h2>The root cause: knowledge that was never extracted<\/h2>\n<p>Experienced engineers work from a mental map built over years: which service depends on which, what the logs look like before a failure, which fix is safe and which only makes things worse. They rarely write it down for three reasons. It feels obvious to them. There is always something more urgent. And the first time a fix works, relief overtakes the instinct to document.<\/p>\n<p>The result is that the team&#8217;s real operating procedure is a collection of habits, and habits do not survive absence. If you are already thinking about the structural side of reliability, this discussion of <a href=\"https:\/\/cotocus.cn\/blog\/managing-massive-web-traffic-through-modern-cloud-infrastructure\/\">keeping busy web services standing under heavy load<\/a> shows why the infrastructure itself must be understood by more than one person.<\/p>\n<h2>The cure: an incident runbook<\/h2>\n<p>An incident runbook is a short, ordered procedure for a failure you have seen before. It is written for the person who is awake, tired and slightly frightened, not for the expert. Each entry covers one scenario, such as &#8220;checkout failing because the worker queue is stuck&#8221;, and follows a fixed shape.<\/p>\n<ol>\n<li><strong>How you will know.<\/strong> The alert or symptom that means this scenario applies.<\/li>\n<li><strong>Impact in one line.<\/strong> Who is affected and how badly, so the responder can judge urgency.<\/li>\n<li><strong>First checks.<\/strong> Two or three quick looks that confirm the diagnosis.<\/li>\n<li><strong>The fix.<\/strong> Numbered actions in order, each with the result to expect.<\/li>\n<li><strong>If the fix fails.<\/strong> The next option and the point at which to escalate.<\/li>\n<li><strong>Afterwards.<\/strong> What to record and who to tell.<\/li>\n<\/ol>\n<h3>What Wei&#8217;s runbook looked like<\/h3>\n<p>When Wei landed, the first thing the team asked was not &#8220;what happened&#8221; but &#8220;what did you actually do?&#8221; Over a coffee he talked it through while a colleague typed. The entry ended up under twenty lines. It named the two dashboards to check first, the exact order for restarting workers so the queue drained rather than flooded, and the one command that looks harmless but should never be run while orders are in flight. The part everyone found most valuable was the note about what <em>not<\/em> to do, which no one had ever heard said aloud.<\/p>\n<p>Three weeks later the same queue stalled during a quiet Tuesday night. The new engineer followed the page, restored service in about twelve minutes, and added a line about a delay she noticed at step four. Nobody was woken, and no one waited for a flight to land.<\/p>\n<h2>Deploys deserve the same treatment<\/h2>\n<p>Incidents are not the only recurring work. Deploying a release is another task that tends to sit in one person&#8217;s hands, and it carries its own risk because it happens on a schedule rather than in a crisis. A deploy runbook looks much like an incident one, but its emphasis shifts to preparation: confirm what is being released, confirm the rollback point, pick a quiet window, announce the change, and agree in advance what result counts as success.<\/p>\n<p>The rollback section deserves special care. Most teams can describe going forward; fewer have rehearsed going back. Write the rollback as its own page, test it in a safe environment, and keep it where the deployer can reach it without logging in to anything that might be broken.<\/p>\n<h2>Writing the first five runbooks<\/h2>\n<p>You do not need a perfect library. Start with five, chosen by pain:<\/p>\n<ul>\n<li>The failure that has happened most often in the last six months.<\/li>\n<li>The failure that caused the longest outage.<\/li>\n<li>The routine deploy that only one person is trusted to do.<\/li>\n<li>The rollback you hope never to need.<\/li>\n<li>The restore-from-backup procedure that has never been tested.<\/li>\n<\/ul>\n<p>For each, sit with whoever knows it best and write while they work. Then ask someone who does not know it to follow the page on a safe environment. Their stumbles are your edits. Performance problems are a good source of material too, and a piece on <a href=\"https:\/\/cotocus.cn\/blog\/optimizing-website-performance-through-modern-architecture\/\">tuning a site&#8217;s speed through better architecture<\/a> is a reminder of how many recurring checks live behind a fast site.<\/p>\n<h2>Common reasons runbooks fail<\/h2>\n<p>Diagnosis is worth doing on the cure itself, because many teams write runbooks and still end up dependent on a hero. These are the usual culprits.<\/p>\n<ul>\n<li><strong>Too long.<\/strong> A twenty-page document cannot be used at three in the morning. One page per scenario is the target.<\/li>\n<li><strong>Stored somewhere fragile.<\/strong> If the runbook lives on the system that is down, it is useless. Keep a copy elsewhere.<\/li>\n<li><strong>Never tested.<\/strong> Untested steps rot. Run each one occasionally in a safe environment.<\/li>\n<li><strong>No owner.<\/strong> Someone must be named responsible for reviewing it after each incident.<\/li>\n<li><strong>Written to impress.<\/strong> Plain beats clever. Say which button, which command, which folder.<\/li>\n<\/ul>\n<p>One more habit is worth building into drills: have the person who wrote an entry stay silent while a colleague follows it. The writer will be tempted to jump in and explain, and every explanation they give aloud is a sentence that should have been on the page.<\/p>\n<h2>Where to read about the data side<\/h2>\n<p>Teams that run data pipelines meet the same hazard in a nightly job only one engineer can recover. If your services move and transform data on a schedule, <a href=\"https:\/\/www.thedataops.org\/\">TheDataOps.org<\/a> publishes DataOps writing for data engineers that pairs well with the pages described here.<\/p>\n<h2>Making the habit stick<\/h2>\n<p>The single most effective change is cultural: make &#8220;write it down&#8221; part of the definition of done for every incident. When a problem is fixed, the incident is not closed until the steps are either added to an existing runbook or recorded as a new one. Review them briefly at the end of each month, retire anything outdated, and rotate who runs the drills so no one becomes the new bottleneck.<\/p>\n<p>Wei now treats his flights as a small test. Before any long trip he asks a colleague to pick a runbook and run it without him. If something needs explaining, that is a gap in the page, and he fixes it before boarding. That is what a healthy team looks like: not one where nothing breaks, but one where the fix belongs to the whole team and not to a single person who happens to be reachable.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When one engineer holds the fix in their head, every outage waits on their flight. A diagnosis of the problem and the short written cure that ends it.<\/p>\n","protected":false},"author":4,"featured_media":1317,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[330],"tags":[],"class_list":["post-1316","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/posts\/1316","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/comments?post=1316"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/posts\/1316\/revisions"}],"predecessor-version":[{"id":1386,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/posts\/1316\/revisions\/1386"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/media\/1317"}],"wp:attachment":[{"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/media?parent=1316"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/categories?post=1316"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.cn\/blog\/wp-json\/wp\/v2\/tags?post=1316"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}