Imagine a small software studio, a composite rather than a real client, that learns the hard way that “everyone will just jump in” is not an operating model. A release goes sideways on a Friday evening, five people open five terminals, two of them revert different changes at the same moment, and the customer hears nothing for ninety minutes. On Monday the team sits down and builds something better, step by step. This how-to follows that story and shows how to set up IncidentOps, the practice of responding to unplanned service disruptions in an organised way, from a standing start. It uses the Ops Atlas as a source and assumes a team of roughly five to twenty engineers.
Step 1: Write down what counts as an incident
The studio begins with a half-page. It says that a problem becomes an incident when customers can see it, when another team has to help, or when nobody has found a fix after about an hour. Those three triggers echo wording used by a large search company in its public guidance. The point of writing them is to remove hesitation. Nobody should have to wonder whether they are overreacting. If the criteria fit, you declare. A declared incident that proves minor costs a few minutes, and the Atlas contrasts that with late declarations that cost hours.
Step 2: Define two or three severity levels
Do not build a seven-tier matrix. Keep it to a couple of levels and attach behaviours to each. For the highest level the studio decided that the on-call person is paged immediately, a commander is assigned within minutes, and customer updates start at once. For the next level, the response happens during working hours with a shared channel. Reviewing the scale after each incident keeps it honest.
Step 3: Build the rota and keep it as code
Rotations that live in a spreadsheet drift. The studio stored schedules and escalation rules in version control and applied them with an infrastructure tool. One caution from the Atlas applies here: marking a variable as sensitive in a Terraform configuration only hides it from plan output. The state file and output commands can still reveal routing keys, so use encrypted remote state and limit who can read it. Also run a quick sum on your on-call fairness. A tiny team that staffs two slots around the clock all year soon blows through the one-quarter ceiling that is often quoted for on-call load.
Step 4: Put the status page somewhere else
The next step is easy to postpone and painful to skip. Choose a hosted status page whose availability is independent of the systems it describes. The studio picked a hosted service on its own domain, then tested it by switching off a staging copy of the main app and confirming the page still loaded. The Atlas calls this one of two core architecture rules. A well-known cloud outage in 2017 hit exactly this trap, when the vendor’s own dashboard relied on the storage service that had broken. If you are also trying to keep visitors happy during slowdowns, see how modern site architecture helps pages stay quick under load.
Step 5: Make role cards
Three hats are enough to begin with. The commander decides and coordinates but does not change production. The operations lead, supported by whoever has the right knowledge, is the only group touching the system. The communicator writes updates. Print a short card for each, with five or six bullets of duties. Then add a rule that anyone trained may wear any hat, since roles are not job titles. When a second incident overlaps, a different person simply picks up a different card.
Step 6: Agree on a communications rhythm
The studio adopted a simple timetable drawn from public playbooks. A first note goes out within about five minutes of the call opening. A second post that sets the scope follows shortly afterwards. From there, a fresh message lands about every twenty minutes while the opening two hours last, even when all it says is “still working, nothing new”. The closing post states whether any data was lost. Writing the template in advance means the communicator is filling blanks rather than composing from scratch.
Step 7: Split the conversation into two channels
This is a small change with a large effect. Use one channel for the people who are working the problem and another read-only channel where updates are posted. Without the split, curious onlookers fill the working channel and responders drift into private messages. The Atlas cites field research in which hundreds of participants forced the core team to retreat into private threads, which is exactly the situation you are trying to avoid.
Step 8: Practise before you need it
Invent a plausible failure and run it as a thirty-minute tabletop. The studio chose a failed database migration. The commander assigned tasks, the communicator drafted updates, and the group argued about what the first mitigation should be. They discovered that nobody knew how to roll back, and fixed it that week. A drill like this costs almost nothing and finds gaps that documents hide. If the exercise brings up weaknesses in how the app protects revenue and customer trust, the team’s reading list can include why strong application security protects both income and credibility.
Step 9: Review without blame
Before the call ends, name whoever will run the review. Pick a realistic date and cap the follow-up list at a few items. Every action gets a single owner plus a deadline, and you look for system conditions instead of culprits. The studio replaced the phrase “developer error” in its template with a prompt asking what allowed the mistake to reach customers. Over a few months the reviews got more candid, and the number of declared incidents went up, which the team treated as a sign of trust rather than decline.
Common stumbling blocks along the way
Teams that follow a plan like this tend to trip over the same three things. The first is over-engineering. People design elaborate severity matrices and approval chains before they have handled a single real incident, and the paperwork collapses under its own weight. Keep the first version embarrassingly small and expand it only when a real event shows what was missing.
The second is the commander who cannot resist fixing things. It takes a conscious effort, and sometimes a gentle reminder from the deputy or the communicator, to stay out of the terminal. If you notice it happening, swap roles for the next incident and let the person who loves debugging take the operations lead seat instead, where that energy is useful.
The third is forgetting the follow-through. A review that produces a tidy document and no changes has not done its job. Put the open actions on a visible board, look at them in a regular meeting, and refuse to close an item merely by downgrading it.
What the studio noticed after three months
Friday evenings stopped being dramatic. The commander seat rotated among six people. Customers started mentioning that the updates were useful. The review actions were mostly small, but they were finished. None of this required a new product or a large budget. It required a few written pages and the discipline to follow them.
If you want more background before you try this yourself, including the sources behind each recommendation and some of the disagreements between experts, read the IncidentOps article in the Ops Atlas and then pick the first two steps. Start small, practise often, and let the habits grow with the team.

