When a web service slows to a crawl, the loudest voice in the room often wins. Someone says the database is the problem, another blames a recent release, and within minutes a fix is under way for a cause nobody has confirmed. This plain-language explainer describes a calmer approach used by careful engineering teams: during an incident, separate confirmed facts from hypotheses, and test the hypothesis before acting on it.
Three kinds of statements
Every sentence spoken during an incident falls into one of three categories. A fact is something an instrument or a log has shown, for example that response times rose at a stated hour or that error counts doubled on one endpoint. A hypothesis is a proposed explanation that fits the facts so far but has not been tested. A guess is an idea with little behind it, perhaps a memory of a similar outage last year.
The categories are not insults. Hypotheses and guesses drive investigation. The problem comes only when they are spoken in the same tone as facts, because listeners then treat all of them as equally settled.
A worked example
Imagine an online shop whose checkout page starts timing out. The team gathers what is known: timeouts began shortly after midday, they affect checkout but not browsing, and server memory looks normal. Those are facts, each with a source. One engineer suggests that a payment provider is responding slowly. Another recalls a configuration change that morning. Both are hypotheses.
The team writes both down and asks a precise question for each: what observation would count against it? If the payment provider were slow, calls to it should show long durations in the application logs. If the configuration change were responsible, reverting it in a test environment should restore normal behaviour. The cheapest and safest of these tests runs first. Only after a hypothesis survives a test does anyone change production.
Why testing before acting matters
Acting on an untested guess has two costs. It may fail to fix the problem, wasting minutes when users are affected. Worse, it may add a second fault, making the situation harder to read. A rollback that was never needed or a restart that wipes useful diagnostic state can bury the clues. Testing first keeps the evidence intact.
Planning work early in a project follows the same logic. Articles on writing detailed engineering requirements before a website build stress writing down needs in detail, and that discipline helps later because you can see which design decisions rest on confirmed requirements and which on assumptions about future demand.
Capacity is a common source of assumption
Teams frequently assume a system can handle a surge because it handled the last one. That is an assumption about load, not a measurement of limits. A guide to coping with huge traffic surges using cloud infrastructure describes the architectural side; the sorting habit adds a question to ask of any such plan: which numbers came from a load test, and which from optimism?
A simple template for incident notes
You can adopt this with no new tooling. Keep a shared document with three headings and fill it as the incident proceeds:
- Confirmed: observation, source and time.
- Working hypotheses: the idea, the test that would challenge it, and who is running it.
- Guesses parked: ideas worth revisiting if the main hypotheses fail.
When a hypothesis passes its test, it moves up to confirmed with a note on how it was shown. When it fails, strike it through but leave it visible, so nobody proposes it again an hour later.
Common mistakes to avoid
The first is promoting a hypothesis because several people agree. Agreement is not an experiment. The second is testing in a way that cannot fail, such as checking only the logs that would support your idea. The third is forgetting to record negative results, which wastes the next responder’s time. The fourth is treating a quick workaround as an explanation: if a restart clears the symptom, you know that restarting helps, not why the fault occurred.
Where to read further
Engineers who want structured reading around these habits sometimes turn to the DevOpsSchool learning site, which says it offers self-paced courses with hands-on labs for people working in DevOps, DevSecOps and related areas. Whatever material you choose, apply the same filter: note which claims you have seen demonstrated and which you are taking on trust.
Bringing it back to everyday decisions
The pattern is not limited to servers. Before signing a vendor contract, separate what the written terms confirm from what the salesperson implied. Before a migration, separate what the test run proved from what the team believes about production. The three-way sort works because it is cheap and because it forces a pause at exactly the moment when confidence is highest and evidence is thinnest.
Why the vocabulary matters
Teams sometimes resist this practice because it sounds like extra ceremony. In reality it is a small change in vocabulary: say “observed” when you looked, “suspected” when you are reasoning, and “unknown” when you have nothing yet. These words cost nothing and carry a great deal of information for the listener. A responder who joins an incident halfway can read the notes and immediately see which statements to trust and which to test, without interrupting those already at work.
The practice also improves the written review afterwards. Because each statement was labelled at the time it was made, the timeline is honest about what people knew when they decided, which supports a fair and useful discussion of how to improve rather than a search for someone to blame.
Teams that practise this tend to write shorter, clearer incident reports and to resolve problems with fewer false starts. It is a small change in vocabulary that produces a visible change in outcomes, and it can begin with the next outage, or even the next planning meeting.

