Same Failure, Third Time? A Friendly Diagnosis and a Real Fix

Same Failure, Third Time? A Friendly Diagnosis and a Real Fix

You have fixed the same failing deployment three times this month. Each fix took an hour, each one felt urgent, and each one was forgotten by the next stand-up. Sound familiar? This post is a friendly diagnosis of that situation. We will start with the problem, look at why teams keep patching instead of curing, and finish with a repeatable way to turn scattered incidents into one clear cause.

Imagine a chat with a colleague who has watched this exact story play out before. No heavy theory, just a practical route from “why does this keep happening?” to “that is why, and here is the change”.

The problem: incidents that look new but are not

Here is the typical picture. On Monday a build fails because a dependency could not be fetched. On Wednesday a test suite fails on a timeout. On Friday a release stalls waiting for an approval. Three tickets, three owners, three different-looking causes.

Put them in a list together and a different story may appear. All three happened during the shared window when the build agents were busiest. The tickets were separate, but the cause could be one: not enough capacity at a particular hour. Nobody saw it, because nobody looked across the tickets.

That is the core problem. Incidents arrive one at a time, and teams respond one at a time, so the connections between them are invisible by default.

Why teams keep patching instead of curing

There are several quite human reasons, and none of them mean your team is doing anything wrong.

  • Urgency crowds out reflection. When something is broken, the priority is to restore it, not to wonder about its siblings.
  • Different people see different incidents. Each engineer remembers the ones they handled, so nobody holds the full picture.
  • Tickets are closed, not studied. Once a ticket is resolved, its details tend to disappear into a tracker nobody reads.
  • Success feels like the end. A working system rarely prompts anyone to ask why it broke last time.

The result is a team that is excellent at recovering and surprisingly poor at preventing.

Diagnosis step one: collect the incidents in one place

The first move is boring and powerful. Pull the last month or two of incidents, failed builds and pages into a single list. One line each. Include the date, the system, the visible symptom and the fix that was applied.

You do not need a new tool. A shared document is fine. The point is to put the events side by side, because patterns are made of comparisons and you cannot compare things you cannot see together.

Diagnosis step two: group by shared traits, not by symptom

This is where beginners often slip. They group by what the problem looked like: “timeouts”, “failed builds”, “access errors”. Symptoms are a poor guide because the same cause can produce several different symptoms.

Group instead by traits that surround the event:

  • Time of day or day of week.
  • Which environment, service or team was involved.
  • What had changed in the previous day or two.
  • Who or what triggered it, such as a person, a schedule or an external dependency.

When three or more incidents share a trait, you have a candidate pattern. When the trait is something you control, such as a schedule or a configuration, you have a candidate fix.

Diagnosis step three: apply the rule of three

Do not leap on a pattern after two matching incidents. Two can be coincidence. Wait for a third, and ideally one that happened under slightly different conditions. If the trait still holds, your confidence goes up considerably.

If you are tempted to act sooner, apply a cheap temporary workaround and keep recording. Premature rebuilding is a classic way to spend a week fixing the wrong thing.

What a grouped list can look like

Here is a made-up fragment, just to show the shape. Imagine six lines in your list: a failed deploy on a Tuesday evening, a stalled job on a Thursday evening, a slow dashboard on another Tuesday evening, a timeout on a Monday morning, a failed deploy on a third Tuesday evening and a login error on a Wednesday afternoon.

Grouped by symptom, you get four unrelated families. Grouped by time, three of the six sit on Tuesday evenings. That alone does not tell you the cause, but it tells you where to look first: what runs on Tuesday evenings? A weekly batch job, a report, a backup? The grouping turned a vague worry into a specific question.

The three leftover lines are worth a glance too. They might belong to a second pattern, or they might be genuine one-offs. It is perfectly fine to leave them unexplained until more evidence arrives.

The solution: one cause, one change, one check

Once you have a believable shared cause, the solution is deliberately modest.

  1. State the cause as a sentence you can disprove. “Builds fail most often when more than a handful start in the same half hour.”
  2. Make one change aimed at it. Stagger schedules, add capacity at that hour, or cache the fetched dependency.
  3. Keep recording. Use the same list and the same traits.
  4. Check after a few weeks. If the cluster has thinned out, the cause was right. If not, return to the grouped list and test the next trait.

Altering a single variable per round is what keeps the outcome readable. Change five and you will never know which mattered.

A story from the field

Picture a small platform team that kept being paged about slow responses from a customer-facing service. Each time, someone restarted the service and it recovered. Over six weeks there were four such pages.

When the team finally listed them together, three happened shortly after a scheduled data export ran. The fourth happened after a manual export. The common element was not the service itself but the heavy query the export triggered. Moving the export to a quiet window, and later giving it its own read replica, ended the pages entirely.

Nobody was careless. The restart fix simply worked well enough to hide the cause. It took a list and a little patience to expose it.

Where scale makes patterns matter more

Patterns become even more important as systems grow. Spikes in demand, dependent services and shared infrastructure all create repeating pressures. If you work on services that face heavy bursts of visitors, our article on coping with enormous visitor surges using cloud capacity shows where the same strain tends to recur. Reading it with your incident list beside you can help you spot which failures are really capacity questions in disguise.

Repetitive human work is another rich source. Manual steps that people redo constantly are where slips keep recurring. For a wider view of how routine tasks produce routine errors, see this piece on how handing routine chores to software cuts mistakes and saves time.

Learning the habit with other engineers

If you want to practise this style of thinking, it helps to see how other engineers approach it. Engineers who want company can try DevOpsSchool.org, where, in its own words, an open crowd of builders shares study tracks, career maps, lab exercises, a lookup glossary and a tool index. Treat a place like that as a source for vocabulary and technique, then arrive with your own incidents, since real cases teach quicker than invented ones.

Common traps in diagnosis

  • Blaming people instead of conditions. A pattern almost always points at a condition, and a condition can be changed.
  • Chasing rare, dramatic incidents. A frequent, boring problem usually costs more than one spectacular outage.
  • Over-fitting the story. Three incidents can suggest a cause, but they cannot prove it. Hold your theory lightly.
  • Forgetting the follow-up. Without a check, you never learn whether the fix worked.

A weekly rhythm that makes it stick

Teams who succeed with this approach usually adopt a very light routine. Once a week, for fifteen minutes, someone reads the incident list aloud. The question is simply, “Has anything happened again?” Nothing needs to be solved in that meeting. The aim is to notice repeats early, and to add a name to the ones that look like candidates for a deeper look.

Over time the list becomes a shared memory. New team members read it and learn the system’s quirks in an afternoon. Old members stop re-diagnosing problems they already solved.

Quick recap

The problem is that incidents arrive separately and get fixed separately. The diagnosis is to gather them, group them by shared traits rather than symptoms, and wait for the third match. The solution is one cause, one change and one check. You could begin it tomorrow, and it turns an exhausting loop of repeat fixes into something you can actually close.