Why Patch Coverage Stalls at 90 Percent and How to Unstick It

Why Patch Coverage Stalls at 90 Percent and How to Unstick It

One pattern appears in nearly every patch programme at some stage. The rollout starts well. Within a week of a release, eighty percent of machines carry the fix, then eighty-five, then ninety. After that the line goes flat. Weeks pass, the report still says ninety, and the team starts to treat the remainder as unavoidable noise. This article diagnoses why coverage stalls and sets out a practical remedy. It draws on the Ops Atlas description of PatchOps, which uses the term informally for enterprise patch management, and on the standards it cites.

The symptom, stated precisely

The Atlas names the stall directly: coverage that stops in the mid-eighties to around ninety percent, with offline or silent agents as the typical cause. The stall matters because the last ten percent is rarely random. It tends to contain the machines that are hardest to reach, least understood and most likely to hold something sensitive. Laptops on long leave, servers in unusual network zones, lab equipment, and anything whose management agent has died quietly all collect there.

Before you chase causes, confirm the symptom is real by checking how the number is built. That brings us to the first diagnosis.

Diagnosis one: the denominator is shrinking

Many dashboards compute coverage as patched machines divided by machines that currently report. If an agent stops reporting, the machine drops out of the division and the percentage improves. The Atlas defines a more honest calculation: on-time effective coverage takes the machines that are fully active at the deadline, divides them by every asset that was supposed to be covered, and credits only hosts that have restarted and are still talking to the server.

The lab example in the source makes the gap concrete. Fifteen patchable assets produced a dashboard figure of one hundred percent, but a proper count gave about fifty-three. The dashboard had counted an expired exception as acceptable and had ignored an agent that had gone stale. If your stalled figure is ninety percent on the reporting denominator, the true number may be materially lower.

Remedy: publish both numbers for one cycle, the reporting-based figure and the all-in-scope figure, and let the gap speak for itself.

Diagnosis two: installed is being mistaken for effective

A fix sitting on disk does nothing until the affected process is restarted. Tools that look only at package state will mark a machine finished at installation, while the old code keeps running. The Atlas treats that waiting-for-restart state as the leading source of false compliance, and notes that some platforms eventually force a restart after a deadline plus a grace period, preceded by a short warning, whereas others reboot regardless of need or restart services automatically.

The behaviour matters twice. If you let the forced restart happen at a bad moment, users lose work and trust. If you never let it happen, the machine stays exposed. The sensible approach is to track a metric the source calls pending-restart age, and to schedule restarts inside defined windows rather than whenever a tool default decides.

Remedy: count a machine as covered only after the restart, and report the age of the oldest waiting machine every week.

Diagnosis three: the silent agents

An agent that cannot reach its management server is invisible. The Atlas lists the unreporting rate as a metric of its own, which is a helpful reminder that silence deserves a number. The usual causes are mundane: expired certificates, changed proxy settings, firewall rules, a disk that filled up, or a machine that was decommissioned without anyone updating the list.

Teams that run customer-facing systems should be particularly careful here. If you have read our piece on why web application security protects revenue and trust, you will recognise the stakes: the exposed machine is rarely the one you were watching.

Remedy: set a threshold for how long a machine may go without reporting before it triggers a ticket, and give each ticket an owner and a due date.

Diagnosis four: exceptions that never expire

Some of the stalled ten percent are there on purpose. A vendor appliance cannot take the update. An application breaks under the new library. Someone approved a deferral last spring and nobody revisited it. The Atlas points to NIST guidance recommending that such assets be handled in their own maintenance group, with mitigations such as isolation, and it warns against exceptions without expiry dates.

Remedy: create an exception register where every row has a reason, a mitigation and an end date. Report its health: how many rows exist, how many are overdue, how many have expired.

Diagnosis five: the update that stopped installing

This one is subtler. On certain server platforms, once a cumulative update has been frozen or superseded in the management tool, later machines can report as compliant because no further update is offered, even though the intended fix never landed. The Atlas calls it a supersedence trap. It is easy to miss, because every indicator looks green.

Remedy: verify against a functional probe or an installed-version check on the machine itself, instead of trusting the management console’s compliance label alone.

Diagnosis six: rings that never finish

Staged rollout is wise, but it can mask a stall. If the broad ring has no deadline, it will quietly drift. The source outlines rings from canary through pilot, broad and sensitive, and the broad ring runs to a stated deadline. A ring without a date is a suggestion.

The sensitive ring, handled node by node, is usually where the last few percent live, so give it a named captain and a schedule rather than leaving it to goodwill. If you are designing a portal or service where such sensitive nodes exist, our guide on building a secure customer portal explains why they deserve extra care.

A short diagnostic checklist

  1. Recalculate coverage with all in-scope assets in the denominator.
  2. List machines waiting for a restart and sort by age.
  3. List machines that have not reported within your threshold.
  4. Review every exception for a reason, mitigation and end date.
  5. Check two or three machines by hand for the true installed state.
  6. Confirm each ring has a deadline and an owner.

Do this once and the stalled ten percent will usually sort itself into four or five buckets, each with a different fix and a different owner. Resist the urge to treat the bucket sizes as a verdict on the team. A large silent-agent bucket usually points at neglected plumbing, while a large exception bucket points at old applications and unmade decisions. Both are fixable, and both are easier to fix once they are visible on a single page.

What to measure from here

The Atlas suggests a small family of measures: time to effective patch (both the median and a high percentile such as the ninetieth), the unreporting rate, pending-restart age, ring failure rate, patch-caused incidents, emergency lead time to reach ninety-five percent active, exception health and automation coverage. Nobody needs the full set at the start. Start with coverage on the honest denominator, the unreporting rate and the pending-restart age. Those three attack the stall directly.

A final practical tip concerns reporting rhythm. Present the stalled bucket every week in the same format, with the same columns and the same owners listed by name. Repetition makes drift obvious, and it turns an awkward annual confession into a routine item that nobody needs to be defensive about. Over a few cycles the buckets shrink, and the conversation shifts from blame to throughput.

Closing the loop

When the numbers start to move, resist the temptation to declare victory. Run a rehearsed emergency drill and see how long it takes to reach ninety-five percent. That figure is a better health check than any monthly report. For the fuller background on these definitions and the standards behind them, consult the Atlas’s overview of PatchOps.

The stall at ninety percent is not a mystery. It is a pile of small, well-understood causes, and each one yields to counting honestly and assigning an owner. Fix the denominator first, then the restarts, then the silence, and the line will start moving again.