This story is invented for illustration, and any resemblance to a real team is coincidence. It follows a small group, five engineers who maintain build pipelines for a few web clients, through the week their builds started failing at random. The lesson sits squarely in ContainerOps, the Ops Atlas’s name for caring about individual containers, images and registries below the level of any cluster manager. Grab a tea and follow the team through the week.
Tuesday: the red builds that fixed themselves
At about ten in the morning the first message appeared in the team chat: a build had failed while pulling a base image, and a rerun passed. By noon four more builds had failed the same way, and by mid-afternoon the whole queue felt unreliable. The errors were short and unhelpful, a “too many requests” status from the public registry where most of their base images lived.
One of the engineers had recently read the Ops Atlas page on ContainerOps and remembered a scenario in it. The Atlas sketches a CI setup with thirty runners that between them make 180 anonymous image pulls an hour, against an anonymous allowance of 100 pulls every six hours. The arithmetic does not work: the budget is gone in minutes and every later pull is refused with a 429. The scenario is flagged as illustrative, but the team’s symptoms matched it uncomfortably well.
Wednesday: counting the pulls
The team’s first task was to confirm the theory. They had twenty-two runners, and each job started by pulling three images. A quick count from the pipeline logs showed well over a hundred pulls an hour at peak, nearly all anonymous because the runners had never been given registry credentials. The CI system had worked for years on a quieter schedule, and nobody had been watching the allowance.
The Atlas lists the published numbers: anonymous pulls capped at 100 per six hours, logged-in personal accounts at 200 in that span, and paying customers described as unlimited within fair use. Moving to a logged-in personal account would double the allowance but still would not cover a busy pipeline. The team decided that was a patch, not a fix.
Thursday: two changes that matter
The senior engineer sketched a plan on the whiteboard in two parts. First, every runner should authenticate, so that failures stop depending on shared anonymous quotas. Second, builds should pull from a cache the team controls. The Atlas describes the pattern as a pull-through cache or mirror, and the scenario’s fix pairs authentication with a pull-through cache in a cloud provider’s registry. Once an image has been fetched once, later pulls are served from the cache and never touch the public allowance.
The team also noted the Atlas’s warning that a cache is no backup. If the upstream image vanishes, the cache may not hold every version they need. They added a note to keep copies of the handful of base images that every project depended on, in a registry of their own.
Planning the change felt familiar. The group had a habit, taught in the client work described in a deep engineering blueprint for web project requirements, of writing down requirements before touching anything. They wrote three: builds must not fail because of an external allowance, every image used in production must be traceable to a digest, and the registry must not grow without bound.
Friday: pinning, and the fear of staleness
While editing build files the team noticed that most of them used a floating tag as the base. That meant two builds an hour apart could produce different images from identical source. The Atlas lists floating tags in production among its anti-patterns and recommends deploying by digest with release tags kept immutable.
One engineer worried aloud that pinning would freeze them on old software. The answer, again from the Atlas, was that digest pinning only stops fixes when nothing is updating the pin. They switched on an update bot whose pull requests announce each fresh base image, so updates became visible, reviewable events, instead of hidden surprises in the middle of a build.
The following Monday: housekeeping
With the cache in place, the team looked at the registry’s size. Every build pushed images for two platforms, and nobody had ever deleted anything. They borrowed the Atlas’s way of estimating growth: services times weekly builds times platforms times image size. Their own numbers were smaller than the Atlas’s illustration of forty services, but the principle was the same, and the total was heading for a figure the finance contact would not enjoy seeing on an invoice. They agreed a retention policy of keeping tagged releases and recent builds and deleting the rest on a schedule, with garbage collection to reclaim the space.
The argument about doing it quietly
Not everyone agreed on how to roll the change out. The most cautious engineer wanted to switch every project at once over a weekend, so that no one would have to think about two modes. The most impatient wanted to change nothing until the following quarter. The compromise was a pilot: one low-traffic client project would move to authenticated, cached pulls first, with the old path left in place as a fallback for a fortnight. If the cache misbehaved, a single environment variable would send builds back to the public registry.
The pilot also produced a useful surprise. Build times on the pilot project fell noticeably, because the cache sat close to the runners and large layers no longer crossed the public internet on every run. The team had expected reliability to improve; speed was a bonus that made the rollout an easier sell to the rest of the staff.
One last detail from the pilot is worth recording. Two jobs failed on the first morning because their credentials had been stored with the wrong scope, and the error message looked just like the original rate-limit complaint. The team added a one-line check to the pipeline that logs in before pulling and prints a clear message if login fails, so the two kinds of failure could never be confused again.
What they measured afterwards
The team chose three numbers to watch, all drawn from the Atlas’s list. Digest coverage, the share of production images referenced by fingerprint, started at about a third and was raised towards the Atlas’s target of complete coverage. Base-image age was tracked so that old foundations stood out. And the delay between a fix becoming available and its arrival in production was recorded, so the team could show a client how quickly a security update would flow through.
They also prepared a simple answer to a question clients had begun asking in design reviews, much like the checklist in a guide to what you should ask before commissioning a bespoke web app: who will operate the application after launch, and what will they inherit? The honest reply included a short operations page listing the images, their owners and the update routine.
Lessons from the week
- Shared anonymous quotas are a hidden dependency. Authenticate runners and cache what you pull.
- A mirror makes builds faster and calmer, though it is not a backup.
- Floating tags make two builds from the same source differ. Pin by digest and let a bot bring fixes.
- Registries grow silently. Put retention on the calendar before the invoice does it for you.
- Choose a few measurements and look at them monthly, not only during an incident.
A note on what the story leaves out
Real incidents are messier than a tidy week. The Atlas flags its own scenario as a teaching device, and this story does the same. It leaves out topics the Atlas places in neighbouring disciplines, such as signing images and scanning them for flaws, or running them on a cluster. Those matter, but they rest on the foundation this team repaired: images that can be traced, fetched reliably and updated deliberately.

