The AutomationOps Audit: Walk Your Estate in a Day

The AutomationOps Audit: Walk Your Estate in a Day

I keep a ritual with every new consulting client: before anyone shows me a dashboard, I ask for their crontabs. Not the architecture diagrams, not the runbooks — the raw schedules. The list is always longer than anyone guessed, and somewhere in it sits a job whose owner has left, whose purpose has faded, and whose permissions would make a security auditor reach for the coffee. Walking a client through that list is the most human part of my week: relief, laughter, a little guilt, and then the good work. This article turns that walk into an AutomationOps audit you can run yourself in a day, checklist in hand, no consultant required.

The AutomationOps Audit, Hour by Hour

Eight focused hours, one honest inventory, zero new tooling. Each hour below ends with its own checklist; tick every box and your estate is measurably safer by dinner. Where a box stays unticked, you have found next quarter’s work — which is the point of the exercise.

Hour One: Find What Actually Runs

Collect every scheduled and triggered job across the estate — cron tables, CI triggers, monitoring hooks, event listeners — into one list with three columns: what it does, who owns it, and what it changes. Expect surprises. In 2021 Facebook lost its backbone for hours because a routine change slipped past a faulty audit tool; the gap between “what we believed was protected” and “what was actually running” is precisely what hour one exists to expose. Do not judge anything yet. Counting comes first, and the simple act of writing owner names into empty cells has a way of assigning them by Friday.

  • Every job listed with a named owner — or flagged unclaimed.
  • Purpose stated in one sentence anyone can understand.
  • Targets, credentials and reach written down, not guessed.
  • Jobs nobody can explain moved to a quarantine list, paused by default.

Hour Two: Grade the Danger Honestly

Now sort. Read-only jobs get a tick and are ignored for the rest of the day. Single-target, reversible changes get a lighter pass. Multi-target reversible changes get the full treatment. And the fourth class — anything irreversible, anything that destroys data — goes to the top of the pile, because those jobs need a second human’s approval before firing, soft deletion with a recovery path, and a staged rollout. The grading instinct is old and respectable: NIST’s paired-approval rules — AC-3(2) with its companion CM-5(4) — rest on exactly that instinct for high-impact actions. While grading, ask the consulting question that saves the most money: can the underlying problem be fixed at the source so this job becomes unnecessary? A retry loop around a flaky dependency is a tax paid on every run; fixing the dependency retires the loop forever. The best automation is the one you get to delete, and grading hour is where those candidates usually surface.

Hours Three and Four: Test the Brakes While Nothing Is Burning

Brakes fail quietly, so this hour is about evidence, not intentions. For each dangerous job, locate its stop switch — per job, plus the estate-wide one — and confirm it works today, not in theory. The Atlas’s guidance is refreshingly specific: teams running risky tiers drill their switches twice a year, and anyone on call may pull any switch without asking first. Then check the failure budget: how many failures does the run tolerate before it halts itself? If the answer is “unlimited”, the job trusts luck, and luck has a queue.

  • Stop switch located, tested, and known to the whole rota.
  • Global stop tested in the last six months.
  • Failure budget declared per job; runs halt themselves at the limit.
  • Drill dates in the calendar for anything graded dangerous.

Hour Five: Rehearse, Then Bound the Reach

A rehearsal — the dry run — should show you exactly what a job would change while changing nothing, and every risky job deserves one before its next real run. Read the footnotes: Ansible’s check mode will wave command and shell probes straight through unless you tell it otherwise, so verify your green results were earned rather than assumed. Reach comes second: no job should sweep a whole fleet in one motion. Waves with a canary up front, batches sized so a sick batch stops the run — the serial setting and max_fail_percentage ceiling in Ansible turn this into a Tuesday afternoon’s work, not a project. Teams worried this slows them down usually find the opposite: automation cuts errors as well as hours, and it cuts the most errors exactly where you have bounded it.

Hour Six: Ask Whether a Repeat Is Safe

Run every job twice in staging. If the second run damages anything, the job is phrased as an action rather than a destination, and that is fixable: prefer “ensure the service is running” over “start the service”, “ensure the record is absent” over “delete”. Google’s repair automations famously looped every fifteen minutes because each pass converged on the same end state — extra laps were free, missed laps self-healed. For jobs that can never safely repeat (payments, notifications, account creation), require a caller-supplied request ID captured atomically at execution, which turns duplicate runs from a hazard into a mathematical impossibility. The person who triggers the run supplies the unique label; the system refuses any repeat of a label it has already seen. Vigilance is replaced with bookkeeping, and bookkeeping does not get tired at 3 a.m.

Hour Seven: Check Ownership, Codify the Rules

Ownership is the difference between an estate and a junk drawer. Every entry from hour one should now carry a tier, limits, an identity that runs it, and a review date — ideally as code, in plain YAML, versioned beside everything else your team versions. Guard tests belong in CI, so a job that loses its owner, its budget or its switch fails a build instead of failing production. Teams that tie this work to broader reliability efforts — guarding revenue through stronger web application security — find the reviews overlap pleasantly: the credentials audit and the automation audit are cousins.

Hour Eight: Score It and Book the Follow-Up

Close the day with numbers rather than vibes. Two figures matter most in the first quarter. First, ownership coverage: what share of the estate has a named, current owner? The Atlas’s ninety-day target of ninety percent or better is a fair bar. Second, time to stop: how long did today’s drill actually take, measured, from “something is wrong” to “everything is paused”? The remaining metrics — hours genuinely returned per month, rescue frequency, guard-trip counts — can wait for their dashboards. Infrastructure teams already fluent in carrying heavy traffic on modern cloud foundations will recognise the pattern: capacity planning and automation governance are both just promises you can measure.

Making AutomationOps a Quarterly Habit

One audit rarely finishes the job, and that is fine — the day exists to replace mystery with inventory, luck with limits, folklore with owners. Repeat it quarterly, retire what no longer earns its keep, and let the checklist grow where your estate grows. When you want the source discipline behind this walkthrough in full — controls, tiers, metrics, the maturity ladder — the Atlas keeps it on one page: AutomationOps, specified end to end at The Ops Atlas. It is the document I leave behind when an audit day ends, and the one I would leave with you.