A SOC Pilot Diary: AIOps for Security in Practice

A SOC Pilot Diary: AIOps for Security in Practice

Meet the three-person security desk at “Meridian Ledger”, a composite mid-size payments firm assembled from patterns we see across consulting engagements. Like most small teams, they bought a triage assistant this year and wanted to know whether to trust it. What follows is their quarter with AIOps for Security, told as a diary — twelve weeks, one candidate model, and a pile of evidence forms. The names are invented; the maths they wrestled with is not.

Week one: a decision score before a purchase

Their first move surprised the vendor: no purchase order yet, just a scoring exercise. Each readiness question got answered 0 to 3, weights applied, out of a possible 27 points. Above 14 means shadow mode only; 9 to 14 buys a time-boxed pilot; below 9, the honest answer is to decline. Meridian’s total came in mid-band, so the pilot got a green light with two conditions attached — nothing automated, and the whole exercise in writing. The security lead put it plainly in the log: “We are scoring our own readiness, not the product.” Teams curious about the broader security-engineering context they worked within can start with Protecting Revenue and Trust Through Strong Web Application Security.

The scoring pass had a second benefit nobody predicted: it forced the team to say out loud which actions existed in their world at all. Closing a phishing ticket, yes. Disabling a service account, yes, but reversible. Wiping a device — nobody could name the last time it happened. The ladder they would later climb was already visible in their own answers, drawn in their own ink.

AIOps for Security, watched from the shadows

Weeks one through four ran the classic blind shadow pattern. They exported a quarter of historical dispositions, chose two alert categories they understood deeply, and had humans label a holdout set the model never saw. The candidate model scored the live queue the whole time — and changed nothing. Volume counts surprised them first: one category turned out far noisier than instinct suggested, which is exactly the base-rate discovery the shadow period exists to force. Their maturity self-assessment, done honestly at the start, sat at stage one of five — Ad hoc, Measured, Governed, Adversarial, Learning — with Measured the realistic quarter-end target. Analysts kept a shared notebook of every disagreement between the model’s read and theirs; by week three the notebook had more entries about labelling inconsistency than about model error, an uncomfortable and useful finding.

What the published trial taught them

Midway through, the team pinned a published blind study to the wall as their reference point. The study had graded an agent against 1,240 emails with labels withheld from the machine: 71 genuine threats, 2 of them missed — 2.8% on its face. The 95% ceiling computed from that sample, though, reached 8.6%, three times over the strict 2% bar required before tickets close themselves. Precision told a split story as well: brilliant at 99.8% on the closures, patchier at 86% whenever the agent called something a threat. And the prize dangled all the same — 93.5% of queue work removable — unclaimed, because the gate came first. The definitions behind every term in that study, from autonomy tiers to evidence gates, come from the AIOps for Security entry in the Ops Atlas, which the team kept open beside the diary. The log entry that week read: “Savings are real. Trust is not yet.”

Canaries before confidence

Weeks seven to ten built the safety net. Sampling closed tickets at random was rejected early — the arithmetic destroys it. Suppose they audited ninety-odd closures blind out of the 1,845 the quarter produced and every single one looked correct; even then the true miss rate could top out near 38%. A perfect record across n runs, the team learned, leaves room for roughly three failures per n once the statistics finish; and a single observed miss lifts the sample required for a 2% claim to 236, with a second miss moving it to 313. So they planted traps instead. Three rules went up on the wall, handwritten:

  • Fifty fake items per quarter, mixed into the real queue where only the model can see them.
  • Every canary the agent closes is logged, timestamped, and reviewed the same day.
  • The third canary eaten ends auto-close for that category — no discussion, no exceptions.

Traps, unlike random audits, deliberately visit the dangerous neighbourhood.

The quarter ends at level two, barely

Review week. One category — and only one — cleared the auto-close bar on the strength of its numbers, with reopen wired to a single click and the canary contract still running. The reversible-action tier stayed shut: they had not rehearsed rollback against live infrastructure, so the 1%-on-300-verdicts standard remained unmet, honestly. Irreversible steps were marked “human, permanently”, which the team noted is less a rule of the tool than a rule of the field. They finished the quarter at Measured, with a signed register naming which action class held which clearance and when that evidence expires. The skills gap they felt most acutely — adversarial testing discipline — maps closely to what Certified Kubernetes Security Specialist (CKS) Skills Guide for DevOps Professionals teaches for cluster hardening, and the team booked study time accordingly.

The demo they declined

Between the shadow month and the canary work, a second vendor came calling with a smoother deck and a bolder number: “97% automated from day one.” The team asked one question — evidence file, per category, blind-labelled — and the meeting ended shortly after. Gartner keeps a name for this pattern, AI washing: capability vocabulary doing the work that capability should do. The team had also read the 2019-to-2022 research series in which F1 readings close to 0.99 turned out to be artifacts of spatial and temporal bias in how test sets were assembled, so a laboratory-perfect score drew a smile rather than a signature. Percent-automated, they had learned, measures how many tickets disappeared, never how many threats did. The safest sentence in their diary that month may have been the shortest: “Show us the misses.”

What AIOps for Security promised versus what the quarter delivered

The marketing promised a smaller queue; the quarter delivered something more valuable and less flashy — a measurement habit. Three outcomes made the retrospective: monthly per-category evaluation is now calendar furniture, the gate values came from Meridian’s own labelled data instead of a brochure, and the residual 6.5% of work that stayed human turned out to be the part where judgement actually earns its salary. The assistant reads faster; the team still decides. That division of labour, formalised in levels and evidence files rather than vibes, is the quiet achievement worth copying. And the cost of the discipline turned out to be smaller than feared — the whole quarter consumed perhaps two analyst-days of extra bookkeeping, a price measured against the alternative of discovering a missed threat in an incident review instead.

Small teams often assume this discipline belongs to enterprises with twenty-person SOCs. Meridian’s diary argues otherwise: the scoring exercise, the shadow month, the fifty traps — none of it needs headcount, only the stubbornness to measure before trusting. That stubbornness, scaled, is what the whole field is really selling.