How to Run a Delivery Maturity Audit in Seven Steps

How to Run a Delivery Maturity Audit in Seven Steps

Engineering leaders tend to discover the state of their delivery organisation the expensive way: a release that would not roll back, an outage that exposed an undocumented dependency, an auditor’s question nobody could answer. A delivery maturity audit is the cheap way — a structured, evidence-led scoring of how your software actually moves from commit to production, run on a schedule rather than after a failure. This guide walks through the full procedure in seven steps, sized for a team that wants results in two weeks, not a quarter.

One framing note before step one. The audit’s output is not a document; it is a pair of numbers separated by time. The first score establishes position. The second, taken months later against identical criteria, establishes direction. Everything in the procedure below is designed to protect the comparability of those two numbers.

Step 1: Fix the scope in writing

Choose one value stream — one product or service, commit to production — and name it. Audits that try to cover every team at once produce averages that describe nobody. Write a scope line such as: “the payments service, from merge to customer traffic, including its pipeline, infrastructure code, and on-call arrangements.” Anything outside the line is out, this cycle.

Step 2: Choose your instrument before you look at anything

The instrument is the fixed set of questions you will score against, and it must be chosen before evidence-gathering, or you will unconsciously write questions your team already passes. You have two honest options. Adopt an existing one: SCMGalaxy, the training and consulting firm behind the SCMGalaxy OS governance platform, assesses delivery organisations with a bank of one hundred questions spanning ten governance areas, and its free assessment entry point makes it a low-friction baseline. Or write your own smaller bank — forty questions is workable for a single value stream — provided every question demands evidence rather than opinion. “Do you have CI?” is a bad question. “What percentage of the last fifty merges ran the full test suite before deploy?” is a good one.

Step 3: Gather evidence, not testimony

Allocate three or four days. For each question, collect the artefact that answers it: pipeline run histories, deployment logs, the actual rollback procedure document (and its last-modified date), access lists, incident timelines. Interview people only to locate artefacts, never as a substitute for them; a claim without a file behind it scores as though the file were absent. Teams routinely believe, sincerely, that their build process is reproducible; the evidence pass is where you learn the release artefact is assembled on one engineer’s laptop. How builds are produced and where their outputs are stored is one of the areas enterprise instruments probe hardest, precisely because testimony and reality diverge there most.

Step 4: Score mechanically

Grade each question 0–5 against written criteria agreed in step 2. The discipline to copy from the platform world is determinism: SCMGalaxy OS is built so a given set of answers yields one and only one score, which is what makes its results usable for benchmarking between teams and across time. Your manual version achieves the same property by banning judgment at scoring time — if two people can read the same evidence and assign different numbers, tighten the criterion until they cannot. Record a one-line evidence citation beside every score. The citations are what the re-audit will check.

Step 5: Rank the gaps by blast radius, not embarrassment

Sort low scores by what they can cost, not by how uncomfortable they are. A 1/5 on deployment safety for a revenue-bearing service outranks a 0/5 on documentation for an internal tool. Three questions sharpen the ranking: Which gap could cause an irreversible loss (data, revenue, trust)? Which gap do other gaps hide behind? Which gap is currently masked by one heroic individual? Security findings deserve particular suspicion of their own scores — a weak grade there compounds quietly, and the case for treating application security as revenue protection rather than overhead is the right lens for pricing those gaps.

Step 6: Convert the ranking into a 30/90/180-day roadmap

Cap the roadmap at the top two or three gaps and stage it on the three horizons the governance platforms have standardised — a first month, a first quarter, a half year — because open-ended improvement plans decay into wishes. The 30-day tranche holds mechanical work: pin the build environment, script the rollback, revoke stale access. The 90-day tranche holds habit formation: the new pipeline gate enforced on every merge, the deployment checklist survived a crunch week. Automation belongs here — most 90-day items reduce to replacing repetitive manual steps with automation that removes error at specific points the audit flagged. The 180-day tranche holds proof: a target number, stated in advance, that the re-audit must find — lead time halved, failed-deploy rate below a threshold, restore rehearsed twice.

Resist folding the roadmap into a general replatforming plan. Rebuild decisions have their own discipline — closer to drafting a full engineering requirements blueprint before a web project than to an audit remediation — and mixing the two lets the audit’s small, dated commitments drown in a large, undated one.

Step 7: Re-audit on the date you booked in step 1

Same instrument, same scoring criteria, same evidence standards, ideally a different person holding the pen. Compare per-question, not just per-total: totals can rise while the specific gaps you funded stay flat, which is a failed quarter wearing a passing grade. Publish both sheets side by side to the team. Then — only then — admit new questions into the instrument for the next cycle, versioning it so old comparisons remain valid.

Failure modes to engineer against

  • Instrument inflation. Every cycle someone proposes ten new questions. Accept one or two, version the instrument, and keep a frozen core so trend lines survive.
  • Self-grading drift. After two cycles, scorers learn which evidence to show. Rotate the pen-holder and spot-check three citations per audit at random.
  • Roadmap laundering. Pre-existing pet projects get relabelled as audit remediation. The test: if the item existed before the audit, it needs its own funding case.
  • Averaging away the signal. Reporting one blended score to leadership hides the 1/5 that matters behind four comfortable 4/5s. Report the floor, not the mean.

What the evidence says about cadence

Teams that audit once never trust the exercise; the first score has nothing to stand against. Twice is the minimum for signal. Quarterly is the practical ceiling for most organisations — below ninety days the roadmap items have not had time to land, and you are re-measuring wet paint. The platforms formalise this rhythm as scheduled re-assessment with progress tracked between runs; a spreadsheet and a calendar invite implement the same contract at team scale. Either way, the contract is identical: fixed questions, mechanical scores, two or three funded fixes, and a date on which the numbers must answer for themselves.