The symptom is familiar to most engineering teams: a large change ships, something regresses, and nobody can say which part of the change is responsible. People argue, roll everything back and lose days. Skill is seldom what is missing. More often the team is making big, bundled bets where small, isolated experiments would have told them what they needed to know. This article sets out the problem, the underlying causes and a practical way to learn through small experiments in a delivery pipeline.
The problem: large bets that cannot be explained
When a release bundles a new query path, a configuration tweak and a front-end change, a performance dip leaves three suspects and no witnesses. Engineers start guessing, and the guess that gets repeated loudest tends to win. Meanwhile customers experience the regression and the incident channel fills with theories.
Three patterns usually sit behind this. First, changes are batched to reduce the overhead of releasing, which makes each release larger and riskier. Second, there is no cheap way to expose a change to only some traffic, so every deployment is all-or-nothing. Third, the team has no habit of writing down a prediction before the change, so afterwards nobody can say whether the outcome matched expectations.
The diagnosis: cost of being wrong is too high
Behind the three patterns is one economic fact. If being wrong is expensive, people either avoid change or bundle it. Both reduce learning. The way out is to lower the cost of a mistake until it is cheap enough to make regularly. That means smaller changes, narrower exposure and faster reversal.
A small experiment does exactly this. It poses one question, alters one thing, exposes it to a limited slice, records the result and decides what to do next. It is a method for turning uncertainty into evidence without betting the whole service.
The solution: a repeatable experiment loop
Teams that adopt the loop tend to follow the same sequence. None of it requires heavy tooling; a configuration flag and a little discipline go a long way.
- Write the question. For instance: “Does the new caching rule reduce response time for the product listing endpoint?”
- Write the prediction. Record what you expect before any traffic sees the change, so you can judge honestly afterwards.
- Change one thing. Isolate the caching rule; do not ride along with unrelated edits.
- Limit the exposure. Route a small slice of requests through the new path behind a flag.
- Define success and stop conditions. Name the metric that confirms the result and the threshold that triggers an immediate switch-off.
- Observe for a fixed window. Long enough to cover normal variation in load.
- Decide and document. Expand, adjust or abandon, and write two lines explaining why.
A concrete example: flagging a slice of traffic
Consider a team that suspects a new serialisation library will cut latency. A big-bang swap would touch every endpoint. Instead, they put the library behind a flag and enable it for a small percentage of requests on a single low-risk endpoint. They watch latency and error rates for that slice against the untouched remainder, which acts as the control.
After a day, the slice shows lower median latency but a handful of malformed responses for one unusual input shape. The team turns the flag off, fixes the edge case in isolation and repeats the test. Total exposure to customers was a sliver, and the failure had a single, obvious cause. In a bundled release, the same defect would have been hidden among other changes.
A second example: a one-day configuration trial
Experiments do not have to involve code. Suppose a team wonders whether a connection pool setting is too tight. They pick a quiet day, alter the value on one instance out of several and compare its behaviour to its siblings under the same load. The rest of the fleet remains the baseline, and the change is trivial to revert. By evening they either have evidence that the setting matters or confidence that it does not.
The common thread is a control group and a boundary. Without a comparison you are just changing things and hoping.
Why teams skip experiments, and how to answer each objection
- “We do not have time.” A small test usually costs less than one incident review, and it prevents the next one.
- “Our traffic is too low.” Use a longer window or a qualitative check; evidence of any size beats none.
- “We lack tooling.” A simple flag and a spreadsheet are enough to begin.
- “It slows delivery.” Small, reversible steps shorten the path to a confident release by removing rollback drama.
- “Stakeholders want certainty.” An experiment is the most honest way to supply it, because it replaces opinion with observation.
Where process and planning meet experimentation
Good experiments rest on clear requirements. If nobody can articulate what a change is supposed to achieve, no metric will rescue the test. Our engineering blueprint for setting out web project requirements covers the groundwork that makes a measurable question possible. Similarly, if your stack struggles under heavy demand, a look at how modern cloud infrastructure copes with large traffic loads helps you decide which slices of traffic are safe to use as test populations.
Choosing what to measure
A test is only as honest as its yardstick. Pick one primary signal tied directly to your question, such as response time for the endpoint you touched, and one guard signal that would reveal collateral damage, such as the rate of failed requests. Agree on both before the flag is switched on. Adding a long list of extra charts invites cherry-picking, because with enough charts something always looks interesting.
Also decide how you will separate your slice from the rest. Random assignment is ideal. If you must choose by account or region, check beforehand that those groups behave similarly under ordinary load, or the comparison will mislead you. Write the assignment rule on the experiment page so the reviewer can see it at a glance.
Guarding against the usual failure modes
Experiments can go wrong in predictable ways. The most common is changing more than one thing, which brings back the original attribution problem. Next is peeking: ending the test the moment the numbers look good, which inflates false positives. Third is forgetting the exit: a flag that stays on for months becomes permanent technical debt.
Mitigate each with a short checklist: one change, a fixed window agreed in advance and an owner responsible for removing the flag once the decision is made. Treat expired flags as defects.
What the wider community can teach
Teams that build or operate physical things often have well-honed instincts for small trials. One public example is DronesBee, a drones-only website whose pages cover news and blog posts, listed gear, shopping, services for providers, lookup directories of pilots and operators, and an active forum. Reading how hobbyists and professionals trade notes on adjusting one setting at a time can reinforce the discipline you want in software: change a single parameter, observe, record. The domain differs; the logic is the same.
How to start on Monday
Do not begin with the riskiest system. Pick a low-stakes service, a question the team genuinely cares about and a change small enough to review in minutes. Write the prediction on a shared page, apply the change to a narrow slice and set a date to look at the result. When the date arrives, hold a short review that asks three things: did the outcome match the prediction, what surprised us and what is the next experiment?
Repeat that rhythm a handful of times. The team will start to find that arguments shrink, because the answer to “will this work?” becomes “let us find out cheaply”. Releases get smaller and confidence rises, not because anyone became bolder, but because being wrong has become affordable.
Summary
The diagnosis is that bundled, all-or-nothing releases make learning expensive and blame easy. The remedy is a disciplined loop of one question, one change, a limited slice, a recorded result and a clear decision. Start small, keep each step modest and let evidence rather than volume settle the debate.

