Six Myths That Keep Teams From Rehearsing a Release

Six Myths That Keep Teams From Rehearsing a Release

Years ago I watched a release rehearsal get cancelled because “the schedule was tight”, and I watched the real release take the whole night. The engineers were capable. The code was fine. What failed was a step nobody had ever performed outside production. Since then I have heard the same excuses for skipping a release rehearsal, and most of them fall apart on inspection. Here are six myths worth retiring, each with a short story from the field.

Myth one: “Our automated tests already cover this”

Tests tell you whether the code behaves. A release rehearsal tells you whether the delivery behaves: the order of steps, the permissions, the migration timing, the configuration that differs between environments. One team I worked with had a spotless test suite and still stalled during a release because a database migration needed a lock that took far longer on realistic data than on the test fixtures.

The rehearsal did not replace testing. It exercised something testing never touches, which is the act of shipping.

Myth two: “Staging never matches live, so the exercise is pointless”

Staging does differ from production, granted. That is an argument for improving the copy, not for abandoning the exercise. Even an imperfect rehearsal catches the clumsy mistakes: the missing environment variable, the script that assumes a path, the step that depends on a person who is on holiday. A rough rehearsal that finds half the surprises beats a perfect belief that finds none.

If you are working on a platform that has to survive heavy demand, a thoughtful read such as this look at keeping large traffic surges under control with cloud infrastructure shows how much the environment itself matters, and why a rehearsal environment should echo it as closely as budget allows.

Myth three: “Rehearsals slow us down”

They do add an afternoon. They also remove the long, tense night. I once timed two comparable releases from the same team. The rehearsed one took a modest part of a morning, with people eating lunch at their desks and chatting. The unrehearsed one swallowed an evening, two escalation calls and a missed family dinner. Speed measured over the whole release, not the preparation, favours the rehearsal.

Myth four: “Only big releases deserve a dry run”

Small changes cause plenty of outages, partly because people relax around them. A rehearsal does not have to be dramatic. For a minor release it can be a ten-minute walk through the exact commands on a test environment. The scale of the rehearsal should match the risk, but the habit should not depend on how impressive the release feels.

Teams that reach for a formal plan before building anything will recognise this logic from a detailed engineering view of gathering web project requirements: a little preparation scaled to the work pays back many times over.

Myth five: “If it goes wrong we can just fix it live”

Sometimes you can. But fixing live means making decisions under pressure, with customers watching, and often without the tools you would use in calm conditions. A rehearsal swaps that pressure for a quiet room. Mistakes become learning, and the person who would have been guessing at midnight is instead writing an improved note at two in the afternoon.

Whatever you learn in production arrives with an audience and a ticking clock, so a gap found there costs far more than the same gap found in rehearsal.

Myth six: “Rehearsal means formality and paperwork”

The best rehearsals I have seen were lightweight. A shared terminal, a short list of commands, a nominated reader and a nominated doer. They talked it through, ran it, noted where it hesitated and fixed the instructions. There was no ceremony and no committee. The only artefact was a better script.

It also helps to separate the rehearsal from a performance review. When people fear that a failed rehearsal will be held against them, they quietly smooth over problems and the exercise loses its value. The healthiest teams I have worked with applaud the person who triggers a failure in the practice run, because that failure is precisely what the rehearsal is for. Say so out loud before you start, and keep the notes about what went wrong focused on the process, never the individual.

Finally, keep the rehearsal honest about timing. Record when each step starts and stops, including the waiting. Teams are often surprised that a migration, a cache warm-up or a certificate renewal takes longer than the plan assumed, and those waits decide how long a maintenance window really needs to be.

A story about a missing credential

One rehearsal I remember particularly well involved a modest team and a routine deployment. About five minutes in, the person running the commands hit a permission error. The service account needed for the final step belonged to someone who had changed roles months earlier. Nobody had noticed because nobody had used it since.

On release night that error would have been a crisis. On a Wednesday afternoon rehearsal, it was a ticket and a cup of tea. By Friday the credential was reissued and documented, and the real release went through without a hiccup.

An unexpected neighbour for this idea

The principle is not confined to software. Cooks have long practised dishes before important gatherings, and communities that care about traditional food keep recipes alive by trying them repeatedly. Blendz presents itself as a world-spanning home for heritage recipes, ingredients and food stories, and it welcomes contributions from readers. A recipe that has been cooked and corrected a few times is the culinary equivalent of a well-rehearsed release.

How the myths survive

These beliefs persist because the cost of skipping a rehearsal is invisible until it is large. A release that goes smoothly without practice feels like proof that practice was unnecessary, even when luck did most of the work. A release that fails is blamed on the code or the individual. Rarely does anyone say that the delivery sequence had simply never been performed before. Writing down the near misses from each rehearsal, and showing them to the doubters, is the quickest way to change the conversation.

What to do on Monday

Take the next change that is going to production. Ask where it could be rehearsed, even crudely, and who should be in the room. Perform every step instead of reading it. Record each hesitation. Update the instructions straight afterwards. Next time you will be shorter, calmer and better prepared, and the myths will have less to say.