Most deployment pipelines are designed around the happy path: build, test, deploy, done. Recovery is treated as an edge case — something to figure out if it ever comes up. It always comes up, and by the time it does, "figuring it out" happens under pressure, in production, with people watching.
A pipeline designed to recover from failure looks different from one designed only to deploy successfully. Here's what that difference actually consists of.
Rollback has to be a tested path, not a theoretical one
If your rollback procedure is "redeploy the previous commit," ask: has anyone actually run that in the last month? Configuration drift, database migrations and dependency changes all make "just redeploy the old version" less reliable than it sounds. A rollback procedure that isn't exercised regularly is a hypothesis, not a plan.
The practical fix is to make rollback a first-class pipeline action — a single command or button that redeploys the last known-good artifact — and to actually run it periodically in a non-production environment, the same way you'd test a backup restore.
Database migrations need a forward-compatible strategy
The single most common reason rollback fails in practice is a database migration that isn't backward compatible with the previous application version. If version N+1 adds a required column and rollback restores version N's code, but the schema is now N+1's schema, rollback breaks.
The standard fix is expand-contract migrations: add new columns as nullable, deploy the code that uses them, backfill data, and only remove old columns in a later, separate release once nothing depends on them. This adds a step to the process, but it's what makes rollback safe rather than aspirational.
Deployment strategy determines your blast radius
A pipeline that deploys 100% of traffic to a new version at once has no way to catch a bad deploy before it affects everyone. Progressive delivery — canary releases, or blue/green with a traffic-shifting step — gives you a detection window before a bad release becomes a full incident.
This doesn't have to be complex. Even a basic canary — 5% of traffic for ten minutes, watched against error-rate and latency SLOs before promoting to 100% — catches a meaningful share of bad releases automatically, without a human in the loop deciding in real time whether something "looks off."
Observability has to exist before the deploy, not get added during the incident
A pipeline can't recover from a failure it can't detect. This means the metrics and alerts that would catch a bad deployment — error rate, latency, saturation — need to exist and be wired to the deployment process before the release, not added retroactively after a bad one. Tie your canary or promotion gate directly to these signals rather than to a manual "looks fine to me" check.
What this adds up to
A recoverable pipeline isn't more infrastructure for its own sake — it's four specific decisions: a tested rollback path, forward-compatible migrations, progressive rather than all-at-once delivery, and observability wired in before release rather than after. Each is a deliberate design choice, not a default you get from any particular CI tool.
Related services