Reliability
Part of Cloud deployment workflows
Rolling back an application release safely
Decide how to respond to a faulty application release by checking version, traffic and data compatibility before verifying live service.
Before returning an application to an earlier version, check what changed besides the running code. Moving traffic back can restore a broken action when the earlier version still works with current data and dependencies. If the release changed a database, messages or external state incompatibly, switching code alone can create another fault.
Identify what needs reversing
Record the live artefact, the last known working artefact and the symptom that stopped the release. Establish whether the new version has written data, changed schema, started jobs or called an external service. Check whether the earlier version can read and write the current state.
A feature flag may disable one behaviour if the application was designed for it. Redeployment or a traffic shift may work when the older version remains viable. A forward fix may be necessary when current data cannot be used by older code. The incident owner must choose from the observed state; a rollback control cannot make that decision.
Set a decision point
Before release, agree on a symptom that pauses rollout, a decision owner and the response window. During an incident, record the failed action, affected users, first known time, current version and data state. Stop increasing exposure while these are checked.
| Situation | Candidate response | Condition to verify |
|---|---|---|
| Limited new-version traffic; shared state remains compatible | Route traffic to the earlier version | It still completes the critical action with current data |
| An incompatible data change has run | Pause affected writes and assess a forward fix or data recovery | The selected path preserves the required data and service state |
| Fault is confined to a switchable feature | Disable it through its approved control | The action returns without masking another failure |
These are decision prompts, not outcomes for a particular application.
Know what the platform reverses
Cloud Run can route traffic to an earlier service revision. Google says routing changes are not instantaneous and requests in progress continue during the transition. Kubernetes can roll a Deployment back to a retained revision, but that revision covers its pod template, not unrelated resources or database changes.
If old revisions have been removed, that rollback route is unavailable. AWS CodeDeploy redeploys an earlier application revision as a new deployment; it does not reconcile arbitrary effects of earlier deployment scripts.
Check the actual service mechanism before writing a runbook. Keep the earlier artefact, required configuration and permissions available for the decision window. Record the exact version to select; a label such as previous can be ambiguous after a partial rollout or another urgent change.
Platform capabilities for rolling back application changes
- Google Cloud RunRoutes traffic to earlier service revision; changes not instantaneous
- Kubernetes (Deployment)Rolls back to retained revision; only affects pod template, not databases or external resources
- AWS CodeDeployRedeploys earlier revision as new deployment; does not reconcile arbitrary effects of prior scripts
Verify the service and data state
After the response, check more than deployment status. Confirm that intended users can complete the affected action, errors and latency are acceptable, and writes reach the intended data store. Check delayed jobs and integrations touched by the release. If two versions handled traffic during transition, inspect records or messages that may need reconciliation.
Record the live version, what happened to data produced by the faulty version, unassessed effects and the follow-up owner. Prevent the faulty release from being promoted again without a new decision. The response is complete when the service and its data state meet the agreed acceptance conditions.
