Reliability
Cloud reliability
Define cloud reliability for business workloads, choose controls for likely failures, and check whether recovery meets agreed targets.
Cloud reliability means a workload performs its important functions when people need them and recovers within agreed limits when it fails. Start with the business action at stake, such as accepting an order or opening a patient record. Set measurable availability and recovery targets for that action. A provider's service commitment alone cannot show whether the whole application meets them.
Define the outcome before choosing an architecture
List the workload's critical user journeys. For each, record what counts as success, when the journey must be available, who is affected by failure and whether a reduced service is acceptable. A reporting dashboard might wait while order capture continues; treating both as equally urgent can misdirect effort.
Measure a journey from a point that reflects the user experience. Server health is useful for diagnosis, but a healthy server may still serve errors, run slowly or depend on an unavailable identity service. Specify the measurement window and which requests count. This gives the team an availability objective it can evaluate, not a vague promise of uptime.
Set recovery time objective (RTO) and recovery point objective (RPO) separately. RTO is the longest acceptable interruption before service is restored. RPO is the maximum acceptable age of the recovered data.
A system can return quickly yet lose too many recent changes, or preserve the data yet take too long to return. Business owners should agree on both. Engineers can then assess whether the design can achieve them.
Distinguish targets from measures and commitments
A service-level indicator (SLI) is a quantitative measure of one aspect of service performance; a service-level objective (SLO) is the measurable target set for a particular customer interaction. Use the SLI to assess whether the workload meets its SLO. A service-level agreement (SLA) is different: it is a contractual commitment between provider and customer, and missing it might have financial consequences.
Mean time to recover (MTTR) measures the time to restore a component after a failure is detected, while mean time between failures (MTBF) measures how long the workload performs its expected function before failing. These measures can help explain operational behaviour, but they do not replace targets tied to a critical user journey.
Reliability targets should also include relevant performance expectations. A service that is technically available but unusably slow may not meet the business need.
Set objectives against business impact
Estimate how the impact of downtime and data loss changes over time. Some costs may rise steadily; others may jump at a particular point, such as when an outage causes a production line to stop until the next shift. This helps business owners choose recovery objectives that reflect consequences rather than convenience.
Where workloads have similar criticality, assign them to predefined recovery tiers with corresponding availability and data-loss expectations. If a tier does not fit, set objectives for that workload and consider creating tiers later. Avoid arbitrary targets, targets that are looser than the business can accept, and zero recovery time or zero data loss when the workload cannot achieve them; unnecessarily strict targets can drive avoidable cost and complexity.
Match the control to the failure
| Failure to plan for | Useful design question |
|---|---|
| One application instance fails | Can healthy capacity take its requests, and does the application handle retries safely? |
| A zone or local facility is impaired | Are every critical component and its data available across the required failure boundary? |
| A region becomes unusable | Can the workload run elsewhere without depending on the affected region? |
| Data is deleted or corrupted | Is there a separate, usable recovery point from before the damage? |
| A bad release disrupts the service | Can the team stop or reverse the change without worsening the incident? |
Redundant instances can absorb some failures. They do not, by themselves, restore data damaged by an application error or make a second region ready for users. Similarly, a backup protects a recovery point but may require substantial time, infrastructure and operator work before the business journey functions again. Choose a combination according to the agreed targets and the failures that matter to the workload.
Check the entire path: traffic entry, application, database, identity, network connections, configuration and external services. A standby application that still calls a database or secrets service in the failed region is not an independent recovery path. Document which dependencies can be unavailable while the critical journey continues in a reduced form.
Choose the failure boundary proportionately
The location strategy should match the failures the business needs to withstand. AWS Well-Architected guidance describes distributing resources across Availability Zones or, where necessary, Regions; its guidance notes that Availability Zones are physically separated and independently supported, helping limit some infrastructure faults to the affected zone.
More locations are not automatically better. AWS identifies adopting a multi-Region design when multi-zone resilience would meet the business requirement, and allowing deployments or data to become desynchronised, as anti-patterns. Also account for components whose resilience requirements differ: a workload is only as recoverable as the dependencies its critical journey still needs.
AWS Investment in Australian Cloud Infrastructure
- Investment Amount
- $13.2 billion
- Region Coverage
- Sydney and Melbourne (multi-region support)
- Compliance Focus
- Australian Government data security standards
Multi-Zone vs Multi-Region Design Trade-offs
- Pros of Multi-Zone
- Lower latency, cost-effective, sufficient for zone-level failures
- Cons of Multi-Zone
- Not resilient to regional outages; shared infrastructure risks
- Pros of Multi-Region
- Resilience to full-region failure, supports disaster recovery
- Cons of Multi-Region
- Higher cost, potential data desynchronisation, complex management
Plan the recovery decision
A recovery plan needs an activation condition, a decision owner and a way to verify the result. Define what evidence distinguishes a regional disruption from an application fault that can be fixed locally. Record who can approve failover, how traffic and writes will move, what data loss is possible and when customers or staff should be updated.
Make the procedure accessible during an outage, including the credentials and tools it requires. Check that the alternative location has suitable capacity, service configuration and access permissions. If the plan depends on restoring from backup, include the time to provision infrastructure, retrieve data, validate it and reopen the user journey. If it depends on replication, know how to handle lag and prevent conflicting writes.
Plan the return path as well. After a failover, determine which copy of the data is authoritative, how changes will be reconciled and what conditions allow service to move back. An improvised failback can create a second incident.
Planning a Cloud Recovery Decision
- Define activation conditionEvidence that distinguishes regional disruption from local fault
- Assign decision ownerPerson authorised to approve failover
- Verify recovery pathConfirm traffic, writes, and data flow work in alternative region
- Plan return pathDetermine authoritative data copy and reconciliation method
Prove the workload works
Test a complete business action after recovery, not just a running virtual machine or a successful database restore. A useful exercise records detection time, decision time, restoration time, the age and integrity of recovered data, and any steps that required unavailable people or tools. Start with a controlled environment; expand the exercise carefully when the procedure and safeguards justify it.
Compare the observed result with the agreed RTO and RPO. If it misses either target, change the architecture or procedure, or renegotiate the target with the business owner. Repeat after meaningful changes to dependencies, data volume, deployment or staffing. A written runbook is a starting point; measured recovery is evidence.
Cloud Reliability Testing Process
- Controlled environment test
- Initial recovery simulation in isolated setup
- Expand testing scope
- Gradually increase complexity as safeguards prove effective
- Measure against RTO/RPO
- Compare observed recovery time and data age to targets
- Update architecture or target
- Adjust design or renegotiate objectives if targets are missed
Test specific resilience hypotheses
A controlled resilience experiment can test a specific hypothesis about how the workload behaves under disruption. For example, deliberately introducing a fault to a component or dependency can reveal whether the expected fallback occurs. Keep experiments bounded by safeguards, and use their results to update the failure scenarios and recovery procedure.
Reliability tests can also expose availability risks caused by security issues, performance problems or gaps in incident response, not just infrastructure failure. Include demand conditions where relevant: resilience includes continuing to provide an acceptable user experience as demand rises, as well as recovering within the agreed RTO and RPO.
Keep a short reliability brief
For each critical journey, maintain one record containing its owner, availability measure, RTO, RPO, main dependencies, credible failure scenarios, recovery path and latest exercise result. Review incidents against that record. This connects spending decisions to actual business needs: a team can strengthen a weak dependency or an untested restore before funding a more elaborate regional design.
Revisit the brief with business owners after an incident or material workload change. Update objectives and recovery procedures when evidence shows assumptions no longer hold.
In this guide
- Defining availability requirements for a business workloadDefine availability around a business journey, choose a useful measure, and record recovery targets alongside the objective.
- Comparing redundancy with recoverabilityCompare what redundancy and recovery each protect, where their limits differ, and how to combine them against business targets.
- Planning for a regional service disruptionPrepare a regional disruption plan with clear failover criteria, independent dependencies, data decisions and a tested return path.
- Testing a recovery procedure rather than assuming it worksTest the full recovery path, measure actual RTO and RPO, inspect recovered data, and record the limits of each exercise.
