Reliability
Part of Managed databases in the cloud
Planning database backup and restore tests
Set a database restore test pass condition, isolate the restored copy and measure data and application recovery.
Plan a restore test around a specific loss event and a working application action. Decide which data must return, how old it may be and when users must be able to use it. A successful backup job alone does not establish that the application can recover within those limits.
Define a pass condition
Choose a scenario such as accidental deletion, a damaging release or loss of a database instance. Record the recovery time objective (RTO), recovery point objective (RPO), test owner and application action that will demonstrate recovery. For an order system, the action could be finding a known order and creating a new one in an isolated test environment. This is an example, not a reported result.
Select a recovery point before the simulated loss. Identify data expected to be present and data expected to be absent. Check the service's actual earliest and latest restorable times and its configured retention before the exercise.
Key Metrics for Backup and Restore Testing
- Recovery Time Objective (RTO)
- Time from incident to usable system
- Recovery Point Objective (RPO)
- Maximum acceptable data loss (in time)
- Earliest Restorable Time
- Service-dependent; check retention settings
Restore into an isolated environment
List the users, network rules, parameters, certificates, secrets and connection settings needed by the application. Check which settings the chosen restore method carries over and which need reconfiguration. Use restricted test access.
Measure usable recovery
Start timing from the incident point defined for the RTO. Record when the issue was identified, the restore requested, database connections accepted and the application action completed. Amazon notes an RDS instance can become available while restored volumes continue loading data blocks in the background, which may affect initial performance.
Validate data at the chosen recovery point with known records or relevant business totals. Then test a read and a controlled write through the application. Record the age of recovered data and whether the observed recovery met the RTO and RPO. If either cannot be assessed, say so.
Measuring Usable Recovery Time
- Incident identified
- Timestamp of detected failure
- Restore initiated
- Time restore request was submitted
- Database connections accepted
- When instance becomes available
- Application action completed
- Successful order lookup or creation in test environment
Keep a useful test record
Record the scenario, source configuration, recovery point, elapsed steps, data checks, settings recreated, application result and any resources created for the exercise. Remove temporary resources through the normal change process when the test is complete. Repeat after meaningful changes to data volume, retention, access settings or application dependencies. An isolated point-in-time restore tests one recovery route; it does not prove regional failover.
