Reliability
Part of Cloud monitoring with golden signals, SLOs and alerts
Choosing alerts that require a clear operator action
Decide which cloud signals deserve an alert by defining impact, owner, urgency and the operator's next step.
Choose an alert when a named operator can recognise the condition, judge its impact and take a useful next step in time to matter. If the response is unclear, keep the signal available for investigation while the team defines one. A dashboard can hold interesting measurements; a page asks someone to act.
Define the response before the rule
For each proposed alert, specify the affected workload, the user action at risk, the first diagnostic view, the owner and the available safe responses. Write what the operator should check and what decision the result will inform. If those steps remain vague, the rule needs more work.
For example, sustained failed payment submissions might lead an operator to check whether failures began after a release, whether the payment dependency is responding and whether other order actions still work. Those checks do not establish the cause. A CPU threshold becomes an actionable alert only when the team understands the risk it predicts and the intervention it calls for.
Route by urgency and consequence
An important user action failing now may justify a page. A capacity trend with time to plan may belong in a ticket. An informational change may need only a dashboard or event record. The owner should choose the route according to impact and the time available to respond.
Make the notification useful under pressure. Include the service and environment, affected action, observed condition and evaluation period, owning team and a way to reach the relevant diagnostic view. Check what the chosen alerting system can actually include and configure the message accordingly.
Alert routing by urgency and consequence
- High urgency / High consequencePage the on-call operator (e.g., failed payments in production).
- Medium urgency / Medium consequenceCreate a ticket for planned review (e.g., increasing CPU usage over 24 hours).
- Low urgency / Low consequenceLog to dashboard or event record only (e.g., minor configuration change).
Set a condition the team can trust
Choose an evaluation window that reflects the service's behaviour. A brief spike may be normal for a bursty workload; an overly long window may delay detection. Consider request volume and affected request count alongside an error percentage. Percentages can swing sharply with few requests, while counts can rise simply because demand increased.
Keep rules understandable. Group related notifications where that prevents several pages for one incident, but ensure the remaining alert still has a clear owner and response. Revisit the condition when traffic, the workload or on-call ownership changes.
Key alerting best practices
- Evaluation windowMatch the workload's normal behaviour (e.g., short for bursty systems).
- Error percentage vs. countUse both metrics—percentages vary with low volume; counts reflect demand changes.
- Group related alertsReduce noise but ensure each alert has a single owner and clear action.
Check before routing it to people
Ask five questions:
- What outcome is at risk?
- Who receives the notification at the time it may fire?
- What is the first safe investigation step?
- What would prompt escalation, mitigation or closure?
- Does another alert already lead to the same response?
Send a controlled notification through the intended route and inspect its recipient and content. After the alert enters service, use actual firings to judge whether it prompted the intended action.
