Cloud monitoring with golden signals, SLOs and alerts

Cloud monitoring shows whether a workload serves its users and gives operators evidence to investigate problems. Start with an important action, such as submitting an order or opening a staff application. Measure whether it succeeds and how long it takes. Then add infrastructure signals that may help explain a failure or slowdown.

Measure the workload users depend on

A healthy virtual machine does not prove a customer can complete a transaction; the application, database, identity service or another dependency may fail. For each important action, define the expected result, where it is measured, its owner and the operating hours that matter.

Latency, traffic, errors and saturation are useful starting measures: Google's SRE guidance calls them the four golden signals and advises that, if you can only measure four metrics of a user-facing system, you focus on these. Define which requests count and what constitutes failure. A successful HTTP status can still accompany an incorrect result or an unfinished business action.

Turn each important action into a service level indicator (SLI), a carefully defined quantitative measure of the service provided, such as the proportion of order submissions that succeed or that finish faster than a threshold. A service level objective (SLO) sets the target; Google's example reads: 99% of Get RPC calls will complete in less than 100 ms.

Application telemetry shows what the service handled. An external check can show whether an action is reachable from its chosen vantage point. Neither view covers every user experience.

Top 4 Golden Signals for Cloud Monitoring (Google SRE)

Use monitoring for planning and change

Track trends in resource use and service health for longer-term planning. Compare behaviour before and after a change to understand its effect.

Where a workload has experimental changes or distinct groups, comparison between those groups can help show whether behaviour differs. Decide which of these uses matters to the business before choosing what information the monitoring strategy needs to retain and present.

Connect symptoms to diagnostic signals

Keep user-facing measures prominent on a dashboard. Place relevant diagnostic signals nearby: application errors, dependency timings, queue depth and resource pressure. A rise in CPU use is a clue, not proof that CPU caused a slow request. Compare it with traffic, errors and the timing of changes before acting.

Metrics reveal patterns; logs provide details of recorded events. Traces can show a request's path through instrumented components. Where practical, propagate and record a request or trace identifier so an operator can connect related records. Decide which logs need to remain searchable during an incident and who may access them.

A dashboard should make it easy to see what users are experiencing, what changed and whether a mitigation helped. Label time ranges, units, workloads and relevant deployments so the graphs can be interpreted. For latency, show percentiles: a high percentile such as the 99th shows a plausible worst case, while the 50th (the median) shows the typical case.

Telemetry can include metrics, text logs, structured event logs, distributed traces and event introspection. OpenTelemetry is an open-source, vendor- and tool-agnostic observability framework for instrumenting applications; it works with backends such as Jaeger and Prometheus but is not a backend itself.

Choose capabilities around workload needs

Monitoring systems collect, process, aggregate and display quantitative data about a system. When evaluating one, consider how fresh the data needs to be and how quickly it must be retrieved; different organisations have different needs for both.

A single system may cover the workload’s use cases, or a combination of systems may be more suitable. Azure Monitor, for example, is Microsoft's observability service that brings metrics, logs, traces and events together; open-source stacks often pair OpenTelemetry instrumentation with Prometheus. Prioritise capabilities according to the work monitoring must support, such as displaying service health, investigating issues or identifying trends for planning.

Azure vs AWS vs Google Cloud: Key Monitoring Capabilities for Australian Businesses

Alert for a decision

Before enabling an alert, identify the affected service, likely impact, owner and first useful check, and write that check into a runbook linked from the alert. Route a condition requiring prompt human action to the on-call operator; send work that can wait through a less urgent channel. A resource threshold deserves a page when it signals a credible problem and someone can intervene.

Choose thresholds and evaluation windows from the workload's behaviour. A sustained rise in failed checkout attempts may call for checking the latest release and dependency errors. A brief CPU spike may call for no immediate action. There is no universal threshold for either condition.

For SLO-based alerting, Google's SRE workbook recommends alerting on burn rate: how fast, relative to the SLO, the service consumes its error budget. In its example, a service with 100% error spikes every 10 minutes used almost 12% of a 30-day budget per spike, yet a simpler alert never triggered.

SLO-Based Alerting: Understanding Error Budget Burn Rate

Check the monitoring path

Assign owners for instrumentation, collection, storage, dashboards and notification routes. Periodically check that expected telemetry arrives and that service labels and timestamps allow records to be compared. Use a safe, controlled condition to check whether a notification reaches its intended recipient with enough context to investigate.

After an incident, compare the first user impact with the first useful signal and the first human response. Customer reports may reveal a missing measure. A flood of pages may reveal alerts that obscured the useful one.