Reliability
Part of Cloud monitoring with golden signals, SLOs and alerts
Monitoring application latency alongside server resources
Read request latency with traffic, errors, dependencies and server resources to investigate slow application actions.
Measure the time users wait for an application action, then read server and dependency measurements beside it. CPU, memory and disk charts can help explain a slowdown, but none measures the whole user experience. Ask which operations slowed, when, and what changed along their path.
Define the operation and timing boundary
Choose a meaningful operation, such as searching a catalogue or saving a form. State whether timing starts and ends at the client, gateway or application handler. Each boundary covers different parts of the request path, so label it clearly, and take start and end times from the same clock where possible.
Server-side timers show what the handler did. Synthetic checks show what an outside client sees: Application Insights availability tests, for example, send web requests to the application at regular intervals and alert if it isn't responding or the response time is too slow. Neither replaces timings from real users.
Track successful and failed requests separately. A fast failure can make combined latency appear better even as users lose the ability to complete an action.
For server timing, the OpenTelemetry convention is the http.server.request.duration histogram, the duration of HTTP server requests, in seconds. With its http.route attribute, the matched route template, each operation can be read on its own where the framework supplies it.
Look at the distribution as well as the average: a median describes a typical request, while the 95th percentile may expose a slower group. With a Prometheus histogram, the calculation of quantiles from the buckets happens on the Prometheus server using the histogram_quantile() function. Read a percentile with its time window, request count and operation label; low traffic can make a short-window estimate unstable.
Bucket boundaries limit precision. In Prometheus's own example, histogram_quantile estimates the 95th percentile at 228ms, but given the buckets the true value could be anywhere between 210ms and 229ms. Set bucket boundaries close to the latency target you care about.
Pros and cons of synthetic vs real-user monitoring
- Pros of Synthetic Checks (e.g., Application Insights)Consistent, repeatable tests from external locations; detects outages before users report them
- Cons of Synthetic ChecksMay not reflect real user behaviour or network conditions; can miss issues under load
- Pros of Real-User Monitoring (RUM)Captures actual user experience across diverse devices and networks; reflects true performance
- Cons of Real-User MonitoringData can be noisy; harder to isolate specific issues without traces or logs
Compare demand, dependencies and resources
When latency rises, compare it first with request volume and failures. Then inspect resources that could constrain the operation: CPU utilisation, memory, connection pools, queues, disk or network capacity. Check recent releases and configuration changes. Graphs that move together suggest where to investigate; they do not establish cause by themselves.
Measure dependencies from the application's perspective where possible. A database can report a quick query while its caller waits for a connection or network response. The OpenTelemetry metric db.client.connection.wait_time records the time it took to obtain an open connection from the pool, which separates pool waiting from query time.
A busy server can also be unrelated to the operation that slowed. OpenTelemetry's system.cpu.utilization is calculated for each logical CPU as the change in cumulative CPU time over a measurement interval, divided by the elapsed time, so align its interval with the latency window. Align charts by time, workload and environment before drawing conclusions.
Observation / Next question
- Latency and traffic rise together
- Is work waiting in a queue or limited pool?
- Latency rises while traffic is steady
- Did a release, configuration or dependency change?
- Failed requests become fast
- Is a path rejecting work early?
- One operation slows while others remain stable
- Which dependency or shared resource does it use?
The table offers investigation prompts, not diagnoses. Check a suspected cause with logs, traces or another relevant comparison before changing capacity or code.
Timeline of events during a latency spike
- 1015:00 – Latency spikes to 228ms — Observation: 95th percentile latency exceeds threshold
- 1015:15 – New deployment released — Change in code affecting user authentication path
- 1017:30 – System rollback initiated — Reverted to previous stable version
Alert on consequential latency
A latency alert should identify an important operation, an evaluation period, an owner and a response. Consider traffic or sample requirements so a tiny number of requests does not produce a misleading percentile.
Keep resource alerts when they predict a concrete failure or require a capacity action. A high CPU reading without current user impact can still inform planning without necessarily paging an operator.
To assess existing monitoring, follow one slow interval from the user-facing latency chart through traffic, errors, dependency timings and server resources. Where the evidence stops, improve the instrumentation needed to answer the next question.
