Reliability
Part of Cloud monitoring with golden signals, SLOs and alerts
Recording logs useful for incident investigation
Choose application log events, fields, identifiers and safeguards that help operators reconstruct an incident.
Record events that help an investigator reconstruct what failed, where it failed and which request or change was involved. Consistent, searchable records are more useful than a large volume of vague messages. Begin with the questions an operator will need to answer, then choose the events and fields that support them.
Choose events by investigation need
Useful application events may include request failures, dependency timeouts, job retries, significant state changes and deployments. Record normal completions where they help establish a sequence or verify an important action. Log an outcome at the point that knows it: a server access log may show a response code without explaining why an application rejected a transaction.
Ask which operation failed first, whether the problem followed a release and whether retries masked a failing dependency. Keep an event when it helps answer a defined operational question. Review its volume and retention if it does not.
Record enough context to connect events
A structured event should identify its time, service, environment, operation, outcome and stable event type. Include a request or trace identifier where one is available and can be propagated across the relevant components. A safe error category or dependency name can make the record easier to interpret. Use consistent field names and time conventions.
A payment timeout record, for example, can identify the operation, outcome, dependency and request ID without storing a customer's full request body. If the logging system supports both event time and collection time, retaining both can help explain delays caused by buffering. Check that the collection and storage path preserves fields needed for investigation; a source emitting a field does not guarantee it remains searchable downstream.
A short message can explain an unusual condition, but do not make free-form text the only way to find an event. Stable event types and outcome fields allow related occurrences to be compared across releases.
Log fields that enhance incident investigation vs. those to avoid
- Recommended: Request/trace ID
- Enables correlation across services and components
- Recommended: Stable event type and outcome
- Allows filtering and comparison across releases
- Avoid: Full customer request body in logs
- Exposes sensitive data; use sanitised versions instead
- Avoid: Raw session identifiers or access tokens
- Risk of credential leakage; never log in plaintext
- Recommended: Time of event and collection time
- Helps identify delays due to buffering or system lag
- Avoid: Free-form unstructured messages only
- Hard to search or analyse at scale; use structured fields
Protect sensitive data and keep logs available
Keep passwords, access tokens, raw session identifiers, connection strings and other secrets out of routine logs. Review personal data before recording it and use the least detail needed for the investigation. Sanitise untrusted text so it cannot break the log format or mislead a viewer. Restrict access to those who need the records.
Logs stored only on an instance that disappears during a fault may be unavailable when needed. Decide which sources need collection elsewhere, how quickly operators must be able to query them and how long they must remain available. Retention should reflect operational, security and organisational requirements; no single period fits every workload.
Best practices for log retention and availability
- Retention period (varies by workload)
- Align with operational, security and compliance needs
- Logs stored locally on transient instances
- Not reliable during outages; must be collected elsewhere
- Access control for logs
- Restrict to authorised personnel only
- Sensitive data handling
- Sanitise or omit secrets like passwords and tokens
Check whether an incident can be reconstructed
In a controlled environment, create a known failure and follow it from the user-facing symptom to the application event and relevant dependency record. Check timestamps, identifiers and outcomes, look for missing records, and inspect the view for sensitive data. Use the gaps to improve the event schema or collection path.
