Operate systems with signals, not guesses

Agenda

By the end, learners should be able to

• Differentiate metrics, logs, and traces and when to use each
• Start monitoring with the Golden Signals (P50/P95/P99, not averages)
• Define an SLI + SLO and compute an error budget
• Adopt structured JSON logging and avoid PII leakage
• Explain common stacks (Prometheus/Grafana, EFK, Loki, OpenTelemetry)

What to monitor first

• Latency: P50/P95/P99 (avoid averages)
• Traffic: requests per second (RPS)
• Errors: 5xx/total, timeouts, failed jobs
• Saturation: CPU, memory, queue depth, connections

Latency:   p99_http_request_duration_seconds
Traffic:   http_requests_total
Errors:    rate(http_requests_total{status=~"5.."}[5m])
Saturation:node_cpu_seconds_total, queue_depth

Names shown are typical Prometheus-style metrics. The idea matters more than the exact names.

Definitions

• SLI: a measurement (e.g., P99 latency, availability)
• SLO: a target for an SLI (e.g., P99 < 200ms, 99.9% monthly)
• SLA: contractual commitment with penalties
• Error budget: 100% - SLO

Example

SLO: 99.9% monthly availability
Error budget: 0.1%
30 days: 43.2 minutes allowed downtime
30.4 days avg: ~43.8 minutes

When the error budget is consumed, freeze risky deploys until it refills.

Unstructured

2024-01-15 ERROR User login failed for john@example.com

Hard to query. Parsing is fragile.

Structured (JSON)

{"ts":"2024-01-15T10:00:00Z","level":"ERROR",
 "event":"login_failed","user":"john@example.com",
 "ip":"1.2.3.4","request_id":"abc-123"}

Queryable and filterable. Add request_id everywhere.

Do log

• Requests: method, path, status, latency
• Correlation: request_id, trace_id (if present)
• Errors: type, stack trace, context
• Business events: order_created, payment_processed

Dont log

• Passwords, tokens, credit cards
• Raw PII; mask/hash if required
• Everything at DEBUG in production
• Duplicate noisy logs without value

When traces pay off

• You have multiple services per request
• You need to find which hop is slow
• You need dependency visibility
• You want end-to-end latency breakdown

Typical approach

• Use OpenTelemetry SDKs
• Sample traces (not 100%)
• Export to Jaeger/Tempo/OTel Collector
• Correlate: trace_id into logs

Expose /metrics
- requests_total{route,status}
- request_duration_seconds{route}

Prefer histograms for latency
Use p95/p99 from histograms

Validate

• Hit /metrics and confirm counters increase
• Ensure labels do not explode cardinality
• Record latency as histogram, not raw values

Log once per request
{"ts":...,"level":"INFO","route":"/pay",
 "status":200,"latency_ms":18,
 "request_id":"...","trace_id":"..."}

Validate

• Every log line includes request_id
• Errors include stack traces + context
• No secrets or PII in logs

Dashboard panels:
- RPS
- Error rate
- p95/p99 latency
- CPU/memory saturation

Alert example:
error_rate > 1% for 5m

Validate

• Force a 500 to see logs + error rate move
• Ensure alerts are actionable (avoid noise)
• Use paging only for user-impacting issues

Start here

• Golden Signals dashboard for every user-facing service
• One SLO per service (availability or latency)
• Structured logs with request_id everywhere
• One paging alert tied to user impact

Then iterate

• Add tracing for multi-hop latency
• Refine alerts using error budgets
• Add log-based alerts for specific failure modes
• Review dashboards monthly with incidents