Designing SLO-Based Alerting
Most on-call pain comes from alerts that measure infrastructure, not outcomes. This playbook shows how to design alerting around Service Level Objectives (SLOs) so pages reflect genuine user impact.
Who Benefits
SRE and platform teams reduce alert fatigue. Product and engineering leadership gain a shared language for reliability through error budgets.
Prerequisites
- A service with defined user journeys (e.g. checkout, API read, background sync)
- Metrics infrastructure (Azure Monitor, Prometheus, or similar)
- Agreement on target reliability (e.g. 99.9% availability over 30 days)
Step 1 — Define SLIs
Choose indicators that reflect user experience:
| Service type | Example SLI |
|---|---|
| HTTP API | Ratio of successful requests (non-5xx) to total requests |
| Web application | Page load time under threshold |
| Async pipeline | Messages processed within latency target |
Step 2 — Set SLO Targets
An SLO is a target percentage over a rolling window:
SLO: 99.9% of API requests return non-5xx over 30 days
Error budget: 0.1% = ~43 minutes of downtime per monthStart conservatively. Tightening an SLO is easier than loosening one after teams adapt.
Step 3 — Design Burn-Rate Alerts
Alert when error budget is consumed too quickly, not on every individual failure:
| Alert tier | Burn rate | Window | Purpose |
|---|---|---|---|
| Page | 14.4× | 1 hour | Critical — budget exhausted in hours |
| Ticket | 6× | 6 hours | Investigate during business hours |
| Review | 1× | 3 days | Trend analysis in weekly review |
Step 4 — Route and Document
- Page alerts go to on-call with a runbook link
- Ticket alerts create a tracked work item
- Review alerts appear on a reliability dashboard, not pager
Each alert should state: affected SLO, current burn rate, remaining error budget, and first diagnostic step.
Step 5 — Verify
- Inject failures in a non-production environment and confirm burn-rate alerts fire at expected thresholds
- Review alert history after 30 days — if on-call receives pages for non-user-impacting events, refine SLI selection
- Confirm leadership can read error budget status without engineering translation
Anti-Patterns to Avoid
- Alerting on CPU or memory without linking to user impact
- Identical thresholds for all services regardless of criticality
- Paging on SLO miss after the error budget is already exhausted (too late to act)
Next Steps
- Define SLIs for your highest-traffic service
- Replace one infrastructure alert with a burn-rate alert this sprint
- Schedule a monthly error budget review with product stakeholders
Last updated on • Steve Rackham