SLO Alert Tuning

Turns a reliability target — an SLO (Service Level Objective, e.g. "99.9% of requests succeed over 30 days") — into alerts that wake a human only when users are actually being hurt, and open a low-urgency ticket when the service is degrading slowly. The goal is alerts that are both precise (few false pages) and fast to fire (short detection time) when a real incident is burning the error budget.

The base model knows what an SLO and an error budget are, but asked to "set up alerting for this service" it defaults to the wrong thing: a pile of static-threshold alarms on causes — CPU over 80%, memory over 90%, latency over 500ms, error rate over 1% for 5 minutes. Those page constantly during harmless spikes, stay silent through slow burns that miss the SLO, and tie the on-call rotation to machine internals no user can feel. Replace that default with the model below.

When to Activate

Activate when the user wants to:

• Define or generate alerting rules for a service that already has an SLO or an error target
• Fix alerting that pages too often (noisy), or that stayed quiet through a real user-facing outage (missed)
• Replace resource-threshold alarms (CPU / memory / disk / queue depth) with something tied to user impact
• Decide which conditions should page a human at 3am versus file a ticket for business hours

Do not activate for choosing which SLI to measure in the first place, for capacity/scaling forecasts, or for writing the postmortem after an incident (that is incident-postmortem).

Rule 1 — Alert on symptoms, not causes

Page on what the user experiences: the request failing or being too slow. Do not make a resource metric the primary alert. High CPU, high memory, a full queue, or a saturated connection pool are causes — a service can run hot and still serve every request inside its SLO, and it can be starved of CPU while users are perfectly happy. Paging on a cause produces a false page every time the system absorbs load gracefully, and misses every incident whose cause you didn't think to alarm.

Resource metrics still belong on dashboards and can drive capacity (auto-scaling, planning). They just aren't the thing that pages. The one narrow exception is a slow, predictable, unrecoverable exhaustion with a long lead time — disk filling at a steady rate, a certificate or credential expiring on a known date, a license quota counting down. Those get a low-urgency ticket ahead of time because by the time they show up as user-facing errors it is already an outage. Everything else pages on the symptom.

Rule 2 — Measure burn rate, not raw error rate

Burn rate is how fast you are spending the error budget relative to the rate that would exactly exhaust it over the whole SLO window. Burn rate 1 means: keep this up and you finish the budget precisely at the end of the window (e.g. exactly at day 30). Burn rate 10 means you would exhaust the entire 30-day budget in 3 days.

Two facts make burn rate the right trigger:

• It normalizes across SLO targets. A 1% error rate is catastrophic for a 99.99% service and a rounding error for a 99% service. Burn rate collapses that into one comparable number.
• It ties urgency to time-to-exhaustion. A high burn rate means the budget disappears in hours (page now); a burn rate just above 1 means it disappears in weeks (a ticket).

The alert threshold on the observed error rate for a given burn rate is:

error-rate threshold = burn-rate × (1 − SLO)

For a 99.9% SLO (1 − SLO = 0.001), a burn rate of 14.4 fires at a 1.44% error rate; a burn rate of 6 fires at 0.6%.

Rule 3 — Use two windows per alert (multi-window)

A single-window burn-rate alert forces a bad trade. A short window (5 min) detects fast but fires on any brief blip. A long window (1 hour) is stable but slow to fire and, worse, slow to stop — it keeps paging for an hour after the incident is already resolved.

Fix it by requiring both a long and a short window to be over threshold for the alert to fire:

• The long window gives precision — a real sustained burn, not a 30-second spike.
• The short window is a reset condition. Because it clears quickly once errors stop, the whole alert de-asserts within minutes of recovery instead of hanging on for the length of the long window.

An alert fires only when the long window and its paired short window are both burning above the threshold.

Rule 4 — Multiple burn rates: page fast, ticket slow (multi-burn-rate)

One burn rate can't cover both a sudden total outage and a slow chronic leak. Run several alerts at different burn rates, each with its own window pair, and route them by urgency. The canonical set for a 30-day SLO:

| Burn rate | Long window | Short window | Budget spent if sustained | Route | |-----------|-------------|--------------|---------------------------|-------| | 14.4 | 1 hour | 5 min | 2% of 30-day budget in 1h | Page | | 6 | 6 hours | 30 min | 5% in 6h | Page | | 1 | 3 days | 6 hours | 10% in 3 days | Ticket |

The two fast, high-burn alerts page the on-call responder — the budget is vanishing in hours. The slow, low-burn alert opens a ticket for business hours — a chronic drip that will breach the SLO if ignored but needs no one woken up tonight. Add the columns for other SLO targets from the reference table when the target isn't 99.9%.

Page vs ticket is decided by burn rate and time-to-exhaustion, never by which subsystem is involved. A fast burn on any user-facing path pages; a slow burn tickets. Do not route by cause ("database alerts page, cache alerts ticket") — route by how fast users are losing the budget.

Reviewing existing alerting

When asked to review or fix an alert set, check it against the four rules and name the specific gap:

• Pages on a resource/cause metric (CPU, memory, queue, pool) → it will false-page on absorbed load and miss unforeseen causes. Move it to symptom-based error-budget burn; keep the resource metric on the dashboard.
• Single static error-rate threshold ("error rate > 1% for 5 min") → no notion of budget; fires on blips and misses slow burns. Replace with multi-window multi-burn-rate.
• Single window → either noisy (short only) or slow to fire and slow to clear (long only). Pair a long window with a short reset window.
• One burn rate only → misses either the fast outage or the slow leak. Add the tiers.
• Everything pages / nothing tickets → the on-call gets woken for slow burns that could wait. Split page (fast burn) from ticket (slow burn).

Reference

references/burn-rate-windows.md — the full multi-window multi-burn-rate table across SLO targets (99%, 99.9%, 99.95%, 99.99%), with each tier's window pair, burn rate, error-rate threshold, budget consumed, and page/ticket routing. Look up the row for the service's actual SLO instead of assuming 99.9%.