--- name: capacity-headroom-planning description: "Sizes a resource (nodes, pool, workers, storage) from a load number the disciplined way instead of \"provision for the current peak.\" Covers deriving true peak from a peak-to-average ratio, dividing by a utilization ceiling instead of running to 100%, adding N+k failover headroom so survivors stay under the ceiling when a node dies, and projecting a time-to-saturation date from the growth trend so you scale ahead of it. Use when asked how many nodes/replicas/connections to run, how big to size an instance or pool, or when capacity runs out. Do NOT use for SLO alert thresholds, autoscaling policy syntax, or writing an incident postmortem." metadata: decimalai-skill-id: "64a50256-65a2-41c1-b3fb-ff2eb5a4f9ed" decimalai-version-id: "e7fda563-da07-4826-8075-126086bf6755" decimalai-version: "1" decimalai-body-sha256: "8fb23ab02e19dcb204ce11d97ae43a5fa914cd453a48358ae42bcb16d0f4a6e8" decimalai-name: "capacity-headroom-planning" decimalai-format: "1" decimalai-invocation: "model" decimalai-category: "devops" decimalai-source: "https://app.decimal.ai/s/capacity-headroom-planning@1/SKILL.md" --- # Capacity Headroom Planning Turns a load number into a resource size — how many nodes, how big an instance, how many pool connections or workers — with enough slack to survive bursts, a node failure, and near-term growth. Asked "we peak at P and each node does C, how many nodes?", the base model defaults to `ceil(P / C)` and stops. That answer provisions for the current peak with **zero headroom**: it runs every node at 100%, has no spare for a single failure, ignores that the measured number may be an average rather than the peak, and carries no runway for growth. A resource at 100% utilization has explosive latency (queueing delay grows without bound as utilization approaches 1) and dies the first time one node drops. Replace that default with the four steps below. ## When to Activate Activate when the user wants to: - Decide how many nodes / replicas / servers / workers to run for a given load - Size an instance, a database connection pool, a cache, a queue, or storage from a throughput or volume number - Set an autoscaling floor (minimum replicas) or answer "will we run out, and when?" - Review a proposed sizing that looks like it was set to the current peak Do **not** activate for choosing SLO or burn-rate alert thresholds (`slo-alert-tuning`), for the exact syntax of an autoscaling policy, or for writing an incident postmortem. ## Step 1 — Size to the peak, and derive the peak from a peak-to-average ratio Never size to the average. Load is spiky, and the number someone hands you is often an average ("we do about 2000 requests a second"). Sizing to an average under-provisions by the whole spikiness factor. The **peak-to-average ratio** (P2A) is `peak_load / average_load`. Get it from the data — measure the busiest minute against the daily mean; typical web traffic runs 2–5×, batch and event workloads far higher. Then: ``` peak_load = average_load × P2A ``` Do every later calculation on `peak_load`. If only an average is available and P2A is unknown, say so and use a measured or conservative ratio rather than sizing to the average silently. Also size to the peak of the **right window**: a daily peak misses a Black-Friday or launch spike — plan against the largest peak the resource must survive, not the median day. ## Step 2 — Divide by a utilization ceiling, not by 100% Pick a target steady-state **utilization ceiling** `u` below 1.0 and size so peak load lands at the ceiling, not at the redline: ``` raw_capacity_needed = peak_load / u ``` The ceiling is the slack that absorbs short bursts, garbage-collection pauses, a rolling deploy taking a node out, and plain measurement error. Common defaults: **~65–70%** for CPU-bound throughput, **lower** (50% or under) for latency-sensitive services because queueing latency climbs steeply well before 100%, and higher only for smooth, batch-like load. A 0.65 ceiling means you buy `1/0.65 ≈ 1.54×` the raw peak — that multiplier is the point, not waste. ## Step 3 — Add failover headroom: N+k means a node can die Redundancy is a capacity requirement, not a checkbox. **N+1** means the cluster must still serve peak load with **one node gone**; N+2 tolerates two. When a node dies its share redistributes onto the survivors, so the survivors need room to absorb it. Size the surviving set to the ceiling, then add the spare: ``` N = ceil( peak_load / (per_node_capacity × u) ) # nodes to serve peak at the ceiling provision N + k # k spares for N+k redundancy ``` By construction, when `k` nodes fail the remaining `N` still serve peak within the ceiling. The consequence the naive answer misses: **at small cluster sizes, failover dominates.** With **2 nodes, N+1**, if one dies the other inherits 100% of the load — so each node's steady-state utilization can be at most **50%**. In general the provisioned steady-state utilization is `u·N/(N+1)`, and the spare fraction you carry is `1/(N+1)`: large for small clusters, negligible for big ones. "Just add one node" is correct in node count, but the effective steady-state ceiling drops the smaller the cluster. ## Step 4 — Project time-to-saturation from the growth trend, and scale ahead of it Today's size is a snapshot; load grows. Extrapolate the trend to the date load crosses the ceiling, and provision the next step **before** that date, allowing for procurement / deploy lead time. - **Linear growth** (constant absolute add per period): `periods_to_ceiling = (ceiling_load − current_load) / growth_per_period`. - **Compounding growth** (constant %): `periods_to_ceiling = ln(ceiling_load / current_load) / ln(1 + g)`. Example: at 65% of the ceiling growing 10%/month, `ln(1/0.65)/ln(1.10) ≈ 4.5 months` of runway. Report the **saturation date or runway**, not just "we have 35% headroom left" — a percentage hides how fast it is disappearing. Then work backward from that date by the lead time to know when to order/deploy. ## Also size the non-hardware ceilings Physical capacity is not the only wall: - **Provider quotas** (e.g. an Azure/AWS vCPU or instance quota, an API rate cap, a license seat count) can stop you scaling before hardware saturates. Treat each quota as its own ceiling, run the same time-to-saturation projection against it, and request the increase ahead of the date — quota bumps have their own lead time. - **Vertical vs horizontal**: a bigger node buys headroom on one axis but not failover (one big node is still a single failure). More nodes buy failover and quota-friendlier steps. Say which axis the headroom is coming from. ## Reviewing a proposed sizing When handed an existing plan, check it against the four steps and name the specific gap: - **Sized to `peak / capacity` exactly** → no utilization ceiling and no failover; it redlines and cannot lose a node. Divide by `u` and add N+k. - **Sized to the average** → under-provisioned by the P2A factor. Re-derive from the peak. - **N+1 claimed but nodes run near the ceiling in steady state** → a single failure pushes survivors over 100%. Lower the steady-state ceiling to `u·N/(N+1)`. - **Only a current-headroom percentage, no date** → no growth runway. Project time-to-saturation and set the trigger date. - **Ignores provider quota** → will hit the quota wall before the hardware wall. Add the quota as a ceiling and request ahead.