Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Guide incident response from detection to post-mortem using SRE principles, severity classification, on-call management, blameless culture, and communication protocols. Use when setting up incident processes, designing escalation policies, or conducting post-mortems.
.claude/skills/ancoleman-managing-incidents/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-03 | ✗→✓ | ▲ Improved | 164% | 0% |
| case-05 | ✗→✓ | ▲ Improved | 295% | 0% |
| case-06 | ✗→✓ | ▲ Improved | 256% | 0% |
| case-08 | ✗→✓ | ▲ Improved | 280% | 0% |
| case-21 | ✗→✓ | ▲ Improved | 114% | 0% |
Provide end-to-end incident management guidance covering detection, response, communication, and learning. Emphasizes SRE culture, blameless post-mortems, and structured processes for high-reliability operations.
Apply this skill when:
Declare Early and Often: Do not wait for certainty. Declaring an incident enables coordination, can be downgraded if needed, and prevents delayed response.
Mitigation First, Root Cause Later: Stop customer impact immediately (rollback, disable feature, failover). Debug and fix root cause after stability restored.
Blameless Culture: Assume good intentions. Focus on how systems failed, not who failed. Create psychological safety for honest learning.
Clear Command Structure: Assign Incident Commander (IC) to own coordination. IC delegates tasks but does not do hands-on debugging.
Communication is Critical: Internal coordination via dedicated channels, external transparency via status pages. Update stakeholders every 15-30 minutes during critical incidents.
Standard severity levels with response times:
SEV0 (P0) - Critical Outage:
SEV1 (P1) - Major Degradation:
SEV2 (P2) - Minor Issues:
SEV3 (P3) - Low Impact:
For detailed severity decision framework and interactive classifier, see references/severity-classification.md.
Incident Commander (IC):
Communications Lead:
Subject Matter Experts (SMEs):
Scribe:
Assign roles based on severity:
For detailed role responsibilities, see references/incident-roles.md.
Primary + Secondary:
Follow-the-Sun (24/7):
Tiered Escalation:
Standard incident lifecycle:
Detection → Triage → Declaration → Investigation
↓
Mitigation → Resolution → Monitoring → Closure
↓
Post-Mortem (within 48 hours)When to Declare: When in doubt, declare (can always downgrade severity)
When to Escalate:
When to Close:
For complete workflow details, see references/incident-workflow.md.
Incident Slack Channel:
#incident-YYYY-MM-DD-topic-descriptionWar Room: Video call for SEV0/SEV1 requiring real-time voice coordination
Status Update Cadence:
Status Page:
Customer Email:
Regulatory Notifications:
For communication templates, see examples/communication-templates.md.
Every runbook should include:
For runbook templates, see examples/runbooks/ directory.
Assume Good Intentions: Everyone made the best decision with information available.
Focus on Systems: Investigate how processes failed, not who failed.
Psychological Safety: Create environment where honesty is rewarded.
Learning Opportunity: Incidents are gifts of organizational knowledge.
1. Schedule Review (Within 48 Hours): While memory is fresh
2. Pre-Work: Reconstruct timeline, gather metrics/logs, draft document
3. Meeting Facilitation:
4. Post-Mortem Document:
5. Follow-Up: Track action items in sprint planning
For detailed facilitation guide and template, see references/blameless-postmortems.md and examples/postmortem-template.md.
Actionable Alerts Only:
Preventing Alert Fatigue:
PagerDuty:
Opsgenie:
incident.io:
For detailed tool comparison, see references/tool-comparison.md.
Statuspage.io: Most trusted, easy setup ($29-399/month) Instatus: Budget-friendly, modern design ($19-99/month)
MTTA (Mean Time To Acknowledge):
MTTR (Mean Time To Recovery):
MTBF (Mean Time Between Failures):
Incident Frequency:
Action Item Completion Rate:
Incident → Post-Mortem → Action Items → Prevention
↑ ↓
└──────────── Fewer Incidents ─────────────┘Is production completely down or critical data at risk?
├─ YES → SEV0
└─ NO → Is major functionality degraded?
├─ YES → Is there a workaround?
│ ├─ YES → SEV1
│ └─ NO → SEV0
└─ NO → Are customers impacted?
├─ YES → SEV2
└─ NO → SEV3Use interactive classifier: python scripts/classify-severity.py
For detailed escalation guidance, see references/escalation-matrix.md.
Prioritize Mitigation When:
Prioritize Root Cause When:
Default: Mitigation first (99% of cases)
Observability: Monitoring alerts trigger incidents → Use incident-management for response
Disaster Recovery: DR provides recovery procedures → Incident-management provides operational response
Security Incident Response: Similar process with added compliance/forensics
Infrastructure-as-Code: IaC enables fast recovery via automated rebuild
Performance Engineering: Performance incidents trigger response → Performance team investigates post-mitigation
Runbook Templates:
examples/runbooks/database-failover.mdexamples/runbooks/cache-invalidation.mdexamples/runbooks/ddos-mitigation.mdPost-Mortem Template:
examples/postmortem-template.md - Complete blameless post-mortem structureCommunication Templates:
examples/communication-templates.md - Status updates, customer emailsOn-Call Handoff:
examples/oncall-handoff-template.md - Weekly handoff formatIntegration Scripts:
examples/integrations/pagerduty-slack.pyexamples/integrations/statuspage-auto-update.pyexamples/integrations/postmortem-generator.pyInteractive Severity Classifier:
bashpython scripts/classify-severity.py
Asks questions to determine appropriate severity level based on impact and urgency.
Books:
Online Resources:
Standards:
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-01 | pass→pass | 14,599 | 11,100 | -24% | 1 | 1 | 0% | 2,205 | 4,977 | +126% | 0 | 0 | — |
case-02 | pass→pass | 14,831 | 12,335 | -17% | 1 | 1 | 0% | 2,198 | 5,234 | +138% | 0 | 0 | — |
case-03 | fail→pass | 10,530 | 6,408 | -39% | 1 | 1 | 0% | 1,656 | 4,372 | +164% | 0 | 0 | — |
case-04 | pass→fail | 8,057 | 5,614 | -30% | 1 | 1 | 0% | 1,230 | 4,217 | +243% | 0 | 0 | — |
case-05 | fail→pass | 17,047 | 3,938 | -77% | 1 | 1 | 0% | 981 | 3,872 | +295% | 0 | 0 | — |
case-06 | fail→pass | 7,273 | 3,824 | -47% | 1 | 1 | 0% | 1,100 | 3,918 | +256% | 0 | 0 | — |
case-07 | pass→pass | 6,327 | 4,242 | -33% | 1 | 1 | 0% | 1,019 | 4,028 | +295% | 0 | 0 | — |
case-08 | fail→pass | 5,724 | 2,911 | -49% | 1 | 1 | 0% | 1,007 | 3,825 | +280% | 0 | 0 | — |
case-09 | pass→pass | 14,580 | 11,547 | -21% | 1 | 1 | 0% | 2,306 | 5,190 | +125% | 0 | 0 | — |
case-10 | pass→pass | 7,255 | 8,316 | +15% | 1 | 1 | 0% | 1,068 | 4,492 | +321% | 0 | 0 | — |
case-11 | pass→pass | 12,640 | 8,498 | -33% | 1 | 1 | 0% | 2,013 | 4,763 | +137% | 0 | 0 | — |
case-12 | pass→pass | 13,687 | 10,330 | -25% | 1 | 1 | 0% | 2,023 | 4,928 | +144% | 0 | 0 | — |
case-13 | pass→pass | 6,806 | 5,828 | -14% | 1 | 1 | 0% | 1,052 | 4,179 | +297% | 0 | 0 | — |
case-14 | pass→pass | 18,021 | 13,315 | -26% | 1 | 1 | 0% | 2,655 | 5,347 | +101% | 0 | 0 | — |
case-15 | pass→pass | 8,296 | 2,482 | -70% | 1 | 1 | 0% | 1,275 | 3,735 | +193% | 0 | 0 | — |
case-16 | pass→pass | 10,835 | 2,463 | -77% | 1 | 1 | 0% | 1,649 | 3,679 | +123% | 0 | 0 | — |
case-17 | pass→pass | 15,099 | 3,580 | -76% | 1 | 1 | 0% | 2,418 | 3,912 | +62% | 0 | 0 | — |
case-18 | fail→fail | 10,472 | 6,593 | -37% | 1 | 1 | 0% | 1,618 | 4,393 | +172% | 0 | 0 | — |
case-19 | pass→pass | 12,756 | 10,438 | -18% | 1 | 1 | 0% | 2,086 | 5,043 | +142% | 0 | 0 | — |
case-20 | pass→pass | 9,498 | 7,208 | -24% | 1 | 1 | 0% | 1,394 | 4,323 | +210% | 0 | 0 | — |
case-21 | fail→pass | 11,421 | 2,503 | -78% | 1 | 1 | 0% | 1,742 | 3,728 | +114% | 0 | 0 | — |
case-22 | pass→pass | 8,357 | 7,957 | -5% | 1 | 1 | 0% | 1,559 | 4,961 | +218% | 0 | 0 | — |
case-23 | pass→pass | 8,054 | 10,019 | +24% | 1 | 1 | 0% | 1,568 | 5,275 | +236% | 0 | 0 | — |
case-24 | pass→pass | 10,778 | 15,701 | +46% | 1 | 1 | 0% | 2,275 | 5,808 | +155% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 24 cases were attempted, and 23 counted toward the lift figure. The other 1 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +17 percentage points is the difference between those two pass rates over the 23 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.