---
name: jeremylongshore/coreweave-incident-runbook
source: https://app.decimal.ai/s/jeremylongshore-coreweave-incident-runbook@1/SKILL.md
source_sha256: b4fb3d763d97
---

# CoreWeave Incident Runbook

> **Community-contributed.** Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

## Triage Steps

```bash
# 1. Check pod status
kubectl get pods -l app=inference -o wide

# 2. Check recent events
kubectl get events --sort-by=.lastTimestamp | tail -20

# 3. Check node status
kubectl get nodes -l gpu.nvidia.com/class -o wide

# 4. Check GPU health
kubectl exec -it $(kubectl get pod -l app=inference -o name | head -1) -- nvidia-smi
```

## Common Incidents

### Inference Service Down

1. Check pod status and events
2. If OOMKilled: reduce batch size or upgrade GPU
3. If ImagePullBackOff: check registry credentials
4. If Pending: check GPU quota and availability

### GPU Node Failure

1. Pods will be rescheduled automatically
2. If no capacity: scale down non-critical workloads
3. Contact CoreWeave support for extended outages

### Model Loading Failure

1. Check HuggingFace token secret exists
2. Verify model name spelling
3. Check PVC has sufficient storage
4. Review container logs for download errors

## Rollback

```bash
kubectl rollout undo deployment/inference
```

## Resources

- CoreWeave Support
- [CoreWeave Status](https://status.coreweave.com)

## Next Steps

For data handling, see `coreweave-data-handling`.