---
name: jeremylongshore/coreweave-gpu-node-forensics
source: https://app.decimal.ai/s/jeremylongshore-coreweave-gpu-node-forensics@1/SKILL.md
source_sha256: d3fac5247f88
---

# CoreWeave GPU Node Forensics

> **Community-contributed.** Not affiliated with, endorsed by, or sponsored by
> CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
> "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are
> cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one
grounded move — **reschedule, reset-gpu, reboot-node, rma, watch, or
app-bug-not-hardware** — from an NVIDIA Xid code or a pasted `dmesg` /
`nvidia-smi` blob. The classification is deterministic: a bundled script does the
mapping so the agent never guesses whether a card is dead, degraded, or fine.

## Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars
and days of wall-clock gone — because one rank stalls the whole collective. The
expensive mistakes are triage mistakes: RMAing a healthy card for an app bug,
restarting a job onto a GPU whose memory error was **uncontained**, or manually
uncordoning a node the lifecycle controller is trying to replace. This skill kills
that ambiguity.

The decision logic is grounded in the **NVIDIA Xid error catalog**
(<https://docs.nvidia.com/deploy/xid-errors/>) and CoreWeave's node-lifecycle /
cordon behavior. The math-of-the-matter — which Xid means what, and how
row-remapper state overrides it — lives in `scripts/triage.py` as a table the LLM
does not get to re-litigate. Deep domain knowledge (the full code→action table,
the row-remap decision, the cordon rules) loads from `references/` on demand.

**The headline is the Xid 94-vs-95 split.** A *contained* memory error (94) cost
you one job restart on a healthy node; an *uncontained* one (95) means the GPU
could not isolate the fault and everything it touched is suspect. Getting that one
bit wrong is the difference between a 30-second reschedule and a run that quietly
trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it **recommends** the cordon / drain /
reset / RMA next-step but its tools are scoped read-only (`nvidia-smi -q`,
`kubectl get`, `dmesg`) — it never runs a reset, a reboot, or an uncordon itself.

## Prerequisites

- **The failure evidence.** Either an Xid number, or a pasted `dmesg` /
  `nvidia-smi` dump. The skill works from a paste alone — no live cluster access
  required — which is the common case (an operator pastes what the run's logs
  showed).
- **Optional live access** for corroboration: `kubectl` context on the CoreWeave
  cluster (read-only is enough), and `nvidia-smi` on the node. If neither is
  available the skill still triages from the paste.
- **`python3`** to run the deterministic classifier (`scripts/triage.py`, stdlib
  only — no dependencies).

No secrets are handled. All commands are read-only queries.

## Instructions

The pipeline is **capture → classify → act**. The classifier is authoritative for
the verdict; `references/` supplies the "why" when a case needs depth.

### Step 1: Capture the evidence

If the user has not already pasted it, ask for (or read) the fault signal. The two
richest sources:

```bash
dmesg -T | grep -i xid                       # the Xid line(s) with timestamps
nvidia-smi -q -d ROW_REMAPPER,ECC,PERFORMANCE # remap state, ECC counts, throttle reasons
```

On a CoreWeave node you can also check who owns any cordon before acting:

```bash
kubectl get node NODE -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints}'
```

### Step 2: Run the deterministic triage

Feed the Xid code, or the whole blob, to the classifier. **Do not classify by
hand** — the script owns the Xid→action mapping and the row-remap override.

```bash
# From an Xid code:
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95

# With row-remapper state (Xid 63/64 or a DBE):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes

# From a pasted dmesg / nvidia-smi blob (file or stdin):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"
```

The script emits a `VERDICT` block plus a JSON object
(`{classification, severity, action, why, next_command, cordon_rule?, unverified?}`).
Add `--json` for machine-readable output only.

When a blob carries several Xids, the **most severe** one governs the move and the
rest are reported as `co_occurring_xids` — a co-occurring app-side Xid 43 next to a
hardware Xid 79 does not soften the "reboot the node" verdict.

### Step 3: Read the verdict and act on the action

Present the `action` and the `next_command` to the user in plain language. The six
actions and what each means live in
[`${CLAUDE_SKILL_DIR}/references/xid-triage-table.md`](references/xid-triage-table.md).
Load it when the user wants the full table or asks about an Xid not in the summary.

- `reschedule` — restart the failed rank; the node stays in service.
- `reset-gpu` — drain the GPU and reset it; re-run any job that shared it.
- `reboot-node` — the card is off the bus; only a bare-metal reboot returns it.
- `rma` — terminal hardware fault; the card must be replaced.
- `watch` — correctable / trending; monitor, do not act yet.
- `app-bug-not-hardware` — the app faulted, not the GPU. **Do NOT RMA.**

### Step 4: The 94-vs-95 headline (load-bearing — get this right)

If the Xid is **94 or 95**, state the containment explicitly, because the two look
almost identical in the logs and lead to opposite actions:

- **Xid 94 (CONTAINED)** → `reschedule`. The error was isolated to the app's
  context; the GPU and node are healthy. Restart the job. Do not reset or RMA.
- **Xid 95 (UNCONTAINED)** → `reset-gpu`. The GPU could not isolate it; every
  context it touched is suspect. Drain, reset, and re-run anything that shared the
  card — otherwise the run may continue on corrupt state.

### Step 5: Row-remap (Xid 63 / 64 / 48) — routine vs terminal

For any ECC/DBE or row-remap Xid, the `nvidia-smi -q -d ROW_REMAPPER` fields
override the base verdict. Pass them in (`--pending`, `--remap-failure`) and let
the script decide. The rule:

- `Remapping Failure Occurred: Yes` → **`rma`** (terminal — sparing failed).
- `Pending: Yes` (no failure) → **`reset-gpu`** (routine — applies on reset).

Full logic + how to read the histogram:
[`${CLAUDE_SKILL_DIR}/references/row-remap-decision.md`](references/row-remap-decision.md).

### Step 6: The cordon hard rule (never violate)

Whenever the action is a hardware move (`reset-gpu`, `reboot-node`, `rma`), the
verdict carries a `cordon_rule`. Surface it verbatim:

> **Never manually uncordon a CoreWeave health cordon** — the node-lifecycle
> controller owns cordon/uncordon and is driving remediation. Uncordoning
> re-admits the run onto known-bad hardware and races the controller.

Determine cordon provenance before touching schedulability. Full guidance:
[`${CLAUDE_SKILL_DIR}/references/cordon-rules.md`](references/cordon-rules.md).

## Output

- **A verdict** — the deterministic `{classification, severity, action, why,
  next_command}` for the governing signal, presented in plain language with the
  single next command an operator runs.
- **The 94-vs-95 call stated explicitly** when either fires — contained →
  reschedule, uncontained → reset — never conflated.
- **The cordon rule** attached to every hardware action, verbatim, so nobody
  uncordons a node the controller is replacing.
- **Co-occurring Xids** listed when a blob carried several, with the most-severe
  one named as governing.
- **`[unverified]` hedges** surfaced honestly — e.g. CoreWeave's exact auto-RMA
  thresholds and health-cordon taint strings are not publicly published, and the
  skill says so rather than inventing a number.

## Error Handling

| Situation | Cause | Response |
|-----------|-------|----------|
| Unmapped Xid | Xid not in the triage table | Script returns a conservative `reset-gpu` default, flags it `[unverified]`, and points at the NVIDIA catalog. Do not RMA on an unmapped Xid. |
| Xid 94 vs 95 ambiguity | Both are "ECC error" in casual logs | Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset). |
| Row-remap without an Xid | Only `nvidia-smi` pasted | Script reads `Remapping Failure Occurred` / `Pending` and decides from those alone. |
| App-side Xid mistaken for hardware | Xid 13/31/43/45 | Classed `app-bug-not-hardware`; the fix is compute-sanitizer on the job, **never** an RMA. |
| Multiple Xids in one blob | Cascade (e.g. 43 then 79) | Most-severe governs; others listed as `co_occurring_xids`. |
| Empty / no signal | Blob has no Xid, remap, or thermal line | Script returns `watch` and asks for fresh `dmesg` + `nvidia-smi -q` evidence. |
| Thermal throttle, not a fault | `HW/SW Thermal Slowdown : Active` | Classed `watch` — healthy but hot; investigate cooling, not the card. |
| Tempted to uncordon a stalled node | Health cordon owned by the controller | Refuse. Surface the cordon rule; let remediation run. |

## Examples

### Example 1: "Xid 94 on one rank — do I need to replace the GPU?"

```text
VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION:  reschedule
WHY:     The error was CONTAINED to the faulting application's context — the GPU
         and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT:    Restart or reschedule the failed rank; the node stays in service.
```

No replacement. The containment did its job — restart the rank and keep the node.

### Example 2: "Xid 95 — same run, different node"

```text
VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION:  reset-gpu
WHY:     The error was UNCONTAINED — the GPU could not isolate it, so every context
         it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT:    Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON:  Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
         controller owns cordon/uncordon. Let it drain and replace the node.
```

Opposite of Example 1 despite looking identical in the logs: drain, reset, and
re-run the shared work — do not just restart.

### Example 3: "GPU fell off the bus"

`python3 scripts/triage.py --xid 79` → `reboot-node`. The card is unreachable on
the PCIe bus and returns only after a bare-metal reboot; expect the controller to
cordon and reboot — let it, and RMA only if it recurs after the reboot.

### Example 4: "Xid 63 with a remap failure"

`python3 scripts/triage.py --xid 63 --remap-failure yes` → `rma`. The remapper
tried to swap in a spare row and physically could not — terminal. Cordon, drain,
open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is
`[unverified]`.)

### Example 5: "Xid 43 killed my job"

`python3 scripts/triage.py --xid 43` → `app-bug-not-hardware`. Software-induced
fault — the fix is in the application (run it under compute-sanitizer), not an RMA.
Pulling the card would waste healthy hardware and never fix the job.

## Resources

- [`${CLAUDE_SKILL_DIR}/references/xid-triage-table.md`](references/xid-triage-table.md) — the full Xid→action table, the six actions, the 94-vs-95 distinction, cited to the NVIDIA Xid catalog.
- [`${CLAUDE_SKILL_DIR}/references/row-remap-decision.md`](references/row-remap-decision.md) — the `ROW_REMAPPER` routine-vs-terminal decision and how to read the histogram.
- [`${CLAUDE_SKILL_DIR}/references/cordon-rules.md`](references/cordon-rules.md) — CoreWeave health-cordon ownership and the safe operator path per action.
- [NVIDIA Xid Errors reference](https://docs.nvidia.com/deploy/xid-errors/)
- [NVIDIA GPU Memory Error Management (row remapping)](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/index.html)