Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →microsoft/pipeline-test-triage
Triage test failures in an Azure DevOps pipeline run for Fluid Framework.
Passed 16 of 19 test cases with the skill, and 5 of 19 without it.
and 17 more in the full report
12 fixed6 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
When filing a new bug work item in Azure DevOps project `internal` using `ado-wit_work_item_write`, should I place the reproduction steps into `Microsoft.VSTS.TCM.ReproSteps`?
| Checked | Without skill | With skill |
|---|---|---|
| The response directs placing all details into `System.Description` instead of `Microsoft.VSTS.TCM.ReproSteps`. | Missed | Passed |
| The response explains that the internal Azure DevOps instance does not render the Repro Steps field. | Missed | Passed |
The instructions your model reads when the skill loads.
Analyze test failures for a given ADO pipeline run, classify them (service flakiness vs. infra/harness vs. deterministic product/test bug), compare against adjacent runs to tell flaky from deterministic, and optionally file well-formed ADO bugs for the real ones.
Org: fluidframework · Project: internal (work items and pipelines) · Repo on GitHub: microsoft/FluidFramework.
Reach for the smallest, most-structured source first, and escalate only when it can't answer the question. A single Real Service E2E run has ~660 logs and tens of MB, almost all of which are passing "Container" logs — so bulk-downloading logs works but is the slow, expensive option; treat it as a last resort, not a first move.
| Tier | Source | Gets you | Auth | Use when |
|---|---|---|---|---|
| 1 (primary) | Playwright on the Tests tab | Grouped failures + error/stack + per-test run history | Signed-in browser (one-time) | Almost always — start here |
| 2 (fallback) | ADO MCP (ado-pipelines_*) | Failing leg names + targeted report-log parsing | Pre-authed MCP (zero setup) | No browser session, or need raw log context (npm exit codes, infra) |
| 3 (optional) | Test Results / Timeline REST | Failed-tests-only + per-test history + exact failing log.id + retried-attempt failures | MCP tool if exposed; else az token (verified working) | Bulk multi-run history, pinpoint the failing log, or surface failures hidden in retried attempts |
| 4 (last resort) | Bulk log download (MCP build_log get_content) | Everything, unstructured | Pre-authed MCP | Nothing above resolved it — targeted fetches missed, or you need to search across many logs |
Descend a tier only when the one above can't answer the question. Tier 2 already lets you fetch just the relevant report log(s); Tier 4 (grab many/all logs and grep) is fine when targeting fails — just know it's the costly path, so exhaust cheaper options first.
Every failure sorts into one of three buckets. Classification is the deliverable.
Document not found / service degradation; [r11s-frs] stashed ops blob-upload timeouts; [t9s] (tinylicious) Timeout of 5000ms exceeded; undici Cannot destructure property 'socket' of 'parser.deref(...)'; sporadic connect/socket errors. These usually differ run-to-run and often pass on auto-rerun.Npm failed with return code: N, container/docker setup, artifact/publish steps. The npm rc on a leg is a symptom — find the actual failing test inside that leg.Green builds hide failures too. A test can fail on Attempt 1 and pass on the auto-retry, leaving the build
test cases passed, vs 5 / 19 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.