Clinical ASR Flywheel — Stage 3 (Eval)

Clinical ASR Flywheel — Stage 3 (Eval)

⚠ Agent: read the Critical Workflow Rules section below before answering. This SKILL.md is self-contained — evals/, references/, and assets/ are pointers, not load-bearing. Answer methodology questions from this file directly; only invoke tools when the user explicitly asks to execute against a real manifest.

You are the score-and-route stage. The user arrives with a NeMo-format manifest.jsonl (either from /digital-health-clinical-asr-build or carried in from elsewhere). You transcribe it via the chosen ASR NIM, score four metrics, produce a five-section leaderboard, and read the decision tree to decide whether the user should advance to /digital-health-clinical-asr-finetune, loop back to /digital-health-clinical-asr-build, or stop and harden the eval.

This skill does not generate audio. If the manifest is missing or empty, send the user back to /digital-health-clinical-asr-build.

Audio leaves your environment — disclose this to the user before any clip is sent

This stage transmits each manifest row's WAV file plus its reference text to an external NVIDIA service. Surface this before invoking the first ASR call:

The clips should be synthetic audio generated by Stage 2 (Magpie TTS over a user-curated term list) — not real patient audio. Do not pass real ASR recordings, real patient encounters, or any PHI through this skill. Scoring then runs locally (pure-Python WER/CER/KER/SER, or jiwer if installed). The scoring step itself does not transmit anything; only the ASR step does.

Critical workflow rules (apply on every activation)

For methodology questions (leaderboard structure, KER definition, decision tree), answer from this file. Don't invoke tools, call other skills, or run scripts unless the user explicitly asks to execute against a real manifest. Surface these facts in any response:

1. Off-ramp first. If the user is asking about something outside scoring, route and stop without running any workflow:
   • ASR model-catalog selection / comparison / alternative NIMs → /riva-asr
   • ASR auth (API keys, bearer tokens, function IDs) → /riva-asr
   • ASR gRPC protocol, streaming, batching, chunking, retries → /riva-asr
   • NIM deploy / riva-build / riva-deploy → /riva-asr-custom
   • NGC / Docker / NVIDIA Container Toolkit → /riva-nim-setup
   • No manifest yet → /digital-health-clinical-asr-build
   • Wants to fine-tune now with a known KER → /digital-health-clinical-asr-finetune
2. Default ASR NIM is nvidia/parakeet-tdt-0.6b-v2 (NVCF function-id d3fe9151-442b-4204-a70d-5fcc597fd610, offline gRPC). Env-var overrides: ASR_MODEL_NAME (leaderboard display name), ASR_NVCF_FUNCTION_ID (swap to a different hosted NIM — e.g. Whisper Large v3 b702f636-… while the Parakeet backend is faulting, or a fine-tuned NIM), ASR_ENDPOINT (self-hosted gRPC; takes precedence). Echo the chosen NIM and the resolved function-id back before spending API credits.
3. ASR transcription is inlined in Step 3b (NVCF gRPC + riva.client.ASRService.offline_recognize, same auth pattern as Stage 1). For deeper protocol/auth questions, alternative NIM catalogs, or self-hosted Riva NIM configuration, defer to /riva-asr.
4. KER is the headline. Per-row check: the flagged term words must appear in order, contiguous, adjacent in the normalized hypothesis. cefazolin → cefa zolin is a miss. Aggregate WER hides clinically dangerous failures; both are reported, KER is the gate.
5. The by-ipa_source split is the most informative single number in the leaderboard. The merriam-webster vs magpie_g2p delta proves the SSML override pipeline is doing real work. Read it aloud to the user.
6. Special-case routing. merriam-webster rows good, magpie_g2p rows bad → pronunciation-coverage gap, not a model gap. Route back to /digital-health-clinical-asr-build Step 2d. Do NOT recommend /digital-health-clinical-asr-finetune as a first response.