Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUDA, Triton, and CuteDSL, including CUPTI timing, correctness checks, generated-code inspection, multi-GPU measurements, and SOL sanity checks.
SkillScore73Maintained
Automatic
◆AI Engineer pack
Last verified 9/27/2026
What it changes
Passed 16 of 22 test cases with the skill, and 9 of 22 without it.
Without skill41%
With skill73%
Checks it passed that the model alone missed
Directs allocating symmetric memory buffers with identical shapes on all ranks.
Recommends executing uncaptured warmup iterations prior to capturing CUDA graphs across ranks.
and 6 more in the full report
7 fixed9 passed either way6 still failing0 broke
Pass-rate lift
+32 pp
Tokens
+17%
Agent turns
0%
Response time
+13%
Model gemini-3.6-flashMeasured 9/27/2026v216 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 7
Prompt
What Python timing module and benchmarking function should we import as the default standard for measuring vLLM single-GPU attention and GEMM kernel latencies with CUDA graph support and L2 cache flushing?
What was checked, without and with the skill
Checked
Without skill
With skill
Recommends using bench_gpu_time_with_cupti from flashinfer.testing as the default timing framework.
Missed
Passed
case-22 · gemini-3.6-flash · 9/27/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md106 linessingle file
Kernel Microbenchmark
Workflow
Create an isolated repro or benchmark when the existing harness is noisy.
Check correctness before timing. Keep tolerances explicit.
Time only the operation under study. Exclude allocation, compilation, random
input generation, logging, and host-device transfers unless those are the
target.
Compare against a baseline and report enough metadata to reproduce the
result: GPU, dtype, shape, command, branch/commit, and relevant env vars.
Treat explanations as hypotheses until backed by an artifact: ablation,
generated PTX/SASS, profiler output, or controlled benchmark.
If the result changes the conclusion, preserve the compact lesson in a note,
comment, benchmark table, or final summary. For experiments, a short
Question / Change / Correctness / Result / Observation / Next note is
usually enough.
Benchmark Defaults
Use FlashInfer CUPTI timing by default, with CUDA graph and cold L2 cache:
from flashinfer.testing import bench_gpu_time_with_cupti.
For compute-heavy kernels, report TFLOPS with the FLOP formula in the
benchmark. For memory-heavy kernels, report estimated bytes moved and GB/s.
For mixed kernels, report the most honest metric available and call out the
caveats. TFLOPS and memory bandwidth should be computed from the theoretical
best for the operation, not from a particular kernel implementation. For
example, memory bandwidth should assume all data is read exactly once from
global memory.
When comparing across shapes, prefer throughput metrics such as TFLOPS or
GB/s as the primary table columns; keep latency for absolute cost.
If a result exceeds expected peak/SOL, first inspect units, FLOP/byte
formulas, skipped work, sparsity, caching, and whether the baseline is doing
the same operation.
Force compilation/autotuning before measuring compiled kernels.
Seed inputs when correctness comparisons matter.
Keep metadata setup, plan construction, allocation, random input generation,
and logging outside the timed region unless that overhead is the experiment.