Jetson Speculative Decoding (vLLM)

Jetson Speculative Decoding (vLLM)

Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is dominated by VRAM headroom, not by the draft quality. This skill encodes the parts an LLM won't already know.

Purpose

Tune an existing Jetson vLLM deployment for faster token generation by appending the right --speculative-config and validating whether it improves single-stream decode speed.

When to use

• TPOT/ITL is the bottleneck (TTFT is fine, output is just slow).
• Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.
• Jetson family is Thor or AGX Orin. Do not suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.

When NOT to use

• High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.
• Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix".
• After applying jetson-inference-mem-tune flags that already pushed --gpu-memory-utilization near the ceiling. Free at least ~2 GB first.

Prerequisites

• A working vLLM server recipe from jetson-llm-serve.
• Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.
• A benchmark baseline from jetson-llm-benchmark before enabling speculation.
• A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.

Instructions

Append --speculative-config to the vllm serve command shown in jetson-llm-serve.

EAGLE-3 (preferred when a head is published for the target model):

--speculative-config '{
  "method": "eagle3",
  "model": "<eagle3-head-repo-id>",
  "num_speculative_tokens": 5,
  "draft_tensor_parallel_size": 1
}'