Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Jetson Inference Memory Tuning · nvidia | DecimalAI

ai mlby nvidia

Jetson Inference Memory Tuning

nvidia/jetson-inference-mem-tune

Source nvidia/skills ↗License Apache-2.0

Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson.

SkillScore77Maintained

Automatic

AI Engineer pack

Last verified 8/12/2026

What it changes

Passed 15 of 20 test cases with the skill, and 3 of 20 without it.

Without skill14%

With skill77%

Checks it passed that the model alone missed

States that Jetson Thor support requires vLLM 0.20 or newer
Includes --target-mb 5000 parameter in the recommendation command
Recommends upstream vllm/vllm-openai container image for Jetson Thor

and 17 more in the full report

14 fixed3 passed either way5 still failing0 broke

Pass-rate lift: +64 pp
Tokens: +98%
Agent turns: 0%
Response time: -43%

Model gemini-3.6-flashMeasured 8/12/2026v117 of 22 cases passed with skill · 20 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/nvidia-jetson-inference-mem-tune/SKILL.md https://app.decimal.ai/s/nvidia-jetson-inference-mem-tune/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add nvidia-jetson-inference-mem-tune, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/nvidia-jetson-inference-mem-tune/ and pins it in .decimal/skills.lock

decimalai skills add nvidia-jetson-inference-mem-tune

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/nvidia-jetson-inference-mem-tune/SKILL.md https://app.decimal.ai/s/nvidia-jetson-inference-mem-tune/SKILL.md

Raw file: /s/nvidia-jetson-inference-mem-tune/SKILL.md ↗ · machine-readable summary at /s/nvidia-jetson-inference-mem-tune.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 13 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 13

Prompt

When invoking the recommendation script with a corrupted JSON file at /tmp/corrupt_audit.json, the process returns exit code 2. What caused this error status, and what remediation step should be requested from the operator?

What was checked, without and with the skill
Checked	Without skill	With skill
Identifies exit code 2 as an unreadable, unparseable, or invalid numeric audit snapshot	Passed	Passed
Advises re-running jetson-memory-audit/scripts/audit.sh to capture a fresh baseline	Missed	Passed

case-10 · gemini-3.6-flash · 8/12/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md182 lines2 filesincludes scripts

Jetson Inference Memory Tuning

Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU/variant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.

Purpose

Turn a live jetson-memory-audit snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.

When to use

"Which serving stack should I use on Orin Nano 8 GB to run a 7B model?"
"vLLM is OOMing — what should --gpu-memory-utilization and --max-model-len be?"
"Same model, less memory — can I switch from vLLM to llama.cpp?"
After jetson-memory-audit shows a model server is the top NvMap / PSS consumer.

Prerequisites

Start with a current jetson-memory-audit/scripts/audit.sh JSON snapshot from the target Jetson.
Know the intended workload: llm-server, vlm-server, embedding, or rag.
If the user gives a desired free-memory target, pass it as --target-mb; otherwise let the script use SKU defaults.

Available Scripts

Script	Purpose	Arguments
`scripts/recommend.py`	Reads an audit JSON and emits runtime plus launch-flag recommendations.	`--audit PATH`, `--runtime`, `--workload`, `--target-mb`, `--human`.

If your agent runtime supports run_script, invoke run_script("scripts/recommend.py", ["--audit", "/tmp/audit.json", "--runtime", "auto", "--workload", "llm-server"]) and summarize the returned JSON. Otherwise run it with python3 from the repository root.

Instructions

Run jetson-memory-audit/scripts/audit.sh to capture the device baseline.
Run scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server --target-mb 6000 to get a JSON of runtime + flag recommendations.
The agent presents the suggested runtime and the exact CLI flags. The user (or an outer agent) launches / restarts the server with those flags.
Re-run the audit to verify.

Expected workflow

Use scripts/recommend.py for the specific prompt and answer from the JSON it emits. If direct execution is blocked, run it as python3 {baseDir}/scripts/recommend.py ....

Track record

Benchmark

15 / 20

test cases passed, vs 3 / 20 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

nvidia / skills

skills/jetson-inference-mem-tune/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1Bundled 1 file · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Jetson Inference Memory Tuning

Purpose

When to use

Prerequisites

Available Scripts

Instructions

Expected workflow

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills