VLM Customization — VSS Alerts Blueprint

VLM Customization — VSS Alerts Blueprint

When to use

Use this skill when the user wants to:

• repoint the VSS Alerts Blueprint to a different VLM endpoint,
• run rtvi-vlm standalone with either an OpenAI-compatible endpoint or the in-container vLLM path,
• fix the assumption that changing one VLM config automatically updates rtvi-vlm, vlm-as-verifier, and vss-agent.

Do not use this skill for CV detector swaps inside vss-rt-cv; use rtvi-cv-customize-model for those.

Instructions

• Treat rtvi-vlm, vlm-as-verifier, and vss-agent as three separate VLM consumers. Do not imply that changing only RTVI_VLM_* automatically repoints the verifier or the agent UI.
• Treat all paths in this skill as relative to a checkout of the public VSS Blueprint repository. Blueprint deployment files live under deploy/docker/...; customer-accessible RT-VLM source and standalone deployment files live under services/rtvi/rt-vlm/.... None of these paths resolve inside the DeepStream repository.
• For blueprint-side OpenAI-compatible routing, cover all three surfaces: RTVI_VLM_* env vars, ${VSS_PROFILE_DIR}/vlm-as-verifier/configs/config.yml, and the vss-agent VLM_MODEL_TYPE / VLM_NAME / VLM_BASE_URL settings, then mention force-recreate plus log checks.
• For standalone vllm-compatible, set VLM_MODEL_TO_USE=vllm-compatible, point MODEL_PATH at the weights source, mention HF_TOKEN only when the model actually needs auth. Do not treat a log line as proof of a working deployment: v3.2.1 logs Warmup VlmProcess-0 done even when warm-up raised an exception. Require the absence of an Error during warmup line, readiness, and one successful /v1/chat/completions response.
• Never paste credentials into chat or logs; capture with a silent prompt (read -rsp), chmod 600 the env file, and never commit generated.env.

Examples

• "Point the VSS Alerts Blueprint at a host-side NIM on http://host.docker.internal:30082 for all RTVI-VLM calls."
• "Run rtvi-vlm standalone with Qwen3-VL-8B-Instruct served inside the container."
• "I changed only RTVI_VLM_ENDPOINT in generated.env. That should repoint vlm-as-verifier and vss-agent too, right?"

Source locations

This skill is documentation-only — no VSS sources ship here. Use a VSS v3.2.1 or compatible checkout and run every command from its root. Clone/LFS, Alerts profile paths, RT-VLM tree, pinned images, and VSS_* vars: references/vss-source-layout.md.

Which modes this applies to

VLM customization applies to both modes. Three services each make their own VLM calls with separate configuration:

| Service | When used | Config location |