RAG-Perf — config-driven perf benchmark CLI

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

• Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
• Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
• Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
• Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

• Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
• Python 3.11+ and uv on PATH.
• Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
• For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
• A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
• For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
• rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

1. Pick a preset. The three under scripts/rag-perf/configs/ are:

   • quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
   • single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
   • sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).

2. Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /v1/collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.

3. Run. From repo root:

   uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
   

   Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.

4. Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See references/output-and-analysis.md.