CLI Testing — Exercise the Assistant End-to-End

CLI Testing — Exercise the Assistant End-to-End

Drive a real assistant from the terminal only. The vellum CLI (cli/, package @vellumai/cli) manages instance lifecycle; vellum message / vellum events exercise a running instance. See cli/AGENTS.md and the root README.md § CLI for command reference.

0. Prerequisites

export PATH="$HOME/.bun/bin:$PATH"   # bun + the linked `vellum` binary
vellum ps                            # sanity check the CLI resolves

If vellum is missing, run ./setup.sh from the repo root once (installs deps, links the vellum command). Docker must be running for the default flow below.

1. Provide an LLM provider key (from the environment)

Local-mode and Docker-mode instances need one LLM provider key. The CLI reads it straight from the host environment — just export it before hatching/setup:

export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY / GEMINI_API_KEY /
                                      # FIREWORKS_API_KEY / OPENROUTER_API_KEY /
                                      # MINIMAX_API_KEY

In Devin sessions ANTHROPIC_API_KEY is typically already present in the environment — check with echo "${ANTHROPIC_API_KEY:0:7}" before asking for one. The CLI maps providers to env vars in cli/src/shared/provider-env-vars.ts.

2. Hatch — default to a Docker hatch built from source

Always default to --remote docker. It runs the assistant, gateway, and credential-executor in isolated containers that mirror production and keep the test off your host process table. Reserve --remote local (§5) for the rare case where Docker is unavailable.