i4h Workflow — Validate

i4h Workflow — Validate

Purpose

Roll out a policy or scripted state-machine controller against an env and record verification episodes to an HDF5. Use when the user asks to validate, evaluate, run, or rollout a policy/checkpoint, or asks for surgical state-machine smoke runs.

Base Code

These steps drive the i4h-workflows base code (the workflows/agentic/ tree). To reuse an existing checkout, set I4H_WORKFLOWS to its path (no clone happens). Otherwise this resolves the current repo, or clones to ~/i4h-workflows — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• Env config (source of truth): workflows/agentic/config/environments/<env>.yaml — read it for the <env> defaults: policy.model_repo/model_revision, policy.task_description, policy.health_port, and arena.max_timesteps.
• Validation runs the policy daemon and Arena together; both processes are required.
• The policy daemon is headless. Arena opens the sim window by default; add --headless --enable_cameras --rendering_mode performance only when the user explicitly asks for headless/no-window execution.
• In Claude Code --print, Codex exec, or any other non-interactive/fresh session, policy evaluation must use Step 2A as one foreground bash command. Do not start the policy and Arena in separate tool calls, do not use Claude background tasks for eval, and do not return to the user until Arena exits and the policy cleanup has run.
• In Claude Code specifically, do not use the Bash tool's background mode for Evaluate ... prompts, do not launch a command ending in &, and do not say "the eval is running in the background." The answer is not complete until the HDF5/log summary has been inspected.
• README quick-run prompts that say "with the state machine" use Arena --state-machine and do not start a policy daemon.
• Do not run the VLM annotator unless the user asks for success labels.
• assemble_trocar is inference-only — validate its YAML default model or a compatible N1.5 checkpoint.

Inputs

• ENV_ID: env YAML id.
• EPISODES: 1 for sanity, more for real eval.
• MAX_TIMESTEPS: use the user-requested cap when the prompt gives one (for example, 300 timesteps -> MAX_TIMESTEPS=300); otherwise read arena.max_timesteps from the env YAML for normal evaluation. Use 200 only when the user explicitly asks for a smoke, sanity, or quick check.
• MODEL_PATH (optional): path to a checkpoint-NNNN/ directory containing model-0000{N}-of-*.safetensors, experiment_cfg/, and processor/. Omit to use YAML policy.model_repo.
• USE_LATEST_CHECKPOINT=1: set this when the prompt says "new checkpoint" or "latest checkpoint" and MODEL_PATH is not already known.
• STATE_MACHINE: true only when the prompt explicitly says state machine.