i4h Workflow — Annotate Dataset

i4h Workflow — Annotate Dataset

Purpose

Use a VLM to verify whether each episode satisfies the env's task description. Use when the user asks to annotate, label episodes, filter demos, or gate finetuning on a success classifier.

Base Code

These steps drive the i4h-workflows base code (the workflows/agentic/ tree). To reuse an existing checkout, set I4H_WORKFLOWS to its path (no clone happens). Otherwise this resolves the current repo, or clones to ~/i4h-workflows — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• Annotation is optional. Do not run it during validation unless the user requests labels.
• For natural-language prompts such as "Run Annotation on all recorded episodes", annotate all episodes in one selected HDF5 recording, not every historical HDF5 under workflows/agentic/runs/. If the user does not name an HDF5, choose the latest annotatable recording: first look inside runs/.latest when it contains an HDF5, otherwise pick the newest non-annotation .hdf5 under workflows/agentic/runs/. Only batch across multiple HDF5 files when the user explicitly asks for all historical recordings, every HDF5 file, or a batch annotation run.
• Env config (source of truth): the annotator reads the success criterion (policy.task_description) from workflows/agentic/config/environments/<env>.yaml. Pass --task-description to override.
• Talks to an OpenAI-compatible endpoint via --base-url (default http://localhost:8000/v1) and --model (default Qwen/Qwen3-VL-8B-Instruct). Point both at a running vision-model server. Do not use text-only/code models such as qwen3-coder-next; offline annotation sends image inputs and requires a VLM.
• Keep every annotation artifact inside workflows/agentic/runs/<run>/. Do not create or access /tmp/annotate_* or other external temp directories.

Start VLM

Skip this section if an OpenAI-compatible endpoint serving a vision model is already running — just set VLM_BASE_URL/VLM_MODEL in Run to point at it. A local-agent server running qwen3-coder-next does not qualify because it is text-only. annotator/vllm.sh defaults to port 8000, so starting it on top of an existing server collides; don't.

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

For readiness, use workflows/agentic/annotator/vllm.sh ensure. Do not replace it with raw docker ps, fixed sleeps, ad hoc model-listing HTTP probes, or separate manual wait steps; the helper owns the start-and-wait policy.

Step 1 — start VLM (if needed)

Run this exact command. Do not add sleep, status, curl, or shell control operators around it: