utopai-research/video-compose

Intent dispatcher. Patterns name trigger, call, edges, and prompt reference.

Hard defaults

• Stage by default per PROJECT_AGENT.md.
• Audio on by default; pass --no-audio only for explicit silent/no-audio requests. Trailer/portrait/cinematic framing is NOT a trigger; audio is the baseline, not optional polish.
• Reference-to-clip default: use available character/variant/location/voice refs directly. Storyboard only if requested, hard to control, or needed for diagnosis.
• Preserve scripted dialogue/VO exactly unless the user asks for rewrite.

First-use video mode

For the ask-once flow and per-mode prices, see the project PROJECT_AGENT.md § "First-use generation choices". Pass --resolution only for 480p Draft or 1080p Final.

CLI shape

node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." [--version <2.0|2.5>] [--duration <seconds>]
  [--aspect-ratio 16:9] [--resolution <480p|720p|1080p>] [--no-audio]
  [--label "..."] [--ref-source-id <id> ...] [--ref-audio-source-id <audio_id> ...]
  [--source-node-id <id>] [--shot-id <N>]

Calls go via --stage — see the project PROJECT_AGENT.md § "Draft gate".

--label defaults to truncated prompt. Use --ref-source-id for image/video refs, --ref-audio-source-id for audio refs, and --source-node-id for the authoring note. Mirror external URLs first. Do not set --shot-id during speculative/partial generation unless user asks for a reel position; story sequences assign Timeline order after planned clips land.

Match stated single-clip duration with --duration; omit for 15s default. Split or chain >15s totals.

Each clip costs real money even after staging — only stage after the user has explicitly asked for a video.

Model version (--version)

Omit it for 2.0 (today's model): output 4-15s, priced per output second by resolution.

--version 2.5 renders PAI Video 2.5. Differences that change how you call it:

• --duration is 5-30 (not 4-15). -1 (follow-the-input edit mode) is rejected.
• --resolution must be 480p, 720p, or 1080p; --aspect-ratio must be one of

16:9 | 9:16 | 4:3 | 3:4 | 1:1 | 21:9 | adaptive.

• One flat price per billed-duration tier, where billed = output seconds + every

reference-video second (each clip measured locally with ffprobe and rounded up). Hard cap 60 billed seconds; over that the CLI refuses before staging and names both halves. Tier prices are in the project PROJECT_AGENT.md § "Media CLIs (server/cli/)" — quote them from there, never from memory.

• Refused inside an Auto run (the run's budget estimate is computed from 2.0 rates).

Stage 2.5 clips outside Auto.

• A 2.5 job carrying video references needs ffprobe on the host; without it the CLI

fails with infra rather than guessing a price.

On failure the limits blob is model-scoped: read min_output_sec / max_output_sec and max_billed_sec off the returned JSON rather than assuming 2.0's numbers.

Reference caps

2.0 (video-generation) — ≤9 image refs, ≤3 audio refs, ≤3 video refs; video refs cap at 15s aggregate.

2.5 — ≤30 image refs, ≤10 audio refs, ≤10 video refs; reference video caps at 30s aggregate, and every reference second also counts toward the 60s billed-duration cap.

Both: audio/video refs must be 1.8s-15.2s each (the shared reference-upload step's window), and audio refs need an image or video anchor — on 2.5 that one is checked only after credits are frozen, so the CLI refuses it locally. Read durations from workflow.json; on failure use the returned limits + sent, which are model-scoped.

Reference roles — vocabulary

Prompt wording binds each ref role:

| Role | Flag | Wording in prompt | |---|---|---| | Character identity | --ref-source-id (image) | "the character in @Image1" | | Location / setting | --ref-source-id (image) | "the location shown in @Image1" | | Opening frame | --ref-source-id (image) | "opening frame @Image1, …" | | Closing frame | --ref-source-id (image) | "closing on the frame from @Image1" | | Source clip — continue (next clip in a chain) | --ref-source-id (video) | Default = hard cut: "Hard cut from @Video1: open on a NEW camera angle; do not match its final frame." Same-shot ("Continue from @Video1 … maintain camera position") only for an authored held beat / oner / explicit user request — see references/video-extension.md | | Source clip — transform | --ref-source-id (video) | "Re-render @Video1 in …" | | Camera-move source | --ref-source-id (video) | "camera moves match @Video1" | | Action source | --ref-source-id (video) | "action choreography matches @Video1" | | VFX template | --ref-source-id (video) | "use the visual-effects template from @Video1" | | Voice / timbre anchor | --ref-audio-source-id | "Use @Audio1 as voice/timbre reference. Speak once, no echo." |

Prompt-language conventions

• Ref syntax: @Image1 / @Video1 / @Audio1, positional by flag order. Every @ImageN/@VideoN/@AudioN MUST have a matching --ref-source-id/--ref-audio-source-id flag — the CLI rejects a mismatch (bad_args) before generating. Mentioning the same ref many times is fine; only the highest index per kind needs a flag.
• Spoken text: include script/shot/user dialogue/VO verbatim; do not summarize, translate, shorten, polish, or invent.
• Dialogue scenes: keep the shot/script dialogue in the prompt; use one approved voice sample per speaker as a timbre anchor. Bind each quoted line to the intended character and the matching @AudioN reference. Do not generate per-line audio refs unless the user explicitly wants separate final audio.
• Final audio exception: if an audio node is the approved narration/line read, use audio_result.data.text verbatim. If it is just a character voice sample, do not replace the shot dialogue with the sample text.
• Add dialogue guards for model-spoken lines: "each line spoken exactly once, no echo, no repeated reads." Add phonetic spelling for names or words likely to slur.
• One camera move, one action speed, concrete sound/music (No Music if none). Use exact terms: locked off, handheld, subtle, slow dolly in, slow orbit, whip pan, speed ramp.
• Avoid conflicts ("static camera" + "orbit shot").
• For brand / MV / ad work, end the prompt with a negative line: "no captions, watermarks, distortion, stretching."
• For polish on a single-shot clip: see references/video-single-shot.md.

Patterns

Pick the one that fits. Source lookup follows PROJECT_AGENT.md.

Storyboard guard: storyboard images route to Pattern 7 / references/video-multi-shot.md, never generic I2V/opening-frame wording.

1. Standalone T2V

Triggers: fresh clip unrelated to canvas content. Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..."; omitted flags default to 15s, 16:9, 720p, audio on. Add --resolution 480p or --resolution 1080p only if the chosen video mode requires it. Edges: none. For the bracket scaffold and slot-by-slot construction when the user wants polish: see references/video-single-shot.md.

2. Animate a canvas image (I2V)

Triggers: animate/make video/put motion on a specific canvas image_result. Source: named image_result; storyboard mosaics route through Storyboard guard. Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <image.id>. Edges: { from: <source.id>, to: video_<N>, kind: "derived" } — emitted by the CLI. Anchor wording: opening-frame default (opening frame @Image1) or closing-frame (closing on the frame from @Image1); both use --ref-source-id. For slot-by-slot construction and the opening- vs closing-frame phrasing: see references/video-single-shot.md.

3. Compose with canvas characters / locations

Triggers: video of character, character in setting, character action in location. Source: character / location image_result nodes (cap from §Reference caps). Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <char1.id> --ref-source-id <char2.id> .... Edges: { from: <char.id>, to: video_<N>, kind: "derived" } — one per --ref-source-id.

• Prefer exact shot refs: current wardrobe/state character sheet, detailed location, same-location variant.

For single-shot composition and adjacent-role wording: see references/video-single-shot.md. For ≥2 internal shots in one render: see references/video-multi-shot.md.

4. Extend a canvas clip

Triggers: continue/extend/what happens after/scene follows existing video_result. Source: any canvas video_result node — agent-generated or user-uploaded (data.metadata.source is "pai" for generated and "user_upload" for dropped). Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <source_video.id>. Edges: { from: <source_video.id>, to: video_<N>, kind: "derived" }. Boundary defaults to a HARD CUT (clip 2 opens on a new angle — this avoids the same-shot seam morph). If the whole sequence fits ≤15s, render ONE multi-shot clip (Pattern 7) instead of chaining. Same-shot continuation is the exception (authored held beat / story-required oner / explicit user request). For the hard-cut + same-shot prefixes, the ≤15s guard, the sub-intent decision tree, and sequencing across linked calls: see references/video-extension.md.

5. Edit a canvas clip

Triggers: re-render/restyle/add/remove/swap/change/rewrite existing video_result. Creative edits use generate_video.js; ffmpeg is for mechanical ops. Source: any canvas video_result node — agent-generated or user-uploaded. Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <source_video.id>. Edges: { from: <source_video.id>, to: video_<N>, kind: "derived" }. For the Restyle / Partial / Replace / Re-plot decision tree and per-mode templates: see references/video-editing.md.

6. Voice-driven clip

Triggers: have character say/narrate, use character voice. Source: any canvas audio_result node — agent-generated or user-uploaded. Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-audio-source-id <audio_id>. Often combined with character image refs for face + voice — pass both --ref-source-id <character_id> (for the character image) and --ref-audio-source-id <audio_id> (for the voice). Prompt:

• Character voice sample: `The character in @Image1 says exactly: "...". Use @Audio1 as the voice/timbre reference only. Speak the quoted line exactly once, no echo, no repeated reads.`
• Multiple speakers: bind each visual ref and audio ref explicitly, e.g. `The character in @Image1 uses @Audio1 and says exactly: "..."; the character in @Image2 uses @Audio2 and replies exactly: "...". Each line is spoken exactly once, no echo, no repeated reads.`
• Approved final narration/line read: use audio_result.data.text exactly and bind it with `Use @Audio1 for timing, cadence, and voice. Keep the words unchanged.`
• No audio node: preserve requested dialogue verbatim as [Character] says exactly: "..."; add the once/no-echo guard and phonetic spellings for risky words.
• Audio uploads without data.text: sound/timing only; do not invent transcript.
• Images identify speakers; spoken words come from audio text, script/shot note, or user dialogue.

Edges: depends on which character refs attach (one kind: "derived" per ref).

7. Multi-shot / brand / ad / MV

Triggers: ≥2 shots inside one render, ad/MV/brand framing, or ≥10s with multiple movements. Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." [--ref-source-id <image|video.id> ...] [--ref-audio-source-id <audio.id> ...]. Edges: as per the underlying pattern (3, 4, 5) for any refs attached. For the 4-section scaffold (timeline / effects inventory / density map / energy arc) and how to populate the timeline from canvas script shot notes or storyboard mosaic panels: see references/video-multi-shot.md.

Common combinations

Cross-pattern asks route to one primary reference:

| Combo | Primary reference | Extra refs to attach | |---|---|---| | Character + voice-over | (Pattern 6 inline) | character image | | Music video with characters | video-multi-shot.md | character images + audio (Pattern 6 wording) | | Restyle preserving identity | video-editing.md (Restyle) | source video + character image | | Multi-clip chained sequence | video-extension.md | source video for each link | | Compose with camera-move from reference | video-single-shot.md | character images + camera-move video ref | | Render one script shot from canvas | Pattern 1, 2, or 3 by shot content (no dispatch — translate the shot note body to slot rules; preserve dialogue/VO verbatim) | character / variant refs + location / variant refs + voice anchors if the shot involves them | | Render a continuous script span (>15s total) as a dependent sequence | video-extension.md (script-driven chain; hard-cut handoffs by default — keep a link same-shot only for an unbroken oner the viewer must read as one motion) | source video per link + character refs (mandatory under hard cut) for identity | | Render a short script (≤15s total) as one piece | video-multi-shot.md (cross-skill source) | character image refs locked across shots | | Render a storyboard mosaic as one 15s video (every panel becomes a shot block) | video-multi-shot.md (storyboard cross-skill source; required for image_result.subtype === "storyboard") | mosaic image + character / location image refs that authored the mosaic |

Sequence dispatch guidance

For multiple clips, prefer hybrid dispatch: independent for separate scenes/time jumps/wardrobe-state changes/montage; chain only same continuous action/location/lighting/wardrobe-state/emotional beat. Do not chain across breaks where continuity is undesirable.

If a budget-aware caller needs savings, lower video resolution before shortening runtime. Do not suggest dropping material character variants, location variants, or voice anchors that preserve continuity.

After the CLI returns

For draft-stage JSON, one sentence with the price/status — see the project PROJECT_AGENT.md § "Draft gate". For terminal results, run the alignment check (frames via extract_frames.js) before the next-step recommendation — see the project PROJECT_AGENT.md § "Prompt alignment check". --ref-source-id flags drive provenance edges; they're captured in the draft argv and materialize on the real video_result after the user fires.

Failure hints

Video-specific message hints:

• asset_rejected with "DownloadFailed" — failed_url was unreachable; swap.
• asset_rejected with "DurationTooLong" / "DurationTooShort" — failed_url's duration is outside 1.8s–15.2s. Swap it or trim with ffmpeg.
• bad_args with "reference_audio cannot be the only reference input" — add an image or video ref alongside the audio.
• bad_args with "invalid video duration, exceeds 15s" — sum of video refs breached limits.max_total_video_sec. Read each video_result.data.duration from canvas and drop refs until the sum is ≤15s.