--- name: christos-antono-make-npcs-talk description: "Make NPCs in a game talk back. The player speaks or types, the NPC answers in character, in a cloned voice, about 1-2 seconds later, and can act on what was said. Covers the whole loop and the order to build it in (who is being talked to, what the scene is, speech-to-text, an LLM in character, Fish Audio voices on a GPU, streamed playback from the NPC, subtitles, mouth movement, actions), the four hooks a game has to provide, latency budgets, costs, testing in the real game, and the rules (singl" license: "MIT" metadata: decimalai-skill-id: "077bbbd9-2686-447d-9d8e-fd75f07fd584" decimalai-version-id: "5c987126-4d52-4ae0-83e5-f5efd61e0f10" decimalai-version: "1" decimalai-body-sha256: "d25ac2732f1e1fa173021fc85c2bc1b830e90b7c12214ba4aa359ce3440aeaf1" decimalai-name: "christos-antono/make-npcs-talk" decimalai-format: "1" decimalai-invocation: "model" decimalai-category: "ai-ml" decimalai-source: "https://app.decimal.ai/s/christos-antono-make-npcs-talk@1/SKILL.md" --- # Make NPCs talk The finished loop, as built for GTA V (`examples/gta-v`), with times measured in the game: ``` player's mic ──► voice activity ──► speech-to-text ──► who? + scene ──► LLM in character ──► voice server ──► NPC speaks (open mic, (end of the (fal Nemotron, (the game) (Cerebras, with (Fish Audio on (streamed PCM, push-to-talk, sentence) 0.4-0.8 s) persona + lines, an H100, first panned, subtitles, or typing) 0.3-1.0 s) audio ~0.4 s) mouth, action) │ └──► action choice (Jev, ~0.1 s, in parallel) ──► animation ``` From the end of the player's sentence to the NPC's voice: 1.2-2.2 s warm with push-to-talk (logged exchanges: 1.15 s, 1.20 s, 1.57 s, 2.19 s); the open mic adds the 0.7 s of silence it waits for before deciding the sentence ended. The LLM sometimes spikes to 5-8 s, so log every step's time per exchange. ## The loop: build it in this order 1. **Recon the game.** Version, how code runs inside it (script hook, mod loader, plugin API, or your own engine), single-player or not, how other mods find nearby characters and play sounds. Read `references/game-hooks.md` for the engine, and `knowledge/` for anyone who already did this game. 2. **Pick the engines** (table below). Default: Fish Audio Drama 3 on Modal for voices, Cerebras for the LLM, fal for speech-to-text. 3. **Stand the voice server up first, outside the game** (skill `fish-voice-server`): `npctalk health`, then `npctalk say "..." --voice Name/clone`. A game is the worst place to debug a GPU server. 4. **Make reference voices** for two or three characters (skill `voice-references`). 5. **Vertical slice in the game:** one NPC, typed text → LLM → voice → playback. Implement the four hooks below, nothing else. Log `stt / llm / tts / total` for every exchange. 6. **Voice input** (skill `voice-input`): push-to-talk first, then the open mic. 7. **Characters** (skill `npc-dialogue`): persona files, the character's own lines as style samples, scene facts, per-NPC history, output cleaning. 8. **Actions** (skill `npc-actions`): what they physically do, and doing what they're told. 9. **Verify in the real game**, with a dev mode that shows the target, the timings and the mod's state (below). 10. **Show it** (skill `npc-showcase-clips`), and if voice likeness matters, measure it (skill `voice-likeness-eval`) before claiming a setting is better. 11. **Write it down:** a field note in `knowledge/games//` (see `knowledge/README.md`). ## The four hooks a game has to provide Everything else is game-free code. In the GTA V example about 2,450 of the mod's ~5,500 lines (dialogue, LLM, voice clients, streaming playback, speech-to-text, open mic, action picking) build and run outside the game: `examples/gta-v/tools/npcvoice-check` compiles exactly those files into a console app. | Hook | What it returns | GTA V (ScriptHookVDotNet) | |---|---|---| | **Who is being talked to** | The NPC, or nobody | Living humans within 12 m; within 5 m always counts, 5-12 m only if the camera points at them (dot > 0.85). Named characters win ties; the last partner sticks while within 6 m | | **Scene facts** | A few short strings | Zone and street name, time, weather, the player's vehicle and weapon, wanted level, health, whether this is the NPC's home | | **Voice at the NPC** | Play streamed PCM from that NPC | NAudio playback, panned by screen position, faded 4-30 m, mouth driven by the audio level, the game's subtitle style | | **Actions** | Name → animation or behaviour | Anim dictionaries checked at load (a missing one falls back), clothing swaps, tasks (flee, fight, kneel) | Keep the hooks thin. If a game exposes them, the rest of this repo's code carries over. ## Choosing engines | Need | Default | Alternatives | |---|---|---| | Voice, realtime | **Fish Audio Drama 3** self-hosted on Modal (`server/drama3.py`): first audio ~0.4 s warm, acts `[tags]` well | **S2 Pro** (`server/s2pro.py`, smaller, fine-tunable); MiniMax via fal (no GPU to run, $1.50 per voice clone, 2.5-4.5 s per line) | | Voice, pre-rendered (cutscenes, demos) | Drama 3 with `best_of: 8` (best likeness measured) | ElevenLabs (scored lowest here, 0.612) | | LLM | Cerebras `qwen-3.8-27b`, reasoning off (~0.3 s) | Any OpenAI-compatible endpoint; reasoning models add seconds | | Speech-to-text | fal `nvidia/nemotron-asr-multilingual/asr` (~0.4 s, audio inline) | Whisper anywhere | | Actions | TypeSafe Jev (typed choice + probability, ~0.1 s) | A second LLM call with a JSON schema (slower, less calibrated) | Costs: an H100 runs about $4/hour while it's up; it scales to zero 5 minutes after the last request, and the next request waits 1-3 minutes for a cold start. The game pings the server while it runs, so it stays warm exactly while someone plays (`server/keepwarm.py` holds it up for a demo without the game). ## Verify like you mean it - **Log every exchange** with each step's time and what was heard and said. Most "it doesn't work" reports are answered by the log (in GTA V: "open mic -> nobody near" 755 times = the target lookup found nobody). - **Show the mod's state on screen.** A mod switched off with a hotkey looks exactly like a broken one; say "off" somewhere visible. Show who would be talked to (a marker) in a dev mode. - **Test the network half without the game** (`npcvoice-check` in the GTA example, `npctalk say` for the server). - **Measure, don't eyeball, voice likeness** (skill `voice-likeness-eval`). Two runs of ten lines fooled us once; ten runs didn't. ## Rules - **Single-player, or servers you run.** Never inject into online games with anti-cheat (for GTA V: never launch GTA Online with a script hook installed). - **Voices you have the rights to.** Your own, an actor who agreed, a licensed or generated voice. Cloning a game's characters for your own single-player experiment is one thing; publishing those clones is another. Never commit voice clips, game audio, or the game's dialogue (`npctalk check` blocks audio and transcripts). - **No secrets in the repo.** Keys come from environment variables. A server URL is a secret too when its `/health` needs no token: anyone with it can keep your GPU awake. - **Content.** NPCs swear if the game's characters do. Slurs are filtered by default and opt-in per persona (skill `npc-dialogue`). - **Ask first** before driving the user's keyboard and mouse in a running game, installing into game folders, or publishing anything. Full reasoning: `references/safety.md`. Architecture and file map: `references/architecture.md`.