--- name: ellmos-ai/music-composer source: https://app.decimal.ai/s/ellmos-ai-music-composer@1/SKILL.md source_sha256: 3f9595864941 --- music-composer banner # music-composer — Storyline → Score Compose a background score that follows a video's dramatic arc, fully local: numpy waveform synthesis (oscillators, ADSR, layering, delay reverb) + ffmpeg for MP3 encoding. No samples, no MIDI hardware, no cloud API. ## When to use - A video/podcast needs background music that hits story beats at exact timestamps (opening, tension dip, drive section, climax, outro). - You want a deterministic, re-renderable score (same seed → same audio). - Chiptune, ambient, or electronic styles are acceptable. ## When NOT to use (honest genre limits) Waveform synthesis can deliver **chiptune, ambient, and electronic** background beds. It cannot deliver **pop, rock, classical, or orchestral film music** with its built-in synth — but the arrangement is backend-neutral: see "Beyond waveform synthesis" below for rendering the same composition through SoundFonts, orchestral libraries, or external generation. ## The method: Storyline → Score Reconstructed as-is from the reference score (179.5 s, C minor): 1. **Map story acts to music sections.** Acts that share one emotional state merge into one section (reference: 7 acts → 5 sections; acts 1–3 formed a single 62 s calm opening). Each section gets exact `start`/`end` in seconds. 2. **Assign emotion + intensity per section.** - *Emotion* selects the chord progression (scale degrees, minor example): `calm` i–VI–III–VII · `tense` iv–i–V–i · `alive` VI–VII–i–III · `driving` i–VII–VI–V · `epic` iv–V–VI–i (V is harmonic-minor major). - *Intensity* (0–1) drives tempo feel (BPM), layer count and volume ramp. Layers join at thresholds: pads always → sparse bells (<0.3) → arp (≥0.25) → bass (≥0.4) → lead (≥0.45) → drums (≥0.5) → drive extras (≥0.65) → pad octave doubling (≥0.85). 3. **Place special events on the timeline.** - `damp`: Gaussian volume duck — `1 - depth · exp(-((t-T)²)/(2σ²))`. Reference: depth 0.5, σ 1 s at T=84 s, on the spoken beat "Two databases. Same cluster." (~1:24). - `climax`: time window forcing full layering, peak volume, octave-doubled pads (reference: 154–170 s = 2:34–2:50, on the 2D→3D morph). - `outro`: from its start, drums/bass/lead drop out and bells fade to 20 % (reference: 170–179.5 s, closing card). 4. **Glue.** Raised-cosine smoothing at section boundaries (~2.5 s), global fade-in/out, feedback-delay reverb (100 ms, decay 0.25), tanh drive + peak normalization to a background level (reference master measured: mean ≈ −22 dB, max ≈ −1.6 dB). 5. **Stay behind the voiceover.** Sections marked "background" cap their volume ramp; the score is a bed, not a lead. ## Usage ```bash # Bundled standalone copy (this skill folder): python compose_music.py storyline.json -o out/score python compose_music.py --init # print a storyline template python compose_music.py --selftest # 3 s render + verification # Canonical engine (preferred when the repo is available): python /tools/compose_music.py storyline.json -o out/score ``` Output: `score.wav` (stereo 16-bit 44.1 kHz), `score.mp3` (192k, needs ffmpeg on PATH; skipped with a warning otherwise), `score.notes.json` (the arrangement/note log: every placed note with time, layer, MIDI pitch, duration, velocity), and `score.mid` — a real **Standard MIDI File** (type 1: tempo track with one tempo event per section, note track with GM program hints per style, drums on GM channel 10). Written dependency-free; no pretty_midi/mido needed. Determinism: same storyline + same `seed` → byte-identical WAV. ## Storyline JSON reference ```json { "title": "my-score", "duration": 179.5, "bpm": 100, "key": "C", "mode": "minor", "style": "chiptune", "seed": 42, "fade_in": 3.0, "fade_out": 5.0, "crossfade": 2.5, "peak": 0.85, "drive": 2.0, "reverb": {"delay_ms": 100, "decay": 0.25}, "sections": [ {"start": 0.0, "end": 62.0, "emotion": "calm", "intensity": 0.18, "bpm": 60, "comment": "optional note"}, {"start": 62.0, "end": 88.0, "emotion": "tense", "intensity": 0.4, "bpm": 72} ], "events": [ {"type": "damp", "time": 84.0, "depth": 0.5, "width": 1.0}, {"type": "climax", "start": 154.0, "end": 170.0}, {"type": "outro", "start": 170.0} ] } ``` - `duration` (required, seconds), `sections` (required, ≥1, sorted, non-overlapping, within duration). - Section fields: `start`, `end` (required); `emotion` ∈ calm | tense | alive | driving | epic | outro; `intensity` 0–1 (default per emotion); `bpm` (default: global `bpm`); `melody` (optional MIDI list, cycled; otherwise a seeded random walk over scale tones is generated). - `key`: note name (C, C#, Db, …), `mode`: minor | major. - `style`: chiptune | ambient | electronic (see below). - Event `damp`: `time` (s), `depth` 0–1, `width` = σ in seconds. - Defaults for fades/crossfade/peak/drive/reverb shown above; all optional. Full worked example (the reference video's reconstruction): [`example-storyline.json`](example-storyline.json) — renders a 179.5 s score matching the original's master level (mean ≈ −22 dB, max ≈ −1.6 dB). ## Style presets | Style | Lead | Bass/Arp | Drums | Character | |---|---|---|---|---| | `chiptune` | square + detune + triangle | triangle bass, fast square arps | noise-based kick/hat/snare | retro console, arpeggio-driven (the reference style) | | `ambient` | soft triangle/sine | slow arps, long-attack pads | only from intensity ≥ 0.75 | sparse, bell-focused, wide reverb feel | | `electronic` | sawtooth + detune + sub | punchy bass, saw arps | full kit | driving EDM-adjacent bed | ## Beyond waveform synthesis: better sound backends The genre ceiling above is the **sound backend, not the composition**. The Storyline→Score method produces a backend-neutral arrangement; the bundled numpy synth is just the free, offline default renderer. Every run also writes a Standard MIDI File (`.mid`) — render it through better instruments: - **Path A — SoundFont (recommended: local, free, no cloud).** Install FluidSynth (fluidsynth.org, Windows: `winget install FluidSynth`) plus a free GM SoundFont (GeneralUser GS by S. Christian Collins, or MuseScore_General.sf2). Then: `fluidsynth -ni soundfont.sf2 score.mid -F score_sf.wav -r 44100` → MP3 via ffmpeg. This unlocks pop/rock band sounds, piano, basic strings. - **Path B — orchestral / film.** Free orchestral libraries: VSCO 2 Community Edition, Soni Musicae, Salamander Grand Piano. Better strings/brass, more epic. Honest note: articulation and humanization (velocity variation, legato, dynamics curves) matter more than the sample set — and true film-score epicness additionally needs arrangement maturity; a backend alone is not enough. - **Path C — external AI generation (Suno etc.).** Possible, but raises privacy/rights questions — only after explicit user approval, never the default. Planned but **not yet implemented**: a `humanize` option in the engine (per-note velocity/timing jitter) so sample-based rendering does not sound mechanical (marked as TODO in the engine docstring). ## Boundaries - English and German storyline comments are fine; keys/values are fixed English. - Very short durations (< ~10 s) work but compress the dramaturgy (fades and crossfades are auto-capped at 20 % of duration / 50 % of the shortest section). - The note log (`*.notes.json`) is the analysis-friendly record; the `*.mid` file is the interchange format for DAWs/SoundFont renderers (GM program numbers are rough per-style hints — remap them to taste).