---
name: jpoindexter/multimodal-voice-and-haptics
source: https://app.decimal.ai/s/jpoindexter-multimodal-voice-and-haptics@1/SKILL.md
source_sha256: 1d9c79ea8e78
---

# Multimodal: Voice, Sound & Haptics

Screens are one channel. Real products speak, vibrate, chime, and listen — in cars, on wrists, across rooms, hands-free. This skill covers designing for **every output and input channel a human has**, and how to combine them without overwhelming the user. The governing principle: **pick the modality that matches the user's context and the task's nature, then layer modalities so each does what it's best at.**

## Modality strengths — pick the right channel

Every modality has a job it does well and jobs it does badly. Design by matching, not defaulting.

| Modality | Best for | Bad at | Context fit |
|---|---|---|---|
| **Visual** | Browsing, comparison, precision, dense info, scanning, persistent reference | Hands-busy, eyes-busy, no-screen | Desk, phone-in-hand, TV |
| **Voice input** | Fast input, hands-busy, eyes-busy, long dictation, search by name, "fire-and-forget" commands | Privacy, noisy rooms, precise editing, browsing options | Car, kitchen, accessibility, walking |
| **Voice/audio output** | Eyes-free confirmation, short answers, ambient notification, alerts | Long lists, comparison, precise data, privacy | Car, smart speaker, screen reader |
| **Haptic** | Confirmation, alert, texture/boundary feedback, silent notification | Conveying content, complex info | Wearable, phone-in-pocket, controller |
| **Touch/gesture** | Direct manipulation, precision, spatial control | Eyes-free, hands-busy, distance | Phone, tablet, touchscreen kiosk |

**Rule:** combine modalities so each does its strength. A smart-display timer shows remaining minutes (visual: precise), confirms "Timer set" (voice: eyes-free), and chimes when done (audio: ambient). Don't make one channel do everything.

**Eyes-free vs hands-free** are different constraints. Driving is *eyes-busy* (can glance briefly, can't read). Cooking is *hands-busy* (can look, can't touch). Walking-with-coffee is both. Identify which the user has before choosing.

---

## Voice UI / conversational design (VUI)

Voice has no buttons, no menus, no visual affordances. The user cannot see what they can say. This makes **discoverability the central VUI problem** — and it shapes every other decision.

### Grice's cooperative principles (the spine of conversation)
Good dialogue obeys four maxims. Violate one and the system feels robotic or rude.

| Maxim | Means | VUI application |
|---|---|---|
| **Quantity** | Say as much as needed, no more | Don't read 10 results; offer the top one + "want more?" |
| **Quality** | Be truthful; don't claim what you can't verify | "I'm not sure" beats a confident wrong answer |
| **Relevance** | Stay on the user's goal | Answer the question asked, not an adjacent one |
| **Manner** | Be brief, orderly, unambiguous | Short sentences, one idea each, no jargon |

### Turn-taking & barge-in
Conversation alternates. The system speaks, then **yields the floor** clearly (a tone, a falling intonation, silence). **Barge-in** = let the user interrupt and speak over the prompt; experienced users know what to say and shouldn't be forced to wait. Always allow it. Detect end-of-speech with an endpointer; don't cut users off mid-sentence (tune the silence timeout — too short clips slow speakers, too long feels laggy).

### Prompt design — open vs directed
- **Open prompt** ("What would you like to do?") — flexible, but the user doesn't know the vocabulary. Use only when the domain is obvious or for expert flows.
- **Directed prompt** ("You can check a balance or transfer money. Which one?") — teaches options, constrains recognition. Default for new users.
- **Tapered prompts** — full guidance first time, terse on repeat: "Which account?" not the full menu again.
- **Implicit menus** — bury the options in natural phrasing rather than reading a numbered list. Never "For X say one, for Y say two" unless legally required.

### Error recovery (the make-or-break of VUI)
Most voice failures aren't recognition — they're **bad recovery**. Three error types, each needs a distinct response:

| Error | Cause | Recovery |
|---|---|---|
| **No-input** (timeout) | User silent/unsure | Reprompt with more help: "Sorry, I didn't catch that. You can say…" |
| **No-match** (not understood) | OOV, noise, accent | Rephrase the prompt differently; offer examples; don't repeat verbatim |
| **Recognition error** (wrong) | Misheard | Confirm before acting; let user correct |

- **Escalating prompts:** each retry gives *more* help, not the same words louder. Reprompt 1 = light nudge, reprompt 2 = examples, reprompt 3 = graceful fallback.
- **Graceful fallback:** after 2–3 failures, hand off — to a human, to a screen, to a simpler yes/no, or "Let me text you a link." Never loop forever. **Three-strikes rule.**
- **Rapid reprompt:** if no-input, a quick "Still there?" beats a full menu re-read.

### Confirmation strategy
| Type | When | Example |
|---|---|---|
| **Explicit** | High-stakes, irreversible (payments, deletes, sends) | "Send $50 to Alex — yes or no?" |
| **Implicit** | Low-stakes, reversible; keeps flow fast | "Okay, $50 to Alex." (echoes back, no yes/no) |
| **None** | Trivial, easily undone | Just do it; allow "undo" |

Match confirmation cost to action cost. Explicit-confirming a volume change is annoying; implicit-confirming a wire transfer is dangerous.

### Persona, voice & tone
A voice *has* a personality whether you design one or not. Define it: warm vs efficient, formal vs casual, concise vs chatty. Keep it **consistent** across every prompt. Choose a TTS voice that fits brand and is intelligible at speed/over noise. Tone should **adapt to context** — terser when the user is in a hurry or has erred repeatedly, never cute during an error. Avoid over-anthropomorphizing (don't claim feelings or imply human understanding the system lacks).

### Brevity & lists
**Never read a long list aloud.** Working memory holds ~3–4 spoken items, not 7. Patterns:
- Offer the **top result** + "Want more options?" rather than all results.
- Read **3 at most**, then "…or hear more?"
- On a **multimodal** device, push the list to the screen and speak only the headline.

### Context & memory
Track conversational state: anaphora ("book *it*", "the *second* one"), carry-over slots ("…and make it *recurring*"), and prior turns. A system that forgets what was just said feels broken. Persist user preferences across sessions where privacy allows.

### Multimodal voice + screen (smart displays)
On Alexa Show / Google Nest Hub / CarPlay, voice and visual **co-present**. Rules:
- Voice carries the **headline**; screen carries the **detail/list**.
- Don't read what's clearly on screen verbatim — summarize, point ("Here are your options").
- Support both **touch and voice** for every action; users switch mid-task.
- Keep visuals **glanceable** — these are seen from across a room, not held.

### Wake words & always-listening
- Wake word ("Hey X") gates the always-on mic; only audio after the wake word is processed/sent. Make the **listening indicator unmistakable** (light ring, tone) so users know when it's hot.
- **Privacy is a first-class design concern:** disclose what's recorded, offer mic-off (hardware switch ideally), let users review and delete history, and never surprise-record. In shared/public space, assume bystanders haven't consented.

### Accessibility — voice two ways
- **Voice as access:** voice control is a primary input for users who can't use touch/keyboard (motor disability). Support full task completion by voice, not just shortcuts.
- **Access of voice:** voice-only interfaces **exclude** Deaf/HoH and speech-disabled users. Always provide a **non-voice path** (screen, text, touch). Voice should be an *option*, never the *only* door.

---

## Conversational / chat & LLM-agent UX

Text chat and LLM agents are voice's sibling — turn-based, but visible. The screen affords things voice can't (scrollback, citations, edit), and latency/uncertainty become visible problems.

**Do**
- **Stream tokens** as they generate — perceived latency drops sharply vs waiting for the full reply. Streaming is the single biggest chat-UX win.
- Show a **typing/thinking indicator** the instant the request fires (latency masking — fill the gap so it doesn't feel frozen).
- Offer a **Stop / cancel** button during generation, and **Regenerate** after.
- Provide **suggested replies / starter prompts** — they solve chat's blank-page version of the discoverability problem (what can I ask?).
- Show **citations / sources** inline for factual claims; make them clickable to the origin.
- Signal **uncertainty** honestly ("I'm not certain, but…") and surface errors as recoverable, not dead ends.
- Make agent **actions transparent** — show what tool ran, what it's about to do, and require confirmation before irreversible steps.
- Let users **edit and resend** a prior message; keep history scrollable and copyable.

**Don't**
- Don't block the UI on a long call with no feedback — always stream or indicate progress.
- Don't hide that an answer came from a model — label AI content.
- Don't auto-execute destructive agent actions without a confirm.
- Don't lose the user's typed message if generation fails — preserve and offer retry.

---

## Sound design

Sound is powerful and **easily abused**. The default should be **quiet**, with sound used deliberately and always under user control.

### Vocabulary
- **Earcon** — *abstract* musical motif mapped to an event (the Slack "knock", a startup chime). Learned, brand-able, composable into families.
- **Auditory icon** — *representational* sound that resembles its referent (trash-crumple for delete, camera-shutter for capture). Intuitive, no learning curve.
- **Notification sound** — short, distinct, recognizable across the room; differentiate priority by sound, not just volume.
- **Feedback sound** — confirms an action (send "whoosh", tap click).

### Principles
| Do | Don't |
|---|---|
| Default to silence/subtle; let sound be opt-in or contextual | Ship loud sounds on by default |
| Give a clear **mute/volume** control per sound category | Make sound un-disableable |
| Keep a **consistent sonic family** (same instrument/timbre across earcons) | Random unrelated sounds |
| Respect **silent mode / Do Not Disturb / focus** | Override the OS silent switch |
| Use **spatial audio** for directionality in AR/VR/gaming (where is the alert?) | Spatialize where it adds nothing |
| Use **sonification** to convey continuous data eyes-free (rising pitch = closer, Geiger counter, parking sensor) | Encode critical info in sound *alone* |

### Accessibility
- **Never rely on sound alone** — every audio cue needs a visual and/or haptic equivalent (Deaf/HoH users, muted phones, noisy rooms all miss it).
- **Caption** all spoken/meaningful audio in media.
- Don't use sound as the *only* signal for an error, success, or alert.

---

## Haptics

Haptic feedback is the most under-used and most over-used channel — a quiet "yes, that registered" or, done wrong, a buzzing nuisance. Restraint is everything.

### iOS — `UIFeedbackGenerator` family
| Generator | Use for |
|---|---|
| `UIImpactFeedbackGenerator` (`.light` / `.medium` / `.heavy`, plus `.soft`/`.rigid`) | A UI element collides/snaps — toggle flip, picker detent, drag-drop landing |
| `UINotificationFeedbackGenerator` (`.success` / `.warning` / `.error`) | Outcome of an operation — payment done, form invalid, action failed |
| `UISelectionFeedbackGenerator` | Discrete value change during a continuous pick — scrolling a wheel/slider through steps |
- **`prepare()`** the generator just before use to remove latency, then fire on the exact event.
- For rich custom textures, use **Core Haptics** (`CHHapticEngine`) — transient/continuous events with intensity & sharpness curves.

### Android
- **`View.performHapticFeedback(HapticFeedbackConstants…)`** — semantic constants (`CONFIRM`, `REJECT`, `LONG_PRESS`, `CLOCK_TICK`, `KEYBOARD_TAP`, `GESTURE_START/END`). Prefer these — they respect device tuning and user settings.
- **`VibratorManager` / `VibrationEffect`** (`createOneShot`, `createWaveform`, predefined `EFFECT_CLICK` / `EFFECT_TICK` / `EFFECT_HEAVY_CLICK`) for custom patterns. Newer APIs support richer actuators; older devices fall back to plain vibrate.
- Always require the `VIBRATE` permission and honor the system haptic setting.

### When to use haptics
| Pattern | Example | Strength |
|---|---|---|
| **Confirm** | Button/toggle activated, item added | Light impact / selection |
| **Alert** | Error, warning, validation fail | Notification warning/error |
| **Boundary / detent** | Slider step, scroll snap, pull-to-refresh threshold, end of list | Selection / light impact |
| **Texture / event** | Game collision, lock/unlock, success celebration | Custom Core Haptics / waveform |

### Do / Don't
| Do | Don't |
|---|---|
| Pair haptic with a visual change (haptic *reinforces*, rarely stands alone) | Buzz on every tap/scroll/keystroke — it desensitizes and drains battery |
| Map **intensity to importance** — heavy for errors, light for routine confirms | Use the same buzz for success and failure |
| Respect the **system haptics toggle** and provide an in-app off switch | Override the user's "reduce haptics" / silent preference |
| Keep patterns **short and distinct** | Long, ambiguous vibrations the user must decode |

### Accessibility
Haptics are a genuine **non-visual feedback channel** — valuable for low-vision users and eyes-free use. But some users disable them (sensory sensitivity) or can't perceive them — so **never make haptic the sole carrier** of information. Reinforce, don't replace.

---

## Beyond the phone — other modalities

### TV (10-foot UI)
- Designed to be read across a room; **large type, high contrast, generous spacing**, few items per screen.
- Input is a **D-pad / remote** (directional focus, not a pointer) or **voice remote**. Every element needs an obvious, high-contrast **focus state**; navigation must be predictable up/down/left/right.
- Minimize text entry — it's painful on a remote; prefer voice search and saved profiles.

### Automotive
- **Driver distraction is the constraint that overrides all else.** Glanceable visuals (≤2-second glances, NHTSA guidance), large touch targets, minimal depth.
- **Voice-first** for anything beyond a single tap — destination entry, messaging, calls. Keep menus shallow.
- Respect the **driving vs parked** state (lock complex tasks while moving). Follow CarPlay/Android Auto templates rather than free-form UI; they encode the safety rules.
- Combine: voice command + audio confirmation + a brief glanceable visual + a haptic on the wheel/seat for alerts.

### Wearables (watch)
- **Glance + haptic** is the core loop: a tap on the wrist (haptic) draws attention, a glanceable card delivers a few words, voice/dictation or a couple of taps respond.
- Sessions are **seconds**. One primary action per screen. Push detail to the phone.
- Haptics replace sound here (silent, on-body) — design a clear haptic vocabulary for notification types.

### AR / VR / spatial
- Input is **gaze + gesture + voice** ("look-and-pinch", point-and-speak); no mouse precision, so targets are large and dwell/confirm is forgiving.
- **Comfort is a hard requirement:** avoid forced rapid motion, keep UI within a comfortable field of view, respect the body's frame, and provide motion-sickness mitigations (vignetting, snap-turn).
- **Spatial audio** does real work — it directs attention to off-screen events. Haptics (controller/glove) confirm grabs and collisions.

---

## Common mistakes (cross-modal)

- **Voice menus that read every option** — violates the working-memory limit; offer top result + "more?".
- **No error recovery** — repeating the same prompt after a failure instead of escalating help, and never falling back to a human/screen.
- **Sound with no off switch** — or sounds that override the OS silent mode.
- **Haptic overuse** — buzzing on every interaction until users disable haptics entirely (and then miss the important ones).
- **No captions / sound-only signals** — excluding Deaf/HoH users and anyone on mute.
- **Voice-only with no alternative path** — locking out users who can't or won't speak.
- **Ignoring eyes-free / hands-free context** — a flow that demands precise touch while the user is driving or cooking.
- **Chat with no streaming or stop** — a frozen-looking UI during long generation, and no way to halt a runaway response.
- **One channel doing everything** — reading a list aloud that belongs on the screen, or showing a confirmation that should have been a haptic tap. Let each modality play its strength.