video-audio-design

When this skill is activated, always start your first response with the :speaker: emoji.

Video Audio Design

Video audio design is the practice of layering narration, sound effects, and background music into programmatic video compositions. Great audio transforms a slide-deck video into a polished production - narration guides the viewer, music sets the emotional tone, and SFX punctuate key moments. This skill covers generating speech with ElevenLabs and alternative TTS providers, creating synthetic sound effects with FFmpeg, sourcing royalty-free background music, implementing audio ducking so speech stays intelligible, and mixing all layers together in Remotion compositions with frame-accurate timing.

When to use this skill

Trigger this skill when the user:

• Wants to add narration or voiceover to a programmatic video
• Needs to generate speech with ElevenLabs, OpenAI TTS, or Edge TTS
• Asks about voice selection, voice settings, or voice cloning
• Wants to add background music or needs royalty-free music sources
• Asks about creating sound effects programmatically
• Wants to implement audio ducking (lowering music during speech)
• Needs to mix multiple audio layers in Remotion
• Asks about audio timing, volume levels, or frame-based audio sync

Do NOT trigger this skill for:

• Video scripting or storyboarding - use the video-scriptwriting skill
• Remotion component architecture or rendering - use the remotion-video skill
• Professional audio production in a DAW (Ableton, Logic, Pro Tools)
• Music composition or MIDI programming

Key principles

1. Layered audio architecture - Every video has three audio layers:

narration on top (loudest), SFX in the middle (accent volume), and background music at the base (lowest).

1. Narration drives timing - Generate narration first, measure its

duration, then set scene timing to match. Never fit narration into arbitrary scene lengths.

1. Duck music during speech - Background music must drop 50-60% when

narration plays. Use smooth ramps (10-15 frames) to avoid jarring jumps.

1. SFX as accents, not distractions - Keep SFX short (under 0.5s),

subtle in volume, and relevant to on-screen action.

1. Test audio in context - Always preview the full mix with all layers

together. Listen for muddy speech, volume spikes, or dead silence.

Core concepts

3-layer audio architecture

| Layer | Role | Base Volume | During Narration | |---|---|---|---| | Narration | Conveys information, drives pacing | 0.8-1.0 | N/A (top layer) | | SFX | Accents transitions and actions | 0.3-0.5 | 0.3-0.5 (unchanged) | | Background Music | Sets emotional tone, fills silence | 0.3-0.5 | 0.15-0.25 (ducked) |

ElevenLabs API model

ElevenLabs provides neural TTS via a REST API. The core flow:

1. Pick a voice (pre-made or cloned) - each has a voice_id
2. Send text + voice settings to /v1/text-to-speech/{voice_id}
3. Receive raw audio bytes (mp3 by default)
4. Write to file and measure duration for scene timing

Voice settings:

| Setting | Range | Low | High | Recommended | |---|---|---|---|---| | stability | 0-1 | More expressive, variable | More consistent, monotone | 0.4-0.6 | | similarity_boost | 0-1 | More creative | Closer to original voice | 0.6-0.8 | | style | 0-1 | Neutral delivery | Exaggerated style | 0.3-0.6 |

Audio ducking concept

Audio ducking reduces background music volume when narration starts and restores it when narration ends. In Remotion, use interpolate():

Music volume:  0.4 ---\              /--- 0.4
                       \            /
               0.15     \__________/
                     narration start → end

Ramps should take 10-15 frames (~0.3-0.5s at 30fps).

Frame-based audio sync in Remotion

• useCurrentFrame() returns the current frame number
• interpolate() maps frame ranges to value ranges (e.g., volume)
• <Sequence from={frame}> places audio at a specific frame
• <Audio volume={fn}> accepts a static number or a per-frame function

Convert seconds to frames: frames = seconds * fps.

Common tasks

1. Set up ElevenLabs API key and generate narration

import fs from 'fs';

const ELEVENLABS_API_URL = 'https://api.elevenlabs.io/v1';

async function generateNarration(
  text: string,
  voiceId: string,
  outputPath: string
): Promise<void> {
  const response = await fetch(
    `${ELEVENLABS_API_URL}/text-to-speech/${voiceId}`,
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'xi-api-key': process.env.ELEVEN_LABS_API_KEY!,
      },
      body: JSON.stringify({
        text,
        model_id: 'eleven_multilingual_v2',
        voice_settings: {
          stability: 0.5,
          similarity_boost: 0.75,
          style: 0.5,
          use_speaker_boost: true,
        },
      }),
    }
  );

  if (!response.ok) {
    const error = await response.text();
    throw new Error(`ElevenLabs API error ${response.status}: ${error}`);
  }

  const buffer = Buffer.from(await response.arrayBuffer());
  fs.writeFileSync(outputPath, buffer);
}

2. Select and configure voice settings

Voice selection questions: gender, age range, accent, energy level, warmth.

interface VoiceSettings {
  stability: number;
  similarity_boost: number;
  style: number;
  use_speaker_boost: boolean;
}

const presets: Record<string, VoiceSettings> = {
  explainer: { stability: 0.6, similarity_boost: 0.75, style: 0.4, use_speaker_boost: true },
  promo: { stability: 0.3, similarity_boost: 0.7, style: 0.7, use_speaker_boost: true },
  tutorial: { stability: 0.7, similarity_boost: 0.8, style: 0.2, use_speaker_boost: false },
};

3. Generate narration per scene from a script

import { execSync } from 'child_process';
import path from 'path';

interface Scene { id: string; narrationText: string; }
interface SceneWithAudio extends Scene {
  audioPath: string;
  durationMs: number;
  durationFrames: number;
}

function getAudioDurationMs(filePath: string): number {
  const output = execSync(
    `ffprobe -v error -show_entries format=duration -of csv=p=0 "${filePath}"`
  ).toString().trim();
  return Math.round(parseFloat(output) * 1000);
}

async function generateSceneNarrations(
  scenes: Scene[], voiceId: string, outputDir: string, fps: number
): Promise<SceneWithAudio[]> {
  const results: SceneWithAudio[] = [];
  for (const scene of scenes) {
    const audioPath = path.join(outputDir, `${scene.id}.mp3`);
    await generateNarration(scene.narrationText, voiceId, audioPath);
    const durationMs = getAudioDurationMs(audioPath);
    results.push({
      ...scene, audioPath, durationMs,
      durationFrames: Math.ceil((durationMs / 1000) * fps),
    });
  }
  return results;
}

4. Source background music

Royalty-free music sources:

• Pixabay Audio: https://pixabay.com/music/ (free, no attribution)
• Freesound: https://freesound.org/ (CC0/CC-BY)
• YouTube Audio Library: download from YouTube Studio
• Local files: place in public/audio/ for Remotion's staticFile()

5. Generate SFX with FFmpeg

# Click sound - short sine burst
ffmpeg -f lavfi -i "sine=frequency=800:duration=0.05" \
  -af "afade=t=out:st=0.02:d=0.03" click.wav

# Keyboard typing - filtered noise burst
ffmpeg -f lavfi -i "anoisesrc=d=0.08:c=white:a=0.3" \
  -af "highpass=f=2000,lowpass=f=8000,afade=t=out:st=0.04:d=0.04" type.wav

# Whoosh - frequency sweep
ffmpeg -f lavfi -i "sine=frequency=200:duration=0.4" \
  -af "vibrato=f=8:d=0.5,afade=t=in:d=0.1,afade=t=out:st=0.2:d=0.2,lowpass=f=1000" \
  whoosh.wav

# Ding/chime - bell synthesis
ffmpeg -f lavfi -i "sine=frequency=1200:duration=0.6" \
  -af "afade=t=out:st=0.1:d=0.5,aecho=0.8:0.88:40:0.4" ding.wav

# Pop - impulse
ffmpeg -f lavfi -i "sine=frequency=400:duration=0.08" \
  -af "afade=t=out:st=0.02:d=0.06,lowpass=f=600" pop.wav

# Transition swoosh
ffmpeg -f lavfi -i "s