claude-video-shorts

claude-video-shorts — Longform to Shortform Pipeline (V3)

One-Command Pipeline

V3 Screen-Aware Pipeline (recommended)

For screen recordings, tutorials, and mixed content — uses VLM for visual analysis:

source ~/.video-skill/bin/activate && bash scripts/screen_shorts_pipeline.sh "$INPUT" \
  [--count N] [--min-duration S] [--max-duration S] [--style STYLE] \
  [--platform PLATFORM] [--skip-captions] [--skip-vlm] [--hook-text "Your hook here"] \
  [--mode screen|talking-head|auto]

V1 Audio-Only Pipeline (legacy)

For simple talking-head content where face tracking is sufficient:

source ~/.video-skill/bin/activate && bash scripts/shorts_pipeline.sh "$INPUT" \
  [--count N] [--duration S] [--style STYLE] [--platform PLATFORM]

V3 Pipeline Architecture

INPUT (any video) → Auto-detect mode (screen vs talking-head)
        │
  [1] WhisperX transcription (word-level timestamps)
  [2] PySceneDetect scene boundaries
  [3] Extract keyframes at scene cuts + intervals
  [4] Auto-detect: face rate <60% → screen mode
  [5] VLM analysis (Gemini 2.5 Flash) → visual interest + zoom regions
  [6] Topic segmentation + multi-modal scoring
  [7] Per-topic: extract → reframe → caption → normalize
  [8] Export for platform → manifest JSON

Framed Layout (Screen Mode)

The validated layout for screen recordings (1080x1920):

+--------1080px--------+
|                       |
|  HOOK TEXT (200px)    |  "SEO Health Score: 57/100"
|  Compelling data pt   |  "What Claude found in seconds"
|                       |
+=======================+
|                       |
|  SCREEN CONTENT       |  Cropped from source, scaled to 1080x1372
|  (1372px)             |  VLM-selected crop position
|                       |  ~63% of source resolution
|  Score tables,        |  Readable on phone
|  data, charts...      |
|                       |
+=======================+
|                       |
|  KARAOKE CAPTIONS     |  Impact 72pt, yellow sweep fill
|  (348px)              |  Dark box background (BorderStyle=4)
|  Centered in zone     |  MarginV=240, word-by-word \kf timing
|                       |
+-----------------------+

FFmpeg Filter Chain

crop={W}:{H}:{X}:0 → scale=1080:1372 → pad=1080:1920:0:200:black
→ drawbox (chrome mask) → drawtext (hook) → ass (captions)

Crop Position Logic

VLM analyzes each frame and suggests zoom regions. The crop position (X) varies:

• Left-aligned content (tables, lists): x_pct ~0.22
• Center-aligned content (title pages, score circles): x_pct ~0.35
• Right-aligned content (sidebars, panels): x_pct ~0.55

Multiple zoom regions within a clip create jump cuts (not smooth pan) — this matches natural scroll/section transitions in screen recordings.

Multi-Modal Scoring (V3)

| Factor | Weight | Source | Description | |--------|--------|--------|-------------| | Visual interest | 0.30 | VLM | Charts, tables, data = high score | | Content completeness | 0.25 | VLM + transcript | Complete data shown, full thoughts | | Audio engagement | 0.20 | Transcript | Emphasis words, questions, emotion | | Hook potential | 0.15 | Transcript | First sentence strength | | Standalone coherence | 0.10 | Transcript | Works without context |

Caption Style (V3)

The validated caption parameters for the framed layout:

Font: Impact, 72pt, Bold
Primary: White (&H00FFFFFF)
Secondary/Karaoke: Yellow (&H0000FFFF)
Outline: Black, 4px
Background: Semi-transparent black (&HC0000000)
BorderStyle: 4 (opaque box behind text)
Alignment: 2 (bottom-center)
MarginV: 240 (centered in 348px bottom zone)
Letter spacing: 2
Animation: \kf word-by-word karaoke sweep
Words per line: 3

Flags Reference

| Flag | Default | Description | |------|---------|-------------| | --count N | 5 | Number of shorts to extract | | --min-duration S | 20 | Minimum clip length in seconds | | --max-duration S | 55 | Maximum clip length in seconds | | --style STYLE | bold | Caption style: default, bold, minimal, neon, shadow | | --platform PLAT | shorts | Export: tiktok, reels, shorts, all | | --skip-captions | false | Skip caption burn-in | | --skip-vlm | false | Skip VLM analysis (audio-only scoring) | | --hook-text TXT | auto | Custom hook text for all shorts | | --language LANG | en | Transcription language | | --output-dir DIR | input dir | Output directory | | --vlm-model MOD | gemini-2.5-flash | VLM model for frame analysis | | --max-frames N | 40 | Max frames for VLM analysis | | --mode MODE | auto | Force: screen, talking-head, auto |

Requirements

Core (always needed)

• FFmpeg with libass support
• WhisperX (pip install whisperx in venv, 6GB VRAM)
• PySceneDetect (pip install scenedetect)

VLM Analysis (screen mode)

• GOOGLE_API_KEY environment variable
• pip install google-genai in venv
• Cost: ~$0.01-0.03 per video (~40 frames via Gemini 2.5 Flash)

Face Tracking (talking-head mode)

• MediaPipe (pip install mediapipe)
• OpenCV (pip install opencv-python)
• CPU-only, no VRAM required

Platform Duration Limits

| Platform | Max Duration | Recommended | Aspect | |----------|-------------|-------------|--------| | TikTok | 10 min | 30-60s | 9:16 | | Instagram Reels | 90s | 30-60s | 9:16 | | YouTube Shorts | 60s | 30-55s | 9:16 |

Safety Rules

1. Always run bash scripts/preflight.sh "$INPUT" "$OUTPUT" before writes
2. Never overwrite source video — all shorts are new files
3. Confirm before processing videos longer than 60 minutes
4. Clean up temp files after pipeline completes
5. Report estimated processing time before starting

Reference

Load references/shorts-pipeline.md for detailed algorithms, scoring formulas, and tuning parameters.