peekaboo

Peekaboo

Native macOS GUI automation via peekaboo, using an AX-first hybrid approach that prioritizes accessibility-tree interaction when reliable and switches to screenshot, coordinate, JavaScript, or vision-based fallbacks when AX coverage breaks down in real apps.

Terminology used in this file:

• AX (Accessibility tree): The macOS accessibility hierarchy used for deterministic element discovery and interaction (B1, T2, etc.).
• peekaboo: The macOS automation CLI used for discovery (see, image), interaction (click, type, hotkey), and app/window/dialog control.
• UI-TARS: Vision model server used when AX discovery fails or times out on complex UIs.
• FSM: Finite State Machine controlling retries, transitions, and recovery paths.
• Retina scale: Display pixel ratio (typically 2.0) used to map screenshot coordinates correctly on Retina displays.
• snapshot ID: The snapshot_id returned by see; required for stable element resolution and invalidated by UI changes.

Setup

brew install jq
brew install --cask peekaboo
mkdir -p ./scripts
chmod +x ./scripts/peekaboo-safe.sh ./scripts/health-check.sh
./scripts/health-check.sh

• Grant Accessibility and Screen Recording permissions for your terminal agent host (for example, Terminal, iTerm2, Claude Code).
• Use ./scripts/peekaboo-safe.sh as the only invocation entrypoint for peekaboo commands.
• For full setup details and troubleshooting, see references/installation-guide.md if present in your local skill copy.

Staying Updated

This skill ships with an UPDATES.md changelog and UPDATE-GUIDE.md for your AI agent.

After installing, tell your agent: "Check UPDATES.md in the peekaboo skill for any new features or changes."

When updating, tell your agent: "Read UPDATE-GUIDE.md and apply the latest changes from UPDATES.md."

Follow UPDATE-GUIDE.md so customized local files are diffed before any overwrite.

Quick Start

Run a 3-step baseline flow: health check, discover, interact.

# 1) Health check
./scripts/health-check.sh

# 2) Discover UI state
./scripts/peekaboo-safe.sh see --app Safari --json --path /tmp/peekaboo/

# 3) Interact
./scripts/peekaboo-safe.sh click --on B1 --app Safari

Hard Rules (MUST FOLLOW)

1. All peekaboo calls MUST use ./scripts/peekaboo-safe.sh wrapper - never direct peekaboo calls
2. Sequential execution only - no parallel GUI operations (causes race conditions)
3. NEVER run tccutil reset - resets ALL app permissions, not just one
4. Always --mode screen - window mode returns 698B stubs (beta3 bug)
5. Health check required - run ./scripts/health-check.sh before automation
6. Fresh snapshots mandatory - re-run see before each click sequence (IDs expire)
7. Use see --analyze for Q&A - 50x more token-efficient than full screenshots

Navigation Strategy: Navigate Programmatically, Interact Visually

Core principle: Never waste turns clicking through UI to reach a target. Get there programmatically, then use peekaboo for the last-mile interaction.

| Method | When | Example | |--------|------|---------| | URL scheme | System Settings, Safari pages | open "x-apple.systempreferences:com.apple.Desktop-Settings.extension" | | AppleScript | Apps with scripting dictionary | osascript -e 'tell app "Finder" to open folder "Desktop"' | | CLI/flags | Apps with CLI args | open -a TextEdit /path/to/file | | Menu command | Navigate within already-open app | ./scripts/peekaboo-safe.sh menu click --path "View > Show Sidebar" | | AX sidebar click | Last resort - only if above fail | Unreliable for tightly-packed items (System Settings proven broken) |

Hostile AX apps (use URL scheme / AppleScript only):

• System Settings - SwiftUI toggles invisible to AX roles, sidebar clicks misfire

Routing Algorithm (Interaction Tier)

if target_is_native_macos_ui():
    use_ax_interaction()  # click --on B1, type, hotkey
elif target_is_swiftui_toggle():
    use_screenshot_coordinate_click()  # screenshot -> identify -> click --coords
elif target_is_web_form():
    use_safari_js_injection()  # osascript JavaScript
elif target_is_web_button():
    use_coordinate_click()  # click --coords X,Y --no-auto-focus
elif see_timeout_complex_app():
    start_ui_tars_server()  # vision provider required
elif dialog_or_sheet():
    add_no_auto_focus_flag()  # prevent focus conflicts
else:
    escalate_to_human()

FSM State Machine

| State | Entry Condition | Actions | Exit Criteria | Retry Budget | Next State | |-------|----------------|---------|---------------|--------------|------------| | PREFLIGHT | Start | health-check.sh, mkdir /tmp/peekaboo | all_checks_pass | 1 | CLASSIFY_TARGET | | CLASSIFY_TARGET | Health pass | Determine interaction tier | tier_identified | 1 | ACQUIRE_WINDOW | | ACQUIRE_WINDOW | Target known | app switch, window focus | app_frontmost | 3 | WAIT_FOR_UI_STABLE | | WAIT_FOR_UI_STABLE | Window focused | sleep, check spinner state | ui_elements_stable | 2 | DISCOVER | | DISCOVER | UI stable | see --app --json --path | elements_found | 3 | INTERACT | | INTERACT | Elements found | click/type/menu based on tier | action_completed | 5 | VERIFY | | VERIFY | Action attempted | see/image for state change | success_confirmed | 2 | EXTRACT or CLEANUP | | EXTRACT | Data needed | see --analyze or JS injection | data_extracted | 2 | CLEANUP | | CLEANUP | Task complete | clean snapshots, restore clipboard | cleanup_done | 1 | DONE | | RECOVER | Error detected | Retry from previous state | retry_limit_hit | 3 | HUMAN_HANDOFF or ABORT | | HUMAN_HANDOFF | Auth/CAPTCHA/TCC | Log issue, preserve evidence | human_resolved | inf | CLASSIFY_TARGET | | ABORT | Fatal error | Log failure, preserve evidence | - | - | - | | DONE | Success | Return results | - | - | - |

Bug-Specific Transitions

• see timeout -> start UI-TARS -> retry DISCOVER
• Safari AX click misfires -> force coordinate click path
• type targets URL bar -> force JS injection path
• daemon file-not-found -> daemon run --mode manual &
• Chrome window capture fail -> switch to Safari
• System Settings AX sidebar misfire -> use URL scheme: open "x-apple.systempreferences:com.apple.PANE_ID"
• SwiftUI toggle invisible to AX -> screenshot + coordinate click (not AX click)

Core Commands (Canonical Forms Only)

Discovery

• ./scripts/peekaboo-safe.sh see --app AppName --json --path /tmp/peekaboo/
• ./scripts/peekaboo-safe.sh see --analyze "What is the current state?" --app AppName
• ./scripts/peekaboo-safe.sh image --mode screen --path /tmp/peekaboo/screenshot.png

Interaction

• ./scripts/peekaboo-safe.sh click --on B1 --snapshot "$sid" --app AppName
• ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app AppName
• ./scripts/peekaboo-safe.sh type "text" --return --profile human --app AppName
• ./scripts/peekaboo-safe.sh hotkey --keys "cmd,w" --app AppName

App Control

• ./scripts/peekaboo-safe.sh app switch --to AppName
• ./scripts/peekaboo-safe.sh window focus --window-title "Title" --app AppName
• ./scripts/peekaboo-safe.sh menu click --path "File > Save As..." --app AppName

Safari Workarounds

• Navigate: ./scripts/peekaboo-safe.sh open "https://url" --app Safari
• Form fill: osascript -e 'tell application "Safari" to do JavaScript "document.querySelector(\"input[name=field]\").value = \"value\"" in document 1'
• Button click: ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app Safari

Agent Mode

• ./scripts/peekaboo-safe.sh agent "Natural language task" --model gpt-4o --max-steps 10
• ./scripts/peekaboo-safe.sh agent --resume --max-steps 5

Token Budget

see --analyze ~100 tokens | see --json ~600 | image --mode screen ~1400. Soft limit 50 turns, hard limit 100. Prefer see --analyze over full AX trees. Details in references/token-budget.md.

Capability Matrix

| Featu