elevenlabs-agents

ElevenLabs Agents Platform

Overview

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:

1. ASR (Automatic Speech Recognition) - Converts speech to text (32+ languages, sub-second latency)
2. LLM (Large Language Model) - Reasoning and response generation (GPT, Claude, Gemini, custom models)
3. TTS (Text-to-Speech) - Converts text to speech (5000+ voices, 31 languages, low latency)
4. Turn-Taking Model - Proprietary model that handles conversation timing and interruptions

🚨 Package Migration (August 2025)

DEPRECATED (Do not use): @11labs/react and @11labs/client

Current packages:

bun add @elevenlabs/react@0.11.0        # React SDK
bun add @elevenlabs/client@0.11.0       # JavaScript SDK
bun add @elevenlabs/react-native@0.5.2  # React Native SDK
bun add @elevenlabs/elevenlabs-js@2.25.0 # Base SDK
bun add -g @elevenlabs/agents-cli@0.2.0  # CLI

If migrating, uninstall old packages first: npm uninstall @11labs/react @11labs/client

Quick Start

Path A: React SDK (Embedded Voice Chat)

For building voice chat interfaces in React applications.

Installation:

bun add @elevenlabs/react zod

Basic Example:

import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Authentication (choose one)
    agentId: 'your-agent-id',  // Public agent (no key needed)
    // apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,  // Private (dev only)
    // signedUrl: '/api/elevenlabs/auth',  // Signed URL (production)

    // Client-side tools
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onEvent: (event) => {
      if (event.type === 'transcript') console.log('User:', event.data.text);
      if (event.type === 'agent_response') console.log('Agent:', event.data.text);
    },

    // Regional compliance
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}

Complete template: See templates/basic-react-agent.tsx

Path B: CLI ("Agents as Code")

For managing agents via code with version control and CI/CD. Load references/cli-commands.md when using CLI workflows.

Quick workflow:

bun add -g @elevenlabs/agents-cli
elevenlabs auth login
elevenlabs agents init
elevenlabs agents add "Support Agent" --template customer-service
# Edit agent_configs/support-agent.json
elevenlabs agents push --env dev
elevenlabs agents test "Support Agent"

See: references/cli-commands.md for complete CLI reference and workflows.

Path C: API (Programmatic Agent Management)

For creating agents dynamically (multi-tenant, SaaS platforms). Load references/api-reference.md when using the API directly.

Quick example:

import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: { prompt: "You are a helpful support agent.", llm: "gpt-4o" },
      first_message: "Hello! How can I help you today?"
    },
    tts: { model_id: "eleven_turbo_v2_5", voice_id: "your-voice-id" }
  }
});

See: references/api-reference.md for complete API reference.

Agent Configuration

System Prompt Framework

ElevenLabs recommends a 6-component prompt structure: Personality (identity/role), Environment (communication context), Tone (formality/speech patterns), Goal (objectives/success criteria), Guardrails (boundaries/ethics), and Tools (available functions).

Example structure:

Personality: You are Alex, a friendly customer support specialist at TechCorp.
Environment: Phone communication, voice-only, potential background noise.
Tone: Professional yet warm. Use contractions. Keep responses to 2-3 sentences.
Goal: Resolve issues on first call. Success = customer confirms resolution.
Guardrails: Never give medical/legal advice. Escalate if customer becomes abusive.
Tools: lookup_order(id), transfer_to_supervisor(), send_password_reset(email)

Complete guide: Load references/system-prompt-guide.md when configuring agent prompts or improving conversation quality.

Turn-Taking Modes

| Mode | Behavior | Best For | |------|----------|----------| | Eager | Responds quickly, jumps in early | Fast-paced support, quick orders | | Normal | Balanced, waits for natural breaks | General customer service (default) | | Patient | Waits longer for detailed responses | Information collection, tutoring |

Configuration: "turn": { "mode": "patient" } in conversation_config

Core Features Summary

Voice & Language

• Multi-Voice: Dynamic voice switching (adds ~200ms latency per switch)
• Pronunciation Dictionary: IPA, CMU, or word substitutions for custom pronunciation
• Speed Control: 0.7x - 1.2x (1.0x = normal)
• Languages: 32+ languages with auto-detection and multi-language presets

Knowledge Base (RAG)

Upload documents (PDF/TXT/DOCX) for semantic search during conversations. Agent retrieves relevant chunks automatically. Adds ~500ms latency per query. Documents must be indexed before use.

See: references/api-reference.md for knowledge base API and configuration.

Tools (4 Types)

1. Client Tools - Execute in browser (UI updates, navigation, local storage)
2. Server Tools (Webhooks) - Execute on your backend (database, payments, CRM)
3. MCP Tools - Connect to Model Context Protocol servers (enterprise APIs)
4. System Tools - Built-in platform tools (end_conversation, transfer_call, mute_microphone, press_digit)

See: references/tool-examples.md when implementing tools. Load references/api-reference.md for webhook tool creation API.

Top 3 Critical Errors

Error 1: Package Deprecation (@11labs/*)

Symptom: Import errors, "module not found"

Solution:

npm uninstall @11labs/react @11labs/client
bun add @elevenlabs/react@0.11.0 @elevenlabs/client@0.11.0

# Update imports:
import { useConversation } from '@elevenlabs/react';  // Not @11labs/react

Error 2: Android Audio Cutoff (First Message)

Symptom: First agent message cuts off on Android only (iOS/web work fine)

Solution:

const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  connectionDelay: {
    android: 3_000,  // 3 seconds for Android audio mode switch
    ios: 0,
    default: 0
  }
});

Error 3: CSP (Content Security Policy) Violations

Symptom: "Refused to load the script..." errors, CSP blocks blob URLs

Solution - Self-host worklet files:

cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/

const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  workletPaths: {
    'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
    'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
  }
});

See all 17 errors: Load references/error-catalog.md when troubleshooting errors, debugging webhook failures, RAG issues, or platform-specific problems.

When to Load References

Load specific reference files