ElevenLabs Agents Platform

Overview

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:

1. ASR (Automatic Speech Recognition) - Converts speech to text (32+ languages, sub-second latency)
2. LLM (Large Language Model) - Reasoning and response generation (GPT, Claude, Gemini, custom models)
3. TTS (Text-to-Speech) - Converts text to speech (5000+ voices, 31 languages, low latency)
4. Turn-Taking Model - Proprietary model that handles conversation timing and interruptions

🚨 Package Updates (November 2025)

ElevenLabs migrated to new scoped packages in August 2025:

DEPRECATED (Do not use):

• @11labs/react → DEPRECATED
• @11labs/client → DEPRECATED

Current packages:

npm install @elevenlabs/react@0.9.1        # React SDK
npm install @elevenlabs/client@0.9.1       # JavaScript SDK
npm install @elevenlabs/react-native@0.5.2 # React Native SDK
npm install @elevenlabs/elevenlabs-js@2.21.0 # Base SDK
npm install -g @elevenlabs/agents-cli@0.2.0  # CLI

If you have old packages installed, uninstall them first:

npm uninstall @11labs/react @11labs/client

When to Use This Skill

Use this skill when:

• Building voice-enabled customer support agents
• Creating interactive voice response (IVR) systems
• Developing conversational AI applications
• Integrating telephony (Twilio, SIP trunking)
• Implementing voice chat in web/mobile apps
• Configuring agents via CLI ("agents as code")
• Setting up RAG/knowledge bases for agents
• Integrating MCP (Model Context Protocol) servers
• Building HIPAA/GDPR-compliant voice systems
• Optimizing LLM costs with caching strategies

Platform Capabilities

Design & Configure:

• Multi-step workflows with visual builder
• System prompt engineering (6-component framework)
• 5000+ voices across 31 languages
• Pronunciation dictionaries (IPA/CMU formats)
• Speed control (0.7x-1.2x)
• RAG-powered knowledge bases
• Dynamic variables and personalization

Connect & Deploy:

• React SDK (@elevenlabs/react)
• JavaScript SDK (@elevenlabs/client)
• React Native SDK (@elevenlabs/react-native)
• Swift SDK (iOS/macOS)
• Embeddable widget
• Telephony integration (Twilio, SIP)
• Scribe (Real-Time Speech-to-Text) - Beta

Operate & Optimize:

• Automated testing (scenario, tool call, load)
• Conversation analysis and evaluation
• Analytics dashboard (resolution rates, sentiment, compliance)
• Privacy controls (GDPR, HIPAA, SOC 2)
• Cost optimization (LLM caching, model swapping, burst pricing)
• CLI for "agents as code" workflow

1. Quick Start (3 Integration Paths)

Path A: React SDK (Embedded Voice Chat)

For building voice chat interfaces in React applications.

Installation:

npm install @elevenlabs/react zod

Basic Example:

import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Public agent (no API key needed)
    agentId: 'your-agent-id',

    // OR private agent (requires API key)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // OR signed URL (server-generated, most secure)
    signedUrl: '/api/elevenlabs/auth',

    // Client-side tools (browser functions)
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      switch (event.type) {
        case 'transcript':
          console.log('User said:', event.data.text);
          break;
        case 'agent_response':
          console.log('Agent replied:', event.data.text);
          break;
      }
    },

    // Regional compliance (GDPR, data residency)
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}

Path B: CLI ("Agents as Code")

For managing agents via code with version control and CI/CD.

Installation:

npm install -g @elevenlabs/agents-cli
# or
pnpm install -g @elevenlabs/agents-cli

Workflow:

# 1. Authenticate
elevenlabs auth login

# 2. Initialize project (creates agents.json, tools.json, tests.json)
elevenlabs agents init

# 3. Create agent from template
elevenlabs agents add "Support Agent" --template customer-service

# 4. Configure in agent_configs/support-agent.json

# 5. Push to platform
elevenlabs agents push --env dev

# 6. Test
elevenlabs agents test "Support Agent"

# 7. Deploy to production
elevenlabs agents push --env prod

Project Structure Created:

your_project/
├── agents.json              # Agent registry
├── tools.json               # Tool configurations
├── tests.json               # Test configurations
├── agent_configs/           # Individual agent files
├── tool_configs/            # Tool configuration files
└── test_configs/            # Test configuration files

Path C: API (Programmatic Agent Management)

For creating agents dynamically (multi-tenant, SaaS platforms).

Installation:

npm install elevenlabs

Example:

import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

// Create agent
const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: {
        prompt: "You are a helpful customer support agent.",
        llm: "gpt-4o",
        temperature: 0.7
      },
      first_message: "Hello! How can I help you today?",
      language: "en"
    },
    tts: {
      model_id: "eleven_turbo_v2_5",
      voice_id: "your-voice-id"
    }
  }
});

console.log('Agent created:', agent.agent_id);

2. Agent Configuration

System Prompt Architecture (6 Components)

ElevenLabs recommends structuring agent prompts using 6 components:

1. Personality

Define the agent's identity, role, and character traits.

Example:

You are Alex, a friendly and knowledgeable customer support specialist at TechCorp.
You have 5 years of experience helping customers solve technical issues.
You're patient, empathetic, and always maintain a positive attitude.

2. Environment

Describe the communication context (phone, web chat, video call).

Example:

You're speaking with customers over the phone. Communication is voice-only.
Customers may have background noise or poor connection quality.
Speak clearly and occasionally use thoughtful pauses for emphasis.

3. Tone

Specify formality, speech patterns, humor, and verbosity.

Example:

Tone: Professional yet warm. Use contractions ("I'm" instead of "I am") to sound natural.
Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max).
Use encouraging phrases like "I'll be happy to help with that" and "Let's get this sorted for you."

4. Goal

Define objectives and success criteria.

Example:

Primary Goal: Resolve customer technical issues on the first call.
Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Offer proactive solutions
- End calls with confirmation that the issue is resolved

Success Criteria: Customer verbally confirms their issue is resolved.

5. Guardrails

Set boundaries, prohibited topics,