Create a python code for this application. Through tavus' echo mode, pass on the audio to elevenlabs conversational ai and get response from it to put into Tavus's replica. Figure out the intricacies of how the audio will be passed through browser/daily meeting etc. Dont rely on me to complete the code, ask me docs if needed, just complete the entire code in one single file Below is the documentation, ask me questions before if needed.

Documentation for Tavus and ElevenLabs:

Tavus Conversational Video Interface (CVI) Documentation

Table of Contents

1. Overview
2. Key Concepts
   • What is CVI?
   • What is a Conversation?
   • What are Personas?
   • What are Replicas?
   • What is a Digital Twin?
3. Getting Started
   • No Code
   • API Quick Start
4. API Reference
   • Create Conversation
   • Get Conversation
   • List Conversations
5. Layers and Modes Overview
   • Layers
   • Pipeline Modes
     • Full Pipeline Mode
     • Custom LLM / Bring Your Own Logic
     • Speech to Speech Mode
     • Echo Mode
       • Text or Audio (Base64) Echo
       • Microphone Echo
6. Echo Mode Quickstart
   • Part 1: Creating the Persona and Conversation
   • Part 2: Using Text and Audio Echo
7. Interactions Protocol
   • Echo Interaction
   • Text Respond Interaction

Overview

Tavus Conversational Video Interface (CVI)

The Tavus Conversational Video Interface (CVI) enables you to create real-time conversations with a digital replica. CVI provides an end-to-end pipeline for developing multimodal video interactions that are responsive and natural, mimicking human-like conversations.

Key Concepts

What is CVI? (Overview)

The Conversational Video Interface (CVI) is an end-to-end pipeline for creating real-time multimodal video conversations with a digital twin that can see, hear, and respond similarly to how a human would. Developers can deploy video AI agents/digital twins in minutes using CVI.

CVI is the world’s fastest interface of its kind, allowing you to add a human face and conversational ability to your AI agent or personality. With CVI, you can achieve utterance-to-utterance latency with SLAs as fast as under 1 second, encompassing the full roundtrip time for a participant to say something and for the replica to respond.

CVI provides a complete pipeline to facilitate conversations while allowing customization and integration with your existing components where necessary.

Key Features

• Face-to-Face Interactions: CVI is multimodal, understanding and utilizing facial expressions, body language, and natural conversational cues like interrupts and turn-taking.
• World’s Lowest Latency: Achieve SLAs as fast as under 1 second latency from utterance to utterance.
• End-to-End Solution: CVI offers a turnkey solution, handling all components required to deploy AI video agents without worrying about WebRTC, ASR, or other underlying technologies.
• Focused on Naturalness: Easily create high-quality AI replicas powered by our state-of-the-art replica model, Phoenix-2.

What is a Conversation?

A conversation is a single session or call with a digital twin using CVI. When you create a conversation, you receive a conversation_url, which provides a full video conferencing solution, eliminating the need to manage WebRTC or websockets. Navigating to this URL allows you to join a prebuilt meeting room UI to chat with your digital twin.

What are Personas?

Personas are the characters or AI agent personalities that contain all settings and configurations for that character or agent. For example, you can create a persona for "Tim the Sales Agent" or "Rob the Interviewer." Personas allow you to customize CVI’s layers and prompt the LLM with personality and context.

What are Replicas?

A replica is a talking-head/avatar of a human containing a voice and face clone, used as the video output layer for CVI. You can use stock replicas from Tavus or create your own with a few minutes of training data. Replicas are essential for video generation in CVI.

What is a Digital Twin?

A digital twin is an AI-powered digital version of a human that looks and sounds like a person and can see and respond similarly to a human.

Getting Started

No Code

You can easily try out CVI using the Tavus dashboard. Note that not all settings and modes are available via the dashboard.

API Quick Start

Check out the Quick Start Guide to learn how to use the APIs to create a persona and conversation. Be sure to grab an API key first!

Visit platform.tavus.io for more information.

API Reference

Create Conversation

Endpoint: POST /v2/conversations

Description:
Create a conversation with a replica in real time. After creating a conversation, a conversation_url will be returned in the response. This URL can be used to join the conversation directly or embedded in a website.

Authorization:

• Header: x-api-key (string, required)

Request Body: (application/json)

Response: (200 - application/json)

cURL Example:

bash
curl -X POST "https://tavusapi.com/v2/conversations" \
     -H "x-api-key: <api-key>" \
     -H "Content-Type: application/json" \
     -d '{
           "replica_id": "r79e1c033f",
           "persona_id": "p5317866",
           "callback_url": "https://yourwebsite.com/webhook",
           "conversation_name": "A Meeting with Hassaan",
           "conversational_context": "You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.",
           "custom_greeting": "Hey there Hassaan, long time no see!",
           "properties": {
             "max_call_duration": 3600,
             "participant_left_timeout": 60,
             "participant_absent_timeout": 300,
             "enable_recording": true,
             "enable_transcription": true,
             "apply_greenscreen": true,
             "language": "english",
             "recording_s3_bucket_name": "conversation-recordings",
             "recording_s3_bucket_region": "us-east-1",
             "aws_assume_role_arn": ""
           }
         }'

Python Example:

python
import requests

url = "https://tavusapi.com/v2/conversations"

payload = {
    "replica_id": "r79e1c033f",
    "persona_id": "p5317866",
    "callback_url": "https://yourwebsite.com/webhook",
    "conversation_name": "A Meeting with Hassaan",
    "conversational_context": "You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.",
    "custom_greeting": "Hey there Hassaan, long time no see!",
    "properties": {
        "max_call_duration": 3600,
        "participant_left_timeout": 60,
        "participant_absent_timeout": 300,
        "enable_recording": True,
        "enable_transcription": True,
        "apply_greenscreen": True,
        "language": "english",
        "recording_s3_bucket_name": "conversation-recordings",
        "recording_s3_bucket_region": "us-east-1",
        "aws_assume_role_arn": ""
    }
}
headers = {
    "x-api-key": "<api-key>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

JavaScript Example:

javascript
const axios = require('axios');

const url = 'https://tavusapi.com/v2/conversations';

const payload = {
  replica_id: "r79e1c033f",
  persona_id: "p5317866",
  callback_url: "https://yourwebsite.com/webhook",
  conversation_name: "A Meeting with Hassaan",
  conversational_context: "You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.",
  custom_greeting: "Hey there Hassaan, long time no see!",
  properties: {
    max_call_duration: 3600,
    participant_left_timeout: 60,
    participant_absent_timeout: 300,
    enable_recording: true,
    enable_transcription: true,
    apply_greenscreen: true,
    language: "english",
    recording_s3_bucket_name: "conversation-recordings",
    recording_s3_bucket_region: "us-east-1",
    aws_assume_role_arn: ""
  }
};

const headers = {
  'x-api-key': '<api-key>',
  'Content-Type': 'application/json'
};

axios.post(url, payload, { headers })
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

PHP Example:

php
<?php
$curl = curl_init();

$data = array(
    "replica_id" => "r79e1c033f",
    "persona_id" => "p5317866",
    "callback_url" => "https://yourwebsite.com/webhook",
    "conversation_name" => "A Meeting with Hassaan",
    "conversational_context" => "You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.",
    "custom_greeting" => "Hey there Hassaan, long time no see!",
    "properties" => array(
        "max_call_duration" => 3600,
        "participant_left_timeout" => 60,
        "participant_absent_timeout" => 300,
        "enable_recording" => true,
        "enable_transcription" => true,
        "apply_greenscreen" => true,
        "language" => "english",
        "recording_s3_bucket_name" => "conversation-recordings",
        "recording_s3_bucket_region" => "us-east-1",
        "aws_assume_role_arn" => ""
    )
);

curl_setopt_array($curl, array(
  CURLOPT_URL => "https://tavusapi.com/v2/conversations",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_POST => true,
  CURLOPT_POSTFIELDS => json_encode($data),
  CURLOPT_HTTPHEADER => array(
    "x-api-key: <api-key>",
    "Content-Type: application/json"
  ),
));

$response = curl_exec($curl);
curl_close($curl);
echo $response;
?>

Go Example:

go
package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
)

func main() {
    url := "https://tavusapi.com/v2/conversations"

    payload := map[string]interface{}{
        "replica_id":           "r79e1c033f",
        "persona_id":           "p5317866",
        "callback_url":         "https://yourwebsite.com/webhook",
        "conversation_name":    "A Meeting with Hassaan",
        "conversational_context": "You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.",
        "custom_greeting":      "Hey there Hassaan, long time no see!",
        "properties": map[string]interface{}{
            "max_call_duration":          3600,
            "participant_left_timeout":   60,
            "participant_absent_timeout": 300,
            "enable_recording":           true,
            "enable_transcription":       true,
            "apply_greenscreen":          true,
            "language":                   "english",
            "recording_s3_bucket_name":   "conversation-recordings",
            "recording_s3_bucket_region": "us-east-1",
            "aws_assume_role_arn":        "",
        },
    }

    jsonData, _ := json.Marshal(payload)
    req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
    if err != nil {
        fmt.Println(err)
        return
    }
    req.Header.Set("x-api-key", "<api-key>")
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer resp.Body.Close()

    var result map[string]interface{}
    json.NewDecoder(resp.Body).Decode(&result)
    fmt.Println(result)
}

Java Example:

java
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;

public class CreateConversation {
    public static void main(String[] args) {
        try {
            URL url = new URL("https://tavusapi.com/v2/conversations");
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn.setRequestMethod("POST");
            conn.setRequestProperty("x-api-key", "<api-key>");
            conn.setRequestProperty("Content-Type", "application/json");
            conn.setDoOutput(true);

            String jsonInputString = "{\n" +
                    "  \"replica_id\": \"r79e1c033f\",\n" +
                    "  \"persona_id\": \"p5317866\",\n" +
                    "  \"callback_url\": \"https://yourwebsite.com/webhook\",\n" +
                    "  \"conversation_name\": \"A Meeting with Hassaan\",\n" +
                    "  \"conversational_context\": \"You are about to talk to Hassaan, one of the cofounders of Tavus. He loves to talk about AI, startups, and racing cars.\",\n" +
                    "  \"custom_greeting\": \"Hey there Hassaan, long time no see!\",\n" +
                    "  \"properties\": {\n" +
                    "    \"max_call_duration\": 3600,\n" +
                    "    \"participant_left_timeout\": 60,\n" +
                    "    \"participant_absent_timeout\": 300,\n" +
                    "    \"enable_recording\": true,\n" +
                    "    \"enable_transcription\": true,\n" +
                    "    \"apply_greenscreen\": true,\n" +
                    "    \"language\": \"english\",\n" +
                    "    \"recording_s3_bucket_name\": \"conversation-recordings\",\n" +
                    "    \"recording_s3_bucket_region\": \"us-east-1\",\n" +
                    "    \"aws_assume_role_arn\": \"\"\n" +
                    "  }\n" +
                    "}";

            try(OutputStream os = conn.getOutputStream()) {
                byte[] input = jsonInputString.getBytes("utf-8");
                os.write(input, 0, input.length);           
            }

            int code = conn.getResponseCode();
            System.out.println("Response Code: " + code);
            // Handle response...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Sample Response:

json
{
  "conversation_id": "c123456",
  "conversation_name": "A Meeting with Hassaan",
  "status": "active",
  "conversation_url": "https://tavus.daily.co/c123456",
  "replica_id": "r79e1c033f",
  "persona_id": "p5317866",
  "created_at": "<string>"
}

Get Conversation

Endpoint: GET /v2/conversations/{conversation_id}

Description:
Retrieve a single conversation by its unique identifier.

Authorization:

• Header: x-api-key (string, required)

Path Parameters:

Response: (200 - application/json)

cURL Example:

bash
curl -X GET "https://tavusapi.com/v2/conversations/{conversation_id}" \
     -H "x-api-key: <api-key>"

Python Example:

python
import requests

conversation_id = "c123456"
url = f"https://tavusapi.com/v2/conversations/{conversation_id}"

headers = {
    "x-api-key": "<api-key>"
}

response = requests.get(url, headers=headers)

print(response.text)

Sample Response:

json
{
  "conversation_id": "c123456",
  "conversation_name": "A Meeting with Hassaan",
  "conversation_url": "https://tavus.daily.co/c123456",
  "callback_url": "https://yourwebsite.com/webhook",
  "status": "active",
  "replica_id": "r79e1c033f",
  "persona_id": "p5317866",
  "created_at": "",
  "updated_at": ""
}

List Conversations

Endpoint: GET /v2/conversations

Description:
Retrieve a list of all conversations generated by the API Key in use.

Authorization:

• Header: x-api-key (string, required)

Query Parameters:

Response: (200 - application/json)

Conversation Object:

cURL Example:

bash
curl -X GET "https://tavusapi.com/v2/conversations" \
     -H "x-api-key: <api-key>"

Python Example:

python
import requests

url = "https://tavusapi.com/v2/conversations"

headers = {
    "x-api-key": "<api-key>"
}

response = requests.get(url, headers=headers)

print(response.text)

Sample Response:

json
{
  "data": [
    {
      "conversation_id": "c123456",
      "conversation_name": "A Meeting with Hassaan",
      "status": "active",
      "conversation_url": "https://tavus.daily.co/c123456",
      "callback_url": "https://yourwebsite.com/webhook",
      "replica_id": "r79e1c033f",
      "persona_id": "p5317866",
      "created_at": "",
      "updated_at": "<string>"
    }
  ],
  "total_count": 123
}

Layers and Modes Overview

CVI provides an end-to-end pipeline that takes in user audio & video input and outputs a real-time replica AV output. This pipeline is hyper-optimized, with layers tightly coupled to achieve the lowest latency in the market. CVI is highly customizable, allowing you to enable, disable, or replace layers as necessary to best fit your use case.

By default, it is recommended to use as much of the CVI end-to-end pipeline as possible to guarantee the lowest latency and provide the best experience for your customers.

Layers

Tavus provides the following customizable layers as part of the CVI pipeline:

1. Transport

   • Description: Video conferencing / end-to-end WebRTC, currently powered by Daily. It handles audio/visual input and output for CVI.
   • Configurability: Allows configuration for input and output, each with either audio/mic or visual/camera properties.
   • Note: The Transport layer cannot be disabled.

2. Vision

   • Description: Processes user input video, enabling the replica to see and respond to user expressions and environments.
   • Configurability: Can be easily disabled if not required.

3. Speech Recognition with VAD (Interrupts)

   • Description: An optimized ASR system with incredibly fast and intelligent interrupts.

4. LLM

   • Description: Tavus provides ultra-low latency optimized Large Language Models (LLMs) or allows you to integrate your own.

5. TTS (Text-to-Speech)

   • Description: Provides TTS audio using a low-latency optimized voice model powered by Cartesia, or allows you to use other supported voice providers.

6. Realtime Replica

   • Description: Delivers high-quality streaming replicas powered by our proprietary Phoenix models.

Pipeline Modes

Tavus offers several modes that come with preconfigured layers tailored to specific use cases. You can configure the pipeline mode in the Create Persona API.

Full Pipeline Mode (Default and Recommended)

Description:
By default, it is recommended to use the end-to-end pipeline in its entirety to provide the lowest latency and the most optimized multimodal experience. Tavus offers a variety of LLMs (e.g., Llama3.1, OpenAI) that are optimized within the pipeline, achieving SLAs as fast as under 1 second for utterance-to-utterance latency. You can load LLMs with your knowledge base and prompt them as desired, as well as update the context live to simulate an asynchronous Retrieval-Augmented Generation (RAG) application.

Custom LLM / Bring Your Own Logic

Description:
Using a custom LLM is ideal for those who already have an LLM or are building business logic that needs to intercept input transcription and decide on the output. Using your own LLM may add latency, as Tavus's LLMs are hyper-optimized for low latency.

Note:
The Custom LLM mode doesn’t require an actual LLM. Any endpoint that can respond to chat completion requests in the required format can be used. For example, you could set up a server that takes in completion requests and responds with predetermined responses without involving an LLM.

Learn More: Custom LLM Mode

Speech to Speech Mode

Description:
The Speech to Speech pipeline mode allows you to bypass ASR, LLM, and TTS by leveraging an external speech-to-speech model. You can use Tavus's speech-to-speech model integrations or bring your own.

Note:
In this mode, vision capabilities from Tavus will be disabled, as there is no context to send to them currently.

Learn More: Speech to Speech Mode

Echo Mode

Description:
Echo Mode allows you to specify audio or text input for the replica to speak out. This mode is recommended only if your application does not require speech recognition (voice) or vision, or if you have a very specific ASR/Vision pipeline that must be used. Using your own ASR is generally slower and less optimized than using the integrated Tavus pipeline.

You can use text or audio input interchangeably in Echo Mode, with two possible configurations based on microphone enablement in the Transport layer.

Text or Audio (Base64) Echo

Description:
By turning off the microphone in the Transport Layer and using the Interactions Protocol, you can achieve Text and Audio (base64) echo behavior.

• Text Echo: Bypasses Tavus Vision, ASR, and LLM, directly sending text to the TTS layer. The replica will speak all provided text, allowing manual control of interrupts.
• Audio (Base64) Echo: Bypasses all layers except the Realtime Replica Layer. The replica will speak the provided base64 encoded audio.

Usage:
Use the Interactions Protocol to send text or base64 encoded audio.

Microphone Echo

Description:
By keeping the microphone on in the Transport Layer, you can bypass all layers in CVI and directly pass an audio stream that the replica will repeat. In this mode, interrupts are handled within your audio stream, and any received audio will be generated by the replica.

Recommendation:
Only use this mode if you have pre-generated audio, a voice-to-voice pipeline, or specific voice requirements.

Learn More: Echo Mode

Echo Mode Quickstart

This guide will help you get started with Echo Mode by setting up a persona and conversation, then sending echo messages.

Part 1: Creating the Persona and Conversation

1. Create a Persona:

   Endpoint: POST /v2/personas

   Request Body:

   json
   {
     "persona_name": "Echo Mode Persona",
     "pipeline_mode": "echo",
     "system_prompt": "You are a helpful assistant that can answer questions and help with tasks."
   }

   Sample Response:

   json
   {
     "persona_id": "p24293d6"
   }

2. Create a Conversation Using the Persona:

   Endpoint: POST /v2/conversations

   Request Body:

   json
   {
     "replica_id": "re8e740a42",
     "persona_id": "p24293d6",
     "conversation_name": "Music Chat with DJ Kot",
     "conversational_context": "Talk about the greatest hits from my favorite band, Daft Punk, and how their style influenced modern electronic music."
   }

   Sample Response:

   json
   {
     "conversation_id": "c12345",
     "conversation_name": "Music Chat with DJ Kot",
     "status": "active",
     "conversation_url": "https://tavus.daily.co/c12345",
     "replica_id": "re8e740a42",
     "persona_id": "p24293d6",
     "created_at": "2024-08-13T12:34:56Z"
   }

3. Join the Conversation and Send Echo Messages:

   Use the received conversation_id to join the conversation and send echo messages.

Part 2: Using Text and Audio Echo

Once you have a conversation_id, you can join the conversation and send echo messages, whether they are text or audio (base64 encoded). It is recommended to use a sample rate of 24000Hz for higher quality, but the default is 16000Hz for backward compatibility.

Python Flask App Example:

python
import sys
from daily import CallClient, Daily, EventHandler
from flask import Flask, jsonify, request
import time

app = Flask(__name__)

# Global variable to store the CallClient instance
call_client = None

class RoomHandler(EventHandler):
    def __init__(self):
        super().__init__()

    def on_app_message(self, message, sender: str) -> None:
        print(f"Incoming app message from {sender}: {message}")

def join_room(url):
    global call_client
    try:
        Daily.init()
        output_handler = RoomHandler()
        call_client = CallClient(event_handler=output_handler)
        call_client.join(url)
        print(f"Joined room: {url}")
    except Exception as e:
        print(f"Error joining room: {e}")
        raise

audio_chunks = ["base64-chunk-1", "base64-chunk-2", "base64-chunk-3"]

@app.route("/send_audio_message", methods=["POST"])
def send_audio_message():
    global call_client
    if not call_client:
        return jsonify({"error": "Not connected to a room"}), 400
    
    try:
        body = request.json
        conversation_id = body.get("conversation_id")
        modality = body.get("modality")
        base64_audio = body.get("audio")
        sample_rate = body.get("sample_rate", 16000)
        inference_id = body.get("inference_id")
        done = body.get("done")

        message = {
            "message_type": "conversation",
            "event_type": "conversation.echo",
            "conversation_id": conversation_id,
            "properties": {
                "modality": modality,
                "inference_id": inference_id,
                "audio": base64_audio,
                "done": done,
                "sample_rate": sample_rate,
            }
        }

        call_client.send_app_message(message)
        return jsonify({"status": "Message sent successfully"}), 200
    except Exception as e:
        return jsonify({"error": f"Failed to send message: {str(e)}"}), 500

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python script.py <room_url>")
        sys.exit(1)

    room_url = sys.argv[1]

    try:
        join_room(room_url)
        app.run(port=8000, debug=True)
    except Exception as e:
        print(f"Failed to start the application: {e}")
        sys.exit(1)

Usage:

1. Run the Flask App:

   bash
   python script.py https://tavus.daily.co/c12345

2. Send an Audio Message:

   bash
   curl -X POST "http://localhost:8000/send_audio_message" \
        -H "Content-Type: application/json" \
        -d '{
              "conversation_id": "c12345",
              "modality": "audio",
              "audio": "base64-encoded-audio",
              "sample_rate": 24000,
              "inference_id": "inference-id-123",
              "done": true
            }'

Learn More:

• Formatting Echo Interactions
• Sending Text or Audio Messages via the Echo Interaction

Interactions Protocol

Echo Interaction

Description:
This event allows developers to broadcast messages to Tavus, instructing the replica on exactly what to say. Any text provided in the text field will be spoken by the replica. This is commonly used in combination with the Interrupt Interaction.

Fields:

Properties Object:

Example:

json
{
  "message_type": "conversation",
  "event_type": "conversation.echo",
  "conversation_id": "c123456",
  "properties": {
    "modality": "text",
    "text": "Hey there Tim, long time no see!",
    "audio": "base64-encoded-audio",
    "sample_rate": 24000,
    "inference_id": "inference-id-123",
    "done": true
  }
}

Interactions Protocol Overview

The Interactions Protocol defines how messages are structured and sent to interact with the replica within a conversation. It ensures that the replica can accurately and efficiently process and respond to user inputs.

Additional Resources

• Learn about recording conversations: Recording Conversations
• Embed Conversation URL in a Website: Embedding Instructions
• Layers and Modes Details: CVI Modes and Layers

ElevenLabs Conversational AI Documentation

Table of Contents

1. Introduction
   • What is Conversational AI?
   • Key Components
     • Speech to Text
     • LLM
     • Text to Speech
     • Turn Taking
   • Pricing
   • Popular Applications
2. Quickstart
   • Agent Setup
     • The Web Dashboard
     • Pierogi Palace Assistant
     • Assistant Setup
       • Prerequisites
       • Access the Dashboard
       • Create Your Assistant
       • Configure Assistant Details
       • Configure Voice Settings
       • Test Your Assistant
       • Configure Data Collection
         • Configure Evaluation Criteria
         • Set Up Data Collection
         • View Conversation History
3. Libraries & SDKs
   • Python SDK
     • Installation
     • Usage
4. API Reference
   • WebSocket
     • Endpoint
     • Authentication
       • Using Agent ID
       • Using a Signed URL
     • Communication
       • Client-to-Server Messages
         • User Audio Chunk
         • Pong Message
       • Server-to-Client Messages
         • Conversation Initiation Metadata
         • Other Server-to-Client Messages
           • user_transcript
           • agent_response
           • audio
           • interruption
           • ping
           • client_tool_call
           • client_tool_result
     • Message Formats
       • user_transcript
       • agent_response
       • audio
       • interruption
       • ping
       • client_tool_call
       • client_tool_result
     • Latency Management
     • Security Best Practices
5. Additional Resources
6. Contact Us

Introduction

Deploy Customized, Conversational Voice Agents in Minutes

ElevenLabs Conversational AI is a platform designed to deploy customized, conversational voice agents swiftly. Built in response to customer needs, our platform eliminates the months typically required to develop conversation stacks from scratch by integrating essential building blocks.

What is Conversational AI?

Conversational AI refers to technologies that enable machines to understand, process, and respond to human language in a natural and interactive manner. ElevenLabs Conversational AI platform facilitates the creation and deployment of intelligent voice agents capable of engaging in real-time conversations with users.

Key Components

ElevenLabs Conversational AI combines the following core components to deliver a robust and scalable solution:

Speech to Text

Our finely-tuned Automatic Speech Recognition (ASR) model accurately transcribes the caller’s dialogue, enabling the AI agent to understand and process user inputs effectively.

LLM

Choose from a variety of Large Language Models (LLMs) such as Llama 3.3, 3.5 Sonnet, GPT-4o, and more. Alternatively, you can integrate your own custom LLM to tailor the conversational capabilities to your specific needs.

Text to Speech

Experience low-latency, human-like Text-to-Speech (TTS) synthesis across 5,000+ voices and 31 languages, ensuring that your AI agents can communicate naturally and effectively with users worldwide.

Turn Taking

Our custom turn-taking and interruption detection service provides a natural conversational flow, allowing the AI agent to manage dialogues seamlessly, similar to human interactions.

Pricing

• Setup & Prompt Testing: 500 credits per minute
• Production: 1,000 credits per minute

Note: Currently, we are covering the LLM costs, though these will be passed through to customers in the future.

Free Tier:
Start with our free tier, which includes 10 minutes of conversation per month.

Upgrade Options:
Need more? Upgrade to a paid plan instantly—no sales calls required. For enterprise usage (6+ hours of daily conversation), contact our sales team for custom pricing tailored to your needs.

Popular Applications

Companies and creators leverage our Conversational AI orchestration platform to create:

• Customer Service Representatives: AI agents trained on company help documentation to handle complex customer queries, troubleshoot issues, and provide 24/7 support in multiple languages.

• Virtual Assistants: Personal AI helpers that manage scheduling, set reminders, look up information, and help users stay organized throughout their day.

• Retail Support: Shopping assistants that help customers find products, provide personalized recommendations, track orders, and answer product-specific questions.

• Personalized Learning: 1-1 AI tutors that help students learn new topics and deepen their understanding. Enhance reading comprehension by interacting with books and articles.

Quickstart

Agent Setup

Deploy customized, conversational voice agents in minutes with ElevenLabs Conversational AI.

The Web Dashboard

The easiest way to get started with ElevenLabs Conversational AI is through our web dashboard. You can access your dashboard here. The web dashboard enables you to:

• Create and Manage AI Assistants: Design and oversee your conversational agents.
• Configure Voice Settings and Conversation Parameters: Tailor the voice and behavior of your agents.
• Review Conversation Analytics and Transcripts: Monitor and analyze interactions for continuous improvement.
• Manage API Keys and Integration Settings: Securely handle access and integrations.

Note: The web dashboard utilizes our Web SDK under the hood to handle real-time conversations.

Pierogi Palace Assistant

In this guide, we’ll create an AI assistant for "Pierogi Palace"—a modern Polish restaurant that takes orders through voice conversations. Our assistant will help customers order traditional Polish dishes with a contemporary twist.

Assistant Responsibilities:

• Menu Selection:

  • Various pierogi options with traditional and modern fillings
  • Portion sizes (available in dozens)

• Order Details:

  • Quantity confirmation
  • Price calculation in Polish złoty
  • Order review and modifications if needed

• Delivery Information:

  • Delivery address collection
  • Estimated preparation time (10 minutes)
  • Delivery time calculation based on location
  • Payment method confirmation (cash on delivery)

Assistant Setup

In this section, we’ll walk through configuring your Pierogi Palace assistant using ElevenLabs Conversational AI. We’ll set up the assistant’s voice, language model, and transcription settings to help customers place orders seamlessly.

Prerequisites

• An ElevenLabs account

1. Access the Dashboard

Step 1: Sign In to ElevenLabs

Navigate to elevenlabs.io and sign in to your account.

Step 2: Navigate to Conversational AI

In the ElevenLabs dashboard, click on Conversational > Agents in the left sidebar.

Navigating to the Conversational AI section

2. Create Your Assistant

Step 1: Start Creating a New Assistant

• Click the + button to create a new AI Agent.
• Choose the Blank Template option and name the agent Pierogi Palace.

Creating a new assistant

Step 2: Configure Assistant Details

Set the First Message & System Prompt fields to the following, leaving the Knowledge Base and Tools empty for now:

• Greeting Message
• System Prompt

plaintext
Welcome to Pierogi Palace! I'm here to help you place your order. What can I get started for you today?

3. Configure Voice Settings

Step 1: Select a Voice for Your Assistant

Choose from over 3,000 lifelike voices available in ElevenLabs. For this demo, we will use Jessica’s voice.

Assistant voice configuration

Note: Higher quality settings may slightly increase response time. For an optimal customer experience, balance quality and latency based on your assistant’s expected use case.

4. Test Your Assistant

Step 1: Converse with Your Assistant

Press the Order button and try ordering some pierogi to see how the assistant handles the conversation.

Testing your assistant

5. Configure Data Collection

Configure evaluation criteria and data collection to analyze conversations and improve your assistant’s performance.

Step 1: Configure Evaluation Criteria

• Navigate to the ANALYSIS section in your assistant’s settings to define custom criteria for evaluating conversations.

Goal Prompt Criteria:
This passes the conversation transcript to the LLM to verify if specific goals were met. Results will be:

• success
• failure
• unknown

Plus a rationale explaining the chosen result.

Configure the following fields:

• Name: Enter a descriptive name
• Prompt: Enter detailed instructions for evaluating the conversation

Example:

• Name: order_completion

• Prompt:
  Evaluate if the conversation resulted in a successful order completion.

  Success criteria:

  • Customer selected at least one pierogi variety
  • Quantity was confirmed
  • Delivery address was provided
  • Total price was communicated
  • Delivery time estimate was given

  Return "success" only if ALL criteria are met.

Step 2: Set Up Data Collection

In the Data Collection section, define specifications for extracting data from conversation transcripts.

• Click Add data collection item and configure:

  Data Type	Description
  Data type:	Select “string”
  Identifier:	Enter a unique identifier for this data point
  Description:	Provide detailed instructions for the LLM about how to extract the specific data from the transcript

Example Data Collection Items:

• Order Type

  • Identifier: order_type
  • Description: Extract the type of order from the conversation. Should be one of:
    • delivery
    • pickup
    • inquiry_only

• Ordered Items

  • Identifier: ordered_items
  • Description: List the items ordered by the customer.

• Delivery Zone

  • Identifier: delivery_zone
  • Description: Determine the delivery zone based on the address provided.

• Interaction Type

  • Identifier: interaction_type
  • Description: Identify the type of interaction (e.g., order placement, inquiry).

Step 3: View Conversation History

You can view evaluation results and collected data for each conversation in the History tab.

Your Pierogi Palace assistant is now ready to take orders 🥟! The assistant can handle menu inquiries, process orders, and provide delivery estimates.

Libraries & SDKs

Python SDK

The Conversational AI SDK allows you to deploy customized, interactive voice agents in minutes.

Also see: Conversational AI Overview

Installation

Install the elevenlabs Python package in your project:

bash
pip install elevenlabs
# or
poetry add elevenlabs

If you want to use the default implementation of audio input/output, you will also need the pyaudio extra:

bash
pip install "elevenlabs[pyaudio]"
# or
poetry add "elevenlabs[pyaudio]"

Note: The pyaudio package installation might require additional system dependencies. See the PyAudio package README for more information.

Linux/macOS:

On Debian-based systems, install the dependencies with:

bash
sudo apt install portaudio19

Usage

In this example, we will create a simple script that runs a conversation with the ElevenLabs Conversational AI agent. You can find the full code in the ElevenLabs examples repository.

1. Import the necessary dependencies:

   python
   import os
   import signal
   from elevenlabs.client import ElevenLabs
   from elevenlabs.conversational_ai.conversation import Conversation
   from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface

2. Load the agent ID and API key from environment variables:

   python
   agent_id = os.getenv("AGENT_ID")
   api_key = os.getenv("ELEVENLABS_API_KEY")

   Note: The API key is only required for non-public agents that have authentication enabled. You don’t have to set it for public agents, and the code will work fine without it.

3. Create the ElevenLabs client instance:

   python
   client = ElevenLabs(api_key=api_key)

4. Initialize the Conversation instance:

   python
   conversation = Conversation(
       # API client and agent ID.
       client,
       agent_id,
       # Assume auth is required when API_KEY is set.
       requires_auth=bool(api_key),
       # Use the default audio interface.
       audio_interface=DefaultAudioInterface(),
       # Simple callbacks that print the conversation to the console.
       callback_agent_response=lambda response: print(f"Agent: {response}"),
       callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
       callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
       # Uncomment if you want to see latency measurements.
       # callback_latency_measurement=lambda latency: print(f"Latency: {latency}ms"),
   )

   Note: We are using the DefaultAudioInterface, which utilizes the default system audio input/output devices for the conversation. You can also implement your own audio interface by subclassing elevenlabs.conversational_ai.conversation.AudioInterface.

5. Start the conversation:

   python
   conversation.start_session()

6. Handle clean shutdown when the user presses Ctrl+C:

   python
   signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())

7. Wait for the conversation to end and print out the conversation ID:

   python
   conversation_id = conversation.wait_for_session_end()
   print(f"Conversation ID: {conversation_id}")

8. Run the script and start talking to the agent:

   • For public agents:

     bash
     AGENT_ID=youragentid python demo.py

   • For private agents:

     bash
     AGENT_ID=youragentid ELEVENLABS_API_KEY=yourapikey python demo.py

Was this page helpful?

API Reference

WebSocket

Create real-time, interactive voice conversations with AI agents using the ElevenLabs WebSocket API. For convenience, consider using the official SDKs provided by ElevenLabs.

Endpoint

wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}

Authentication

Using Agent ID

For public agents, you can directly use the agent_id in the WebSocket URL without additional authentication:

wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>

Using a Signed URL

For private agents or conversations requiring authorization, obtain a signed URL from your server, which securely communicates with the ElevenLabs API using your API key.

Example using cURL:

Request:

bash
curl -X GET "https://api.elevenlabs.io/v1/convai/conversation/get_signed_url?agent_id=<your-agent-id>" \
     -H "xi-api-key: <your-api-key>"

Response:

json
{
  "signed_url": "wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>&token=<token>"
}

Note: Never expose your ElevenLabs API key on the client side.

Communication

Client-to-Server Messages

User Audio Chunk

Send audio data from the user to the server.

Format:

json
{
  "user_audio_chunk": "<base64-encoded-audio-data>"
}

Notes:

• Audio Format Requirements:

  • PCM 16-bit mono format
  • Base64 encoded
  • Sample rate of 16,000 Hz

• Recommended Chunk Duration:

  • Send audio chunks approximately every 250 milliseconds (0.25 seconds)
  • This equates to chunks of about 4,000 samples at a 16,000 Hz sample rate

• Optimizing Latency and Efficiency:

  • Balance Latency and Efficiency: Sending audio chunks every 250 milliseconds offers a good trade-off between responsiveness and network overhead.
  • Adjust Based on Needs:
    • Lower Latency Requirements: Decrease the chunk duration to send smaller chunks more frequently.
    • Higher Efficiency Requirements: Increase the chunk duration to send larger chunks less frequently.
    • Network Conditions: Adapt the chunk size if you experience network constraints or variability.

Pong Message

Respond to server ping messages by sending a pong message, ensuring the event_id matches the one received in the ping message.

Format:

json
{
  "type": "pong",
  "event_id": 12345
}

Server-to-Client Messages

Conversation Initiation Metadata

Provides initial metadata about the conversation.

Format:

json
{
  "type": "conversation_initiation_metadata",
  "conversation_initiation_metadata_event": {
    "conversation_id": "conv_123456789",
    "agent_output_audio_format": "pcm_16000"
  }
}

Other Server-to-Client Messages

Message Formats

user_transcript

Format:

json
{
  "type": "user_transcript",
  "user_transcription_event": {
    "user_transcript": "Hello, how are you today?"
  }
}

agent_response

Format:

json
{
  "type": "agent_response",
  "agent_response_event": {
    "agent_response": "Hello! I'm doing well, thank you for asking. How can I assist you today?"
  }
}

audio

Format:

json
{
  "type": "audio",
  "audio_event": {
    "audio_base_64": "SGVsbG8sIHRoaXMgaXMgYSBzYW1wbGUgYXVkaW8gY2h1bms=",
    "event_id": 67890
  }
}

interruption

Format:

json
{
  "type": "interruption",
  "interruption_event": {
    "event_id": 54321
  }
}

ping

Format:

json
{
  "type": "ping",
  "ping_event": {
    "event_id": 13579,
    "ping_ms": 50
  }
}

client_tool_call

Format:

json
{
  "type": "client_tool_call",
  "client_tool_call": {
    "tool_name": "weather_lookup",
    "tool_call_id": "tool123",
    "parameters": {
      "location": "New York"
    }
  }
}

client_tool_result

Format:

json
{
  "type": "client_tool_result",
  "tool_call_id": "tool123",
  "result": "The current weather in New York is 75°F and sunny.",
  "is_error": false
}

Latency Management

To ensure smooth conversations, implement the following strategies:

• Adaptive Buffering: Adjust audio buffering based on network conditions.
• Jitter Buffer: Implement a jitter buffer to smooth out variations in packet arrival times.
• Ping-Pong Monitoring: Use ping and pong events to measure round-trip time and adjust accordingly.

Security Best Practices

• Rotate API Keys Regularly: Enhance security by regularly updating your API keys.
• Use Environment Variables: Store API keys securely using environment variables.
• Implement Rate Limiting: Prevent abuse by limiting the number of requests per user or IP address.
• User Prompt Clarity: Clearly explain the intention when prompting users for microphone access.
• Optimized Chunking: Tweak the audio chunk duration to balance latency and efficiency.

Additional Resources

• ElevenLabs Conversational AI Documentation: Documentation
• ElevenLabs Conversational AI SDKs: SDKs
• Conversational AI Overview: Overview
• Recording Conversations: Recording Instructions
• Embedding Instructions: Embed Conversation URL
• CVI Modes and Layers: Modes and Layers Details

First of all, create production level code. Also you're using elevenlabs conversational agent api, not asr then llm and then tts. Adjust likewise