Complete Guide to Building Real-Time AI Voice Agent with PipeCat and LiveKit on VPS
Detailed guide to deploying a real-time AI voice agent using PipeCat and LiveKit on VPS, covering setup, integration, testing, and performance optimization.
Introduction to Real-Time AI Voice Agent
Real-time AI Voice Agents are automated systems that process voice calls, respond to spoken input, and interact with users without human intervention. With advancements in AI, businesses can deploy voice assistants on self-hosted platforms like VPS to maintain full control over data and processing workflows.
In this guide, we will build a real-time AI Voice Agent using PipeCat—a voice pipeline framework—and LiveKit—a real-time audio streaming platform. Both support self-hosting and integrate with AI models such as Whisper (speech recognition), LLMs (automated responses), and TTS (text-to-speech) engines.
Prerequisites and Environment Setup
Hardware and Software Requirements
- Cloud VPS with at least 8GB RAM and 4 CPU cores (recommended: 16GB RAM)
- Operating System: Ubuntu 22.04 LTS
- Installed: Docker, Docker Compose, Git
- Static domain (or subdomain) with SSL (Let's Encrypt or self-signed)
- Open ports: 80, 443, 7880 (LiveKit), 3000 (API), 5000 (TTS)
Set Up VPS and Install Required Tools
On your VPS, run the following commands to update the system and install essential tools:
# Update system
sudo apt update && sudo apt upgrade -y
# Install Docker and Docker Compose
sudo apt install docker.io docker-compose -y
sudo systemctl enable docker
sudo usermod -aG docker $USER
Integrating PipeCat and LiveKit
Configure LiveKit on VPS
LiveKit provides real-time audio streaming via WebRTC. We’ll self-host LiveKit using Docker Compose.
Create a `docker-compose.yml` file:
version: '3.8'
services:
livekit:
image: livekit/livekit-server:latest
container_name: livekit-server
ports:
- "7880:7880"
- "7881:7881"
environment:
- LIVEKIT_SERVER_URL=https://your-domain.com
- LIVEKIT_API_KEY=your-api-key
- LIVEKIT_API_SECRET=your-api-secret
- LIVEKIT_RECORDING_DIR=/recordings
volumes:
- ./recordings:/recordings
restart: unless-stopped
Install PipeCat and Integrate with LiveKit
PipeCat is a voice processing framework that supports integration with AI models. We’ll use it to process audio streams from LiveKit, analyze speech, and generate responses.
Run the following commands to install PipeCat:
git clone https://github.com/pipercat/pipercat.git
cd pipercat
pip install -r requirements.txt
Create a configuration file `config.yaml`:
livekit:
url: wss://your-domain.com:7880
api_key: your-api-key
api_secret: your-api-secret
whisper:
model: medium
tts:
provider: elevenlabs
api_key: your-elevenlabs-api-key
llm:
provider: openai
model: gpt-4-turbo
api_key: your-openai-api-key
Run the System and Test
Start the Services
Launch the services using Docker Compose:
docker-compose up -d livekit
Start PipeCat to begin processing audio:
python pipercat/main.py --config config.yaml
Test Real-Time Calls
Use the LiveKit Web SDK to connect a browser or mobile app. When a call starts, audio is sent to PipeCat, processed by Whisper, and responded to via LLM and TTS.
To test, you can call a simulated number or use tools like curl to send sample audio.
Optimize Performance and Security
Load Balancing and Scaling
To handle multiple concurrent calls, use HAProxy or Nginx as a load balancer. Configure load balancing across multiple LiveKit and PipeCat instances.
Data Protection and Authentication
- Store API keys and secrets in environment variables, not in Git
- Use HTTPS across the entire system
- Close unnecessary ports; only expose ports 443, 7880, and 3000
- Configure firewall using UFW
Alternative Solutions and Extensions
- Clawbot on Mac Mini: If you prefer not to use a VPS, deploy on a Mac Mini as shown in this guide.
- Transform VPS into AI Privacy Proxy Engine: To protect business data from public cloud AI, see this article.
Troubleshooting
- Cannot connect to LiveKit: Check port 7880, domain configuration, and API key.
- Whisper fails to recognize speech: Verify model, audio format (16kHz, mono), and network connectivity.
- TTS not working: Check ElevenLabs API key or switch to an internal TTS like Piper.
References
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
