Back to articles
AI & Cloud

Complete Guide to Building Real-Time AI Voice Agent with PipeCat and LiveKit on VPS

Detailed guide to deploying a real-time AI voice agent using PipeCat and LiveKit on VPS, covering setup, integration, testing, and performance optimization.

DPTCloud Engineering4 min read
Complete Guide to Building Real-Time AI Voice Agent with PipeCat and LiveKit on VPS

Introduction to Real-Time AI Voice Agent

Real-time AI Voice Agents are automated systems that process voice calls, respond to spoken input, and interact with users without human intervention. With advancements in AI, businesses can deploy voice assistants on self-hosted platforms like VPS to maintain full control over data and processing workflows.

In this guide, we will build a real-time AI Voice Agent using PipeCat—a voice pipeline framework—and LiveKit—a real-time audio streaming platform. Both support self-hosting and integrate with AI models such as Whisper (speech recognition), LLMs (automated responses), and TTS (text-to-speech) engines.

Prerequisites and Environment Setup

Hardware and Software Requirements

  • Cloud VPS with at least 8GB RAM and 4 CPU cores (recommended: 16GB RAM)
  • Operating System: Ubuntu 22.04 LTS
  • Installed: Docker, Docker Compose, Git
  • Static domain (or subdomain) with SSL (Let's Encrypt or self-signed)
  • Open ports: 80, 443, 7880 (LiveKit), 3000 (API), 5000 (TTS)

Set Up VPS and Install Required Tools

On your VPS, run the following commands to update the system and install essential tools:

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker and Docker Compose
sudo apt install docker.io docker-compose -y
sudo systemctl enable docker
sudo usermod -aG docker $USER

Integrating PipeCat and LiveKit

Configure LiveKit on VPS

LiveKit provides real-time audio streaming via WebRTC. We’ll self-host LiveKit using Docker Compose.

Create a `docker-compose.yml` file:

version: '3.8'

services:
  livekit:
    image: livekit/livekit-server:latest
    container_name: livekit-server
    ports:
      - "7880:7880"
      - "7881:7881"
    environment:
      - LIVEKIT_SERVER_URL=https://your-domain.com
      - LIVEKIT_API_KEY=your-api-key
      - LIVEKIT_API_SECRET=your-api-secret
      - LIVEKIT_RECORDING_DIR=/recordings
    volumes:
      - ./recordings:/recordings
    restart: unless-stopped

Install PipeCat and Integrate with LiveKit

PipeCat is a voice processing framework that supports integration with AI models. We’ll use it to process audio streams from LiveKit, analyze speech, and generate responses.

Run the following commands to install PipeCat:

git clone https://github.com/pipercat/pipercat.git
cd pipercat
pip install -r requirements.txt

Create a configuration file `config.yaml`:

livekit:
  url: wss://your-domain.com:7880
  api_key: your-api-key
  api_secret: your-api-secret

whisper:
  model: medium

tts:
  provider: elevenlabs
  api_key: your-elevenlabs-api-key

llm:
  provider: openai
  model: gpt-4-turbo
  api_key: your-openai-api-key

Run the System and Test

Start the Services

Launch the services using Docker Compose:

docker-compose up -d livekit

Start PipeCat to begin processing audio:

python pipercat/main.py --config config.yaml

Test Real-Time Calls

Use the LiveKit Web SDK to connect a browser or mobile app. When a call starts, audio is sent to PipeCat, processed by Whisper, and responded to via LLM and TTS.

To test, you can call a simulated number or use tools like curl to send sample audio.

Optimize Performance and Security

Load Balancing and Scaling

To handle multiple concurrent calls, use HAProxy or Nginx as a load balancer. Configure load balancing across multiple LiveKit and PipeCat instances.

Data Protection and Authentication

  • Store API keys and secrets in environment variables, not in Git
  • Use HTTPS across the entire system
  • Close unnecessary ports; only expose ports 443, 7880, and 3000
  • Configure firewall using UFW

Alternative Solutions and Extensions

  • Clawbot on Mac Mini: If you prefer not to use a VPS, deploy on a Mac Mini as shown in this guide.
  • Transform VPS into AI Privacy Proxy Engine: To protect business data from public cloud AI, see this article.

Troubleshooting

  • Cannot connect to LiveKit: Check port 7880, domain configuration, and API key.
  • Whisper fails to recognize speech: Verify model, audio format (16kHz, mono), and network connectivity.
  • TTS not working: Check ElevenLabs API key or switch to an internal TTS like Piper.

References

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478