Back to articles
AI & Cloud

Build a New Generation AI Voice Assistant with Piper TTS and Whisper on ARM VPS

Detailed guide to building an automated AI voice assistant using Piper TTS and Whisper on ARM VPS, supporting Vietnamese and English, suitable for businesses and individuals.

DPTCloud Engineering4 min read
Build a New Generation AI Voice Assistant with Piper TTS and Whisper on ARM VPS

Overview of AI Voice Assistant System on ARM VPS

AI voice assistants are becoming essential in automation, customer support, and automated communication. With advances in language models and speech processing, building a conversational AI system on ARM-based VPS platforms is now both feasible and efficient. This guide walks through creating a new-generation AI voice assistant using Piper TTS (Text-to-Speech) and Whisper (Speech-to-Text) on ARM VPS, supporting both Vietnamese and English.

Requirements and Prerequisites

To deploy this system, you need:

  • An ARM VPS (e.g., from High-RAM Cloud VPS)
  • Linux operating system (Ubuntu 22.04 or equivalent)
  • Required software: Python 3.9+, Git, Docker (optional)
  • Additional tools: FFmpeg, SoX
  • Installed and configured Private Hugging Face Mirror for faster model downloads

Building the System

1. Install and Configure Environment

Update the system and install necessary tools:

sudo apt update && sudo apt upgrade -y
sudo apt install -y git python3 python3-pip python3-venv ffmpeg sox

2. Install Whisper and Piper TTS

Whisper is a speech-to-text model by OpenAI, while Piper is a lightweight, high-performance text-to-speech model supporting Vietnamese. Install the required libraries:

pip install torch torchaudio transformers soundfile scipy
pip install git+https://github.com/openai/whisper.git
pip install piper-tts

3. Download Whisper and Piper Models

Choose models based on your needs. For example, download the small Whisper model for Vietnamese:

whisper --model tiny --language vi --task transcribe --output_dir ./transcripts audio.wav

For Piper, download a Vietnamese model (e.g., vi_VN-lingvo):

piper --model vi_VN-lingvo --output audio.mp3 --text "Hello, I am an AI assistant."

4. Create a Simple API Interface with Flask

Build a basic API to handle speech processing. Use Flask to receive audio, convert to text, process, and generate speech:

from flask import Flask, request, jsonify
import whisper
import subprocess
import os

app = Flask(__name__)

# Load Whisper model
model = whisper.load_model("tiny")

@app.route('/transcribe', methods=['POST'])
def transcribe():
    audio_file = request.files['audio']
    audio_file.save("input.wav")
    result = model.transcribe("input.wav")
    return jsonify({"text": result["text"]})

@app.route('/synthesize', methods=['POST'])
def synthesize():
    text = request.json['text']
    # Call Piper to generate speech
    subprocess.run(["piper", "--model", "vi_VN-lingvo", "--output", "output.mp3", "--text", text])
    return jsonify({"status": "success"})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Testing and Verification

Run the application and verify functionality:

  • Start Flask: python app.py
  • Send audio to /transcribe to get text
  • Send text to /synthesize to get audio

Optimization and Troubleshooting

1. Optimize Performance on ARM

ARM platforms offer high performance but require kernel and memory optimization. Use solutions such as:

2. Common Issues and Fixes

  • Model download fails: Check network connectivity and use Private Hugging Face Mirror.
  • Piper fails to generate audio: Verify model path and access permissions.
  • Whisper fails to recognize Vietnamese: Ensure the correct model is downloaded and supports Vietnamese.

References

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478