Deploy OpenWebUI Connecting to Self-Built VLLM Cluster for Secure Shared AI Platform Across Enterprises
Detailed guide on deploying OpenWebUI connected to a self-built VLLM cluster to create a secure shared AI platform for enterprises, with comprehensive security measures from network...
Introduction
Deploying a shared AI platform across an enterprise requires robust security, performance, and scalability. In this context, building a self-hosted VLLM (Very Large Language Model) cluster integrated with OpenWebUI offers an effective solution. This guide provides a detailed walkthrough of deploying such a system on high-security VPS infrastructure at DPTCloud, incorporating comprehensive security measures from infrastructure to application layers.
Requirements and Preparation
To successfully deploy the system, you need:
- Cloud VPS with L3/L4/L7 Anycast Anti-DDoS at DPTCloud
- Network Security & 99.98% SLA Infrastructure at DPTCloud
- Software: Docker, Docker Compose, OpenWebUI, VLLM, Nginx, Let's Encrypt (SSL)
- Hardware: Minimum 16GB RAM, 4 CPU cores, 100GB SSD or larger
Deploying the VLLM Cluster
We will deploy VLLM across multiple nodes to form a cluster. Each node will run a VLLM instance, using Redis as the state storage and inter-node communication system.
VLLM Configuration
On each node, create a docker-compose.yml file:
version: '3.8'
services:
vllm:
image: vllm/vllm:latest
command: [
"--model", "meta-llama/Llama-2-7b-chat-hf",
"--host", "0.0.0.0",
"--port", "8000",
"--tensor-parallel-size", "1",
"--trust-remote-code"
]
ports:
- "8000:8000"
volumes:
- ./models:/models
restart: unless-stopped
deploy:
resources:
limits:
memory: 16G
reservations:
memory: 8G
Connecting Nodes
Use Redis to synchronize state. On each node, start Redis:
docker run -d --name redis -p 6379:6379 redis:alpine
Then configure VLLM to connect to Redis using the REDIS_URL environment variable.
Deploying OpenWebUI
OpenWebUI provides a user-friendly interface to interact with the VLLM cluster.
OpenWebUI Configuration
Create a docker-compose.yml file for OpenWebUI:
version: '3.8'
services:
openwebui:
image: openwebui/openwebui:latest
ports:
- "8080:8080"
environment:
- VLLM_API_BASE_URL=http://vllm-cluster:8000
- REDIS_URL=redis://redis:6379
volumes:
- ./data:/app/data
restart: unless-stopped
System Security
To ensure enterprise-level security:
- Integrate WireGuard VPN for secure system access.
- Use Cloudflare Tunnels to protect remote access without requiring a VPN.
- Apply UFW firewall and IP-based access control.
- Scan Docker images for vulnerabilities before deployment using Trivy.
Testing and Optimization
Run the following commands to test:
docker-compose up -d
curl http://localhost:8000/v1/models
If the response returns a list of models, the system is operational.
Alternative Solutions and Advanced Security
For enterprises requiring maximum security:
- Deploy OpenBao and Vaultwarden to manage secrets and passwords.
- Use RustDesk with a private IDRelay server for secure remote control.
Secure Remote Access Solutions
To access OpenWebUI remotely without a VPN:
- Set up Cloudflare Tunnel with a free Let's Encrypt SSL certificate.
- Enforce IP-based and time-based access policies.
- Ensure OpenWebUI is protected with OAuth or API key authentication.
Troubleshooting
- Cannot connect to VLLM: Verify port 8000 is open, Redis is running, and VLLM is active.
- OpenWebUI does not display models: Check the
VLLM_API_BASE_URLenvironment variable and network connectivity. - Low performance: Increase RAM, use GPU if available, or reduce the number of concurrent models.
Conclusion
Deploying OpenWebUI connected to a self-built VLLM cluster delivers a secure shared AI platform for enterprises. With security measures spanning from infrastructure to application layers, the system ensures safety and efficiency for internal AI operations.
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
