Running DeepSeek-R1 Distill Qwen-15B on K3s Cluster Raspberry Pi VPS ARM: Edge AI Hybrid Dot-Pha Architecture
A technical guide to deploy the DeepSeek-R1 Distill Qwen-15B large language model on a K3s cluster using Raspberry Pi and ARM VPS, optimized for Edge AI Hybrid architecture.
Introduction
In the growing landscape of AI, deploying large language models (LLMs) on ARM-based processors like Raspberry Pi and ARM VPS is becoming a popular trend. This guide provides a step-by-step walkthrough to run the DeepSeek-R1 Distill Qwen-15B model on a K3s cluster built from Raspberry Pi and ARM VPS devices, forming an efficient Edge AI Hybrid architecture. This setup is ideal for small businesses or individuals developing AI at home, offering cost-effectiveness and high performance.
DeepSeek-R1 Distill is a distilled and optimized version of the Qwen-15B model, delivering improved performance on ARM processors through distillation techniques. Combined with K3s – a lightweight Kubernetes distribution – it enables effective deployment and management of LLMs in edge environments.
Requirements and Preparation
Hardware
- 2–3 Raspberry Pi 4 (4GB RAM or higher, ARM64)
- 1 ARM VPS (e.g., Linode, Vultr, or DigitalOcean ARM) with at least 8GB RAM
- SSD or microSD card (32GB+ capacity) for Raspberry Pi
- 1Gbps network switch to connect all devices
Software
- OS: Raspberry Pi OS 64-bit or Ubuntu Server ARM64
- K3s (lightweight Kubernetes)
- Docker
- Python 3.9+
- Git
- Installed
ollamaorllama.cppfor LLM inference
Deploying K3s Cluster on Raspberry Pi and ARM VPS
We will build a K3s cluster with one ARM VPS as the master node and Raspberry Pi devices as worker nodes.
Step 1: Install K3s on ARM VPS (Master)
curl -sfL https://get.k3s.io | sh -
Step 2: Install K3s on Raspberry Pi (Worker)
Run the following command on each Raspberry Pi, replacing <MASTER_IP> with the ARM VPS IP address:
curl -sfL https://get.k3s.io | K3S_URL=https://<MASTER_IP>:6443 K3S_TOKEN=<TOKEN> sh -
Token can be retrieved from /var/lib/rancher/k3s/server/node-token on the VPS.
Step 3: Verify Cluster
kubectl get nodes
Ensure all nodes are in Ready state.
Running DeepSeek-R1 Distill Qwen-15B on the Cluster
We will use llama.cpp to run the model on ARM, as it offers optimized performance for ARM processors.
1. Download the Model
Fetch the DeepSeek-R1-Distill-Qwen-15B model from Hugging Face or a local repository:
git clone https://huggingface.co/DeepSeek/DeepSeek-R1-Distill-Qwen-15B.git
cd DeepSeek-R1-Distill-Qwen-15B
# Convert model to GGUF format if needed
2. Install llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)
3. Run the Model on the Cluster
Execute the following command on a worker node (Raspberry Pi):
./main -m ./models/DeepSeek-R1-Distill-Qwen-15B.gguf -p "Hello, who are you?" -n 128 --ctx-size 2048
Edge AI Hybrid Architecture: Dot-Pha
This is the Dot-Pha architecture – a hybrid model combining:
- Edge: Raspberry Pi handles lightweight, real-time tasks (e.g., chatbots, voice recognition)
- Cloud: ARM VPS manages heavier workloads such as fine-tuning, batch inference, or large data queries
This approach optimizes cost, reduces latency, and enhances scalability.
Verification and Testing
1. Performance Check
# Measure response time
time ./main -m ./models/DeepSeek-R1-Distill-Qwen-15B.gguf -p "Explain AI?" -n 128
2. Stability Check
- Run continuous tasks for 24 hours
- Monitor CPU/RAM usage and temperature on Raspberry Pi
- Use
htoporglancesfor real-time monitoring
Troubleshooting
- Worker nodes cannot connect: Check firewall settings, port 6443, and K3s token
- Memory shortage: Optimize
ctx-sizeor use a smaller model - Low performance: Install
libomp, useGGMLwith ARM NEON, or reduce CPU cores - Model fails to load: Verify GGUF format, file size, and path
References
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
