Back to articles
AI & Cloud

Running DeepSeek-R1 Distill Qwen-15B on K3s Cluster Raspberry Pi VPS ARM: Edge AI Hybrid Dot-Pha Architecture

A technical guide to deploy the DeepSeek-R1 Distill Qwen-15B large language model on a K3s cluster using Raspberry Pi and ARM VPS, optimized for Edge AI Hybrid architecture.

DPTCloud Engineering4 min read
Running DeepSeek-R1 Distill Qwen-15B on K3s Cluster Raspberry Pi VPS ARM: Edge AI Hybrid Dot-Pha Architecture

Introduction

In the growing landscape of AI, deploying large language models (LLMs) on ARM-based processors like Raspberry Pi and ARM VPS is becoming a popular trend. This guide provides a step-by-step walkthrough to run the DeepSeek-R1 Distill Qwen-15B model on a K3s cluster built from Raspberry Pi and ARM VPS devices, forming an efficient Edge AI Hybrid architecture. This setup is ideal for small businesses or individuals developing AI at home, offering cost-effectiveness and high performance.

DeepSeek-R1 Distill is a distilled and optimized version of the Qwen-15B model, delivering improved performance on ARM processors through distillation techniques. Combined with K3s – a lightweight Kubernetes distribution – it enables effective deployment and management of LLMs in edge environments.

Requirements and Preparation

Hardware

  • 2–3 Raspberry Pi 4 (4GB RAM or higher, ARM64)
  • 1 ARM VPS (e.g., Linode, Vultr, or DigitalOcean ARM) with at least 8GB RAM
  • SSD or microSD card (32GB+ capacity) for Raspberry Pi
  • 1Gbps network switch to connect all devices

Software

  • OS: Raspberry Pi OS 64-bit or Ubuntu Server ARM64
  • K3s (lightweight Kubernetes)
  • Docker
  • Python 3.9+
  • Git
  • Installed ollama or llama.cpp for LLM inference

Deploying K3s Cluster on Raspberry Pi and ARM VPS

We will build a K3s cluster with one ARM VPS as the master node and Raspberry Pi devices as worker nodes.

Step 1: Install K3s on ARM VPS (Master)

curl -sfL https://get.k3s.io | sh -

Step 2: Install K3s on Raspberry Pi (Worker)

Run the following command on each Raspberry Pi, replacing <MASTER_IP> with the ARM VPS IP address:

curl -sfL https://get.k3s.io | K3S_URL=https://<MASTER_IP>:6443 K3S_TOKEN=<TOKEN> sh -

Token can be retrieved from /var/lib/rancher/k3s/server/node-token on the VPS.

Step 3: Verify Cluster

kubectl get nodes

Ensure all nodes are in Ready state.

Running DeepSeek-R1 Distill Qwen-15B on the Cluster

We will use llama.cpp to run the model on ARM, as it offers optimized performance for ARM processors.

1. Download the Model

Fetch the DeepSeek-R1-Distill-Qwen-15B model from Hugging Face or a local repository:

git clone https://huggingface.co/DeepSeek/DeepSeek-R1-Distill-Qwen-15B.git
cd DeepSeek-R1-Distill-Qwen-15B
# Convert model to GGUF format if needed

2. Install llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)

3. Run the Model on the Cluster

Execute the following command on a worker node (Raspberry Pi):

./main -m ./models/DeepSeek-R1-Distill-Qwen-15B.gguf -p "Hello, who are you?" -n 128 --ctx-size 2048

Edge AI Hybrid Architecture: Dot-Pha

This is the Dot-Pha architecture – a hybrid model combining:

  • Edge: Raspberry Pi handles lightweight, real-time tasks (e.g., chatbots, voice recognition)
  • Cloud: ARM VPS manages heavier workloads such as fine-tuning, batch inference, or large data queries

This approach optimizes cost, reduces latency, and enhances scalability.

Verification and Testing

1. Performance Check

# Measure response time
time ./main -m ./models/DeepSeek-R1-Distill-Qwen-15B.gguf -p "Explain AI?" -n 128

2. Stability Check

  • Run continuous tasks for 24 hours
  • Monitor CPU/RAM usage and temperature on Raspberry Pi
  • Use htop or glances for real-time monitoring

Troubleshooting

  • Worker nodes cannot connect: Check firewall settings, port 6443, and K3s token
  • Memory shortage: Optimize ctx-size or use a smaller model
  • Low performance: Install libomp, use GGML with ARM NEON, or reduce CPU cores
  • Model fails to load: Verify GGUF format, file size, and path

References

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478