Guide to Optimizing Qwen2.5-Coder 7B as a Private AI Assistant on Oracle Free Tier ARM VPS for High Performance
A detailed guide to optimizing the performance of Qwen2.5-Coder 7B on Oracle Free Tier ARM VPS for a private AI assistant, covering installation, configuration, kernel optimization,...
Optimizing Performance of Qwen2.5-Coder 7B on Oracle Free Tier ARM VPS
Qwen2.5-Coder 7B is a powerful open-source language model suitable for deploying a private AI assistant on self-hosted platforms. With Oracle Free Tier offering ARM64 VPS with 24GB RAM, it’s an effective and cost-free option to run this model. However, to achieve maximum performance, several optimization steps are required for the operating system, software, and configuration.
Requirements and Preparation
To get started, you need:
- An Oracle Cloud account and registration for an ARM Free Tier VPS (approximately 24GB RAM, ARM64).
- SSH access to the server.
- Ubuntu 22.04 or equivalent operating system.
- Necessary tools: git, curl, python3, pip, and Docker (if using).
Optimizing the Operating System and Kernel
Oracle Free Tier uses the default Ubuntu kernel, but for better performance in LLM inference, it’s recommended to use an optimized kernel like XanMod. This kernel improves I/O and CPU handling for heavy workloads.
Install XanMod using the following commands:
sudo apt update
sudo apt install -y wget gnupg2
wget -qO - https://xanmod.org/kernel-signing-key.asc | sudo apt-key add -
echo "deb http://xanmod.org/repos/apt jammy main" | sudo tee /etc/apt/sources.list.d/xanmod.list
sudo apt update
sudo apt install -y linux-xanmod
After installation, reboot the system and verify the new kernel:
uname -r
Installing and Running Qwen2.5-Coder 7B
Use Ollama to easily run the Qwen2.5-Coder 7B model. Ollama supports automatic model download and execution on multiple platforms, including ARM64.
Follow these steps:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b
To run as an API service, start Ollama with the listen option:
ollama serve
Or use Docker to run in an isolated environment:
docker run -d --name ollama -p 11434:11434 ollama/ollama
Configuring and Optimizing Performance
To ensure stable and high-performance operation, adjust several system parameters:
- Reduce swap usage: Oracle Free Tier has limited swap, so disable or minimize swap usage.
- Optimize memory: Use
zramto enhance memory performance. - Set CPU limits for Ollama if needed.
Configure zram by installing zram-generator:
sudo apt install zram-generator
sudo systemctl enable zram-generator-defaults
Testing Performance
After installation and optimization, test performance by sending simple API requests:
curl -X POST http://localhost:11434/api/generate \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-coder:7b",
"prompt": "Write a Python function to calculate the sum of even numbers in a list."
}'
Measure response time and monitor CPU and RAM usage to ensure the system operates stably.
Troubleshooting
- If the model fails to load: Check network connectivity and ensure Ollama is installed correctly.
- Memory shortage: Disable unnecessary services and reduce concurrent requests.
- Kernel not working: Verify the current kernel and ensure the system was rebooted after installing XanMod.
References
For further information, refer to:
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
