Back to articles
AI & Cloud

Guide to Optimizing Qwen2.5-Coder 7B as a Private AI Assistant on Oracle Free Tier ARM VPS for High Performance

A detailed guide to optimizing the performance of Qwen2.5-Coder 7B on Oracle Free Tier ARM VPS for a private AI assistant, covering installation, configuration, kernel optimization,...

DPTCloud Engineering4 min read
Guide to Optimizing Qwen2.5-Coder 7B as a Private AI Assistant on Oracle Free Tier ARM VPS for High Performance

Optimizing Performance of Qwen2.5-Coder 7B on Oracle Free Tier ARM VPS

Qwen2.5-Coder 7B is a powerful open-source language model suitable for deploying a private AI assistant on self-hosted platforms. With Oracle Free Tier offering ARM64 VPS with 24GB RAM, it’s an effective and cost-free option to run this model. However, to achieve maximum performance, several optimization steps are required for the operating system, software, and configuration.

Requirements and Preparation

To get started, you need:

  • An Oracle Cloud account and registration for an ARM Free Tier VPS (approximately 24GB RAM, ARM64).
  • SSH access to the server.
  • Ubuntu 22.04 or equivalent operating system.
  • Necessary tools: git, curl, python3, pip, and Docker (if using).

Optimizing the Operating System and Kernel

Oracle Free Tier uses the default Ubuntu kernel, but for better performance in LLM inference, it’s recommended to use an optimized kernel like XanMod. This kernel improves I/O and CPU handling for heavy workloads.

Install XanMod using the following commands:

sudo apt update
sudo apt install -y wget gnupg2
wget -qO - https://xanmod.org/kernel-signing-key.asc | sudo apt-key add -
echo "deb http://xanmod.org/repos/apt jammy main" | sudo tee /etc/apt/sources.list.d/xanmod.list
sudo apt update
sudo apt install -y linux-xanmod

After installation, reboot the system and verify the new kernel:

uname -r

Installing and Running Qwen2.5-Coder 7B

Use Ollama to easily run the Qwen2.5-Coder 7B model. Ollama supports automatic model download and execution on multiple platforms, including ARM64.

Follow these steps:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b

To run as an API service, start Ollama with the listen option:

ollama serve

Or use Docker to run in an isolated environment:

docker run -d --name ollama -p 11434:11434 ollama/ollama

Configuring and Optimizing Performance

To ensure stable and high-performance operation, adjust several system parameters:

  • Reduce swap usage: Oracle Free Tier has limited swap, so disable or minimize swap usage.
  • Optimize memory: Use zram to enhance memory performance.
  • Set CPU limits for Ollama if needed.

Configure zram by installing zram-generator:

sudo apt install zram-generator
sudo systemctl enable zram-generator-defaults

Testing Performance

After installation and optimization, test performance by sending simple API requests:

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "prompt": "Write a Python function to calculate the sum of even numbers in a list."
  }'

Measure response time and monitor CPU and RAM usage to ensure the system operates stably.

Troubleshooting

  • If the model fails to load: Check network connectivity and ensure Ollama is installed correctly.
  • Memory shortage: Disable unnecessary services and reduce concurrent requests.
  • Kernel not working: Verify the current kernel and ensure the system was rebooted after installing XanMod.

References

For further information, refer to:

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478