Maximizing AI Efficiency: Deploying DeepSeek-R1 Distilled 8B on Oracle Cloud Infrastructure ARM Free Tier using vLLM and PagedAttention

Introduction: The Intersection of Open-Source AI and Cost-Effective Enterprise Infrastructure

In the rapidly evolving landscape of large language models (LLMs), organizations are increasingly seeking methods to deploy state-of-the-art models without incurring prohibitive infrastructure costs. The release of DeepSeek-R1 Distilled 8B has emerged as a watershed moment, offering reasoning capabilities that rival significantly larger models while maintaining a lightweight footprint. However, deployment strategy remains a critical bottleneck for many enterprise teams.

Oracle Cloud Infrastructure (OCI) provides a remarkably generous Always Free Tier, offering up to 4 ARM-based Ampere Altra CPU cores and 24 GB of RAM. While traditionally overlooked for LLM inference due to the lack of dedicated GPU acceleration, the convergence of advanced model quantization, architectural optimizations, and high-performance inference engines like vLLM has made CPU-based hosting not only viable but highly efficient. This guide provides an end-to-end blueprint for deploying DeepSeek-R1 Distilled 8B on an OCI ARM Free Tier instance, utilizing vLLM's revolutionary PagedAttention algorithm to extract maximum token throughput per second.

---

Understanding the Architecture: DeepSeek-R1, ARM Ampere, and vLLM

Before initiating the technical deployment, it is vital to understand why this specific stack functions cohesively. DeepSeek-R1 Distilled 8B is built upon the robust LLaMA-3 architecture, optimized through distillation to excel at structured reasoning, mathematical computation, and coding tasks. Deploying an 8-billion parameter model in FP16 precision requires approximately 16 GB of VRAM/RAM just for model weights, making the 24 GB RAM capacity of the Oracle ARM instance an ideal, tight-knit environment.

The Challenge of Memory Bandwidth on CPUs

Unlike GPUs, which possess massive parallel memory bandwidth (often exceeding 1 TB/s), standard server CPUs rely on traditional system memory channels. This results in an inference bottleneck during the autoregressive generation phase, where memory bandwidth dictates tokens-per-second performance. To counteract this, we employ two distinct strategies:

Model Quantization: Utilizing INT4 or INT8 precision formats (such as AWQ, GPTQ, or GGUF) to drastically reduce the memory footprint and the volume of data transferred per token generation cycle.
PagedAttention via vLLM: Traditional LLM serving architectures suffer from severe memory fragmentation due to the dynamic growth of the Key-Value (KV) cache. vLLM solves this by treating the KV cache similarly to virtual memory in operating systems, cutting down on overhead and maximizing available computational capacity.

Key Insight: By utilizing PagedAttention, we eliminate physical KV cache fragmentation entirely, allowing the ARM Ampere Altra processors to focus purely on execution cycles rather than dynamic memory reallocation management.

---

Step 1: Provisioning and Configuring the OCI ARM Instance

To begin, log into your Oracle Cloud Console and provision an Always Free Ampere instance with the optimal configuration required for heavy LLM operations.

Instance Specifications

When configuring the instance, ensure you select the following parameters precisely:

Shape: VM.Standard.A1.Flex (ARM-based Ampere Altra processor).
OOCUs: Allocate 4 cores (the maximum allowed under the free tier).
Memory: Allocate 24 GB of RAM.
Operating System: Ubuntu 22.04 LTS or Ubuntu 24.04 LTS (Minimal image preferred for reduced system overhead).
Boot Volume: Minimum 100 GB (to accommodate Docker images, Python environments, and model weights).

Network and Security Configuration

Once the instance is provisioned, you must expose the port that vLLM will use to serve its OpenAI-compatible API. Navigate to your Virtual Cloud Network (VCN), select your Public Subnet, and edit the Ingress Rules to add the following entry:

Source CIDR: 0.0.0.0/0 (or your specific corporate IP range for enhanced security)
IP Protocol: TCP
Destination Port Range: 8000

Connect to your instance via SSH and update the internal firewall rules to match the security group adjustments:

sudo ufw allow 8000/tcp
sudo ufw reload

---

Step 2: Preparing the Environment and Installing vLLM on ARM64

Deploying vLLM on an ARM64 architecture requires a structured approach, as many pre-compiled wheels are optimized predominantly for x86_64 and NVIDIA CUDA platforms. We will utilize a containerized environment to ensure dependency isolation and reproducibility.

System Dependencies and Docker Setup

Execute the following commands to update the system and install Docker, which will act as our primary deployment vector:

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y curl git build-essential python3-pip python3-venv

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

Log out and log back in to apply the Docker user group permissions.

Building or Pulling the vLLM ARM64 Image

vLLM officially supports CPU-only inference via the OpenVINO or basic CPU backends. For ARM architecture, building from source or utilizing a community-maintained ARM64 image optimized for multi-threading is necessary. Ensure your configuration uses the VLLM_TARGET_DEVICE=cpu flag during execution initialization to leverage the ARM Neon instruction sets effectively.

---

Step 3: Optimizing the DeepSeek-R1 Weights for ARM

Running an unquantized 8B model on a 4-core CPU will lead to high latency. To achieve viable token production rates, we use a quantized variant. For CPU deployment via vLLM, the GGUF format (using Q4_K_M or Q8_0 precision) or AWQ formats optimized for CPU backends are highly recommended.

We will download the DeepSeek-R1 Distilled LLaMA 8B model quantized to 4 bits. This drops the memory requirement to approximately 4.8 GB, leaving ample room in our 24 GB system for the KV cache and OS operational tasks.

# Create a dedicated directory for model storage
mkdir -p ~/models
cd ~/models

# Download the quantized model weights using huggingface-cli
pip3 install huggingface_hub
huggingface-cli download DeepSeek-R1-Distilled-Llama-8B-GGUF --local-dir . --local-dir-use-symlinks False

---

Step 4: Deploying vLLM with Advanced PagedAttention Tuning

With the environment prepared and the model downloaded, we can now launch the vLLM engine. To achieve maximum throughput, we must meticulously tune the configuration flags to match the physical architecture of the Ampere Altra processor.

Execute the following command to initiate the vLLM server:

python3 -m vllm.entrypoints.openai.api_server \
    --model ~/models/DeepSeek-R1-Distilled-Llama-8B-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --block-size 16 \
    --gpu-memory-utilization 0.0 \
    --kv-cache-dtype fp16 \
    --max-model-len 4096 \
    --tensor-parallel-size 1

Deep-Dive into Optimization Flags

--block-size 16: Specifies the block size for PagedAttention. A block size of 16 aligns perfectly with the L1/L2 cache lines of ARM Ampere architecture, reducing cache misses during token evaluation.
--max-model-len 4096: Restricting the context length to 4096 tokens prevents the KV cache from scaling past the physical boundaries of the available system RAM, maintaining consistent generation velocity.
--kv-cache-dtype fp16: Ensures precision retention during long-context reasoning operations without causing memory bandwidth starvation.

---

Step 5: Benchmarking and Performance Evaluation

To validate the deployment, we can issue a structured request to our new OpenAI-compatible API endpoint using curl from an external terminal:

curl http://:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-R1-Distilled-Llama-8B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Explain quantum computing in three sentences."}],
    "temperature": 0.6
  }'

Analyzing Token Throughput

When monitoring the execution logs, pay specific attention to the avg_prompt_throughput and avg_generation_throughput metrics. By coupling 4-bit quantization with PagedAttention on the ARM Ampere architecture, you should observe highly stable token generation performance. PagedAttention actively mitigates latency spikes during concurrent requests by continuously optimizing memory layouts, allowing the CPU cores to operate at sustained 100% utilization without thermal or architectural throttling.

---

Conclusion and Next Steps

Deploying DeepSeek-R1 Distilled 8B on Oracle Cloud's ARM Free Tier proves that production-ready AI inference is achievable without costly GPU clusters. Through the strategic implementation of vLLM and PagedAttention, the inherent constraints of CPU memory bandwidth are effectively mitigated, providing an optimal architecture for development, testing, and light production workloads. To further scale this setup, consider wrapping the execution layer in a production-grade ASGI server like Uvicorn and establishing a reverse proxy using Nginx with TLS termination.