Maximizing DeepSeek-R1 Inference on AMD EPYC VPS: Advanced Techniques for Peak Token/s

Introduction: The Case for CPU-Bound DeepSeek-R1 Inference

The release of the DeepSeek-R1 series, particularly its distilled variants ranging from 1.5B to 70B parameters, has democratized access to frontier-class reasoning models. While GPUs remain the default choice for large-scale enterprise AI deployments, they introduce substantial cost overheads and availability bottlenecks. For many enterprise applications, background processing tasks, and localized deployments, running inference on high-performance CPU servers is not just a cost-saving measure—it is a strategic necessity.

AMD EPYC processors, renowned for their high core counts, massive memory bandwidth, and robust architecture, present an ideal platform for cost-effective LLM deployment. However, out-of-the-box performance often leaves much to be desired. Achieving acceptable tokens per second (Token/s) on a Virtual Private Server (VPS) powered by AMD EPYC requires a deep understanding of hardware architecture, memory bottlenecks, and compiler-level optimizations. This technical guide outlines the exact engineering methodologies required to maximize DeepSeek-R1 (Distilled) inference speeds on AMD EPYC virtualized infrastructure.

Understanding the Bottlenecks: Memory Bandwidth vs. Compute

Before implementing optimizations, we must identify the primary limiting factor in CPU-based LLM inference. Large Language Models during the autoregressive generation phase (the decoding phase) are overwhelmingly memory-bandwidth bound, not compute-bound.

Every time a single token is generated, the entire weights matrix of the model must be fetched from the system RAM to the CPU caches. For an unquantized DeepSeek-R1 8B model running at 16-bit precision, this requires transferring approximately 16 GB of data per token. If your server's memory bandwidth is limited to 50 GB/s, your theoretical maximum throughput is strictly capped at roughly 3 Token/s, regardless of how many CPU cores you throw at the problem. Therefore, our optimization strategy must focus heavily on reducing data transfer sizes and maximizing memory bus utilization.

Step 1: Selecting and Quantizing the Model (GGUF Format)

To overcome the memory bandwidth bottleneck, quantization is mandatory. For CPU inference, the GGUF (GPT-Generated Unified Format) container, managed via tools like llama.cpp, is the industry standard. It allows for efficient execution and highly optimized execution kernels specifically designed for CPU instruction sets.

When deploying DeepSeek-R1 Distilled variants (such as the Llama- or Qwen-based architectures), we must balance perplexity (model accuracy) with inference speed. The following table provides a recommended framework for AMD EPYC setups:

DeepSeek-R1-Distill-Qwen-8B: Use Q4_K_M or Q5_K_M quantization. This reduces the memory footprint to ~5-6 GB, allowing the model weights to fit into smaller cache boundaries and significantly accelerating data transfer.
DeepSeek-R1-Distill-Llama-70B: Use Q3_K_L or Q4_K_M. This requires a higher-tier VPS with at least 64GB of RAM but offers state-of-the-art reasoning capabilities.

Engineering Note: Avoid 2-bit quantizations (e.g., Q2_K) as they cause severe degradation in DeepSeek's complex chain-of-thought reasoning capabilities. The Q4_K_M variant remains the sweet spot for performance per token.

Step 2: Optimizing the Execution Environment with Llama.cpp

The foundational software engine for our deployment is llama.cpp. To unlock the full potential of AMD EPYC, the binaries must be compiled natively on the target host to leverage specific hardware instruction sets, specifically AVX2 and FMA (Fused Multiply-Add), which are highly optimized in AMD Zen architectures.

Native Compilation Steps

Avoid generic Docker images or pre-compiled binaries. Instead, clone the repository and compile using modern build tools:

sudo apt update && sudo apt install build-essential cmake git -y
git clone [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)
cd llama.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release

By compiling natively, the compiler detects the capabilities of your underlying EPYC virtual CPU (vCPU) and optimizes vector mathematical operations accordingly.

Step 3: Advanced Threading and Core Affinity Configuration

A common mistake in VPS deployments is assigning all available virtual cores to the inference engine. On an AMD EPYC processor, allocating too many threads can drastically slow down performance due to thread synchronization overhead and cache thrashing.

The Golden Rule of CPU Threading

For optimal token throughput, set the number of execution threads (--threads or -t) equal to the number of physical CPU cores allocated to your VPS, not the number of logical threads (Hyper-Threading/SMT). If your VPS has 8 vCPUs representing 4 physical cores with SMT enabled, your thread count should be configured to 4.

NUMA Awareness and Execution Isolation

AMD EPYC processors utilize a Multi-Chip Module (MCM) architecture, structured around Non-Uniform Memory Access (NUMA) domains. Accessing memory located in a remote NUMA node introduces severe latency penalties.

If you are operating a large VPS instance or a dedicated bare-metal EPYC server, you must force the operating system to allocate memory and compute within the same NUMA node. Use the numactl utility to wrap your inference command:

numactl --cpunodebind=0 --membind=0 ./llama-cli -m deepseek-r1-distill-qwen-8b-Q4_K_M.gguf -p "Why is the sky blue?" -t 8

This ensures that the model weights are loaded into the memory channels directly connected to the executing CPU cores, slashing latency and maximizing Token/s.

Step 4: Memory Management and Kernel Tuning

Linux default memory behaviors are optimized for general-purpose applications, not memory-bound LLM execution. We must adjust system parameters to avoid swap usage and memory fragmentation.

1. Enable HugePages

HugePages reduce the overhead of translation lookaside buffer (TLB) misses by increasing the page size from the standard 4KB to 2MB or 1GB. This is highly effective for large memory footprints typical of LLM weights.

sudo sysctl -w vm.nr_hugepages=3000

Configure llama.cpp to use locked memory by passing the --mlock flag. This prevents the OS from swapping out parts of the model to disk, maintaining consistent token generation speed over long sessions.

2. Adjust Memory Allocator Behavior

Force the system to prioritize aggressive memory allocation by setting the virtual memory overcommit settings via sysctl:

sudo sysctl -w vm.overcommit_memory=1

Benchmarking and Verification: Measuring Token/s

Once the optimizations are applied, execute a standard generation benchmark to verify performance metrics. Run the llama-bench utility included in the repository tools:

./llama-bench -m deepseek-r1-distill-qwen-8b-Q4_K_M.gguf -n 128 -b 1

Pay close attention to the Eval Time (tokens per second during generation). On a properly tuned AMD EPYC VPS with DDR5 memory channels and AVX2 optimizations, you should expect a 2x to 3x improvement in Token/s compared to an unoptimized, stock configuration. For an 8B model, a throughput of 15-25 Token/s is fully achievable on mid-tier EPYC systems, matching or exceeding human reading speed.

Conclusion: Production-Ready CPU Inference

Optimizing DeepSeek-R1 (Distilled) on AMD EPYC VPS architectures proves that enterprise-grade AI deployment does not always require prohibitive capital expenditure on GPU infrastructure. By systematically targeting the primary memory bandwidth constraint through **model quantization**, **native compilation**, **NUMA pinning**, and **thread isolation**, systems engineers can unlock high-performance, predictable inference speeds directly on standard enterprise CPUs.

As reasoning models grow more efficient, the ability to run them on CPU-only infrastructure will remain an invaluable asset for building scalable, sustainable, and highly resilient AI application architectures.