Back to articles
AI & Cloud

Running VLLM on Cheap Cloud GPU to Optimize Throughput for Small Language Model SLM in Agency Scale

A detailed guide on running VLLM on affordable Cloud GPU to optimize throughput for Small Language Models (SLM) in agency environments. Includes setup, optimization, testing, and...

DPTCloud Engineering4 min read
Running VLLM on Cheap Cloud GPU to Optimize Throughput for Small Language Model SLM in Agency Scale

Introduction

Small Language Models (SLM) are increasingly popular among small businesses and agencies seeking to deploy self-hosted AI at low cost. However, achieving high throughput during inference requires selecting a cost-effective GPU platform. This guide shows how to run VLLM—a high-performance inference framework—on affordable Cloud GPU services to optimize throughput for SLMs in agency environments.

Requirements and Preparation

To deploy successfully, you need:

Configuring and Deploying VLLM on Cheap Cloud GPU

Running VLLM on cost-effective Cloud GPUs requires optimizing both software and hardware. Follow these steps:

1. Choose the Right Platform

Select a GPU-enabled cloud service that balances cost and performance:

  • Cloud VPS GPU: Ideal for small-scale workloads requiring high flexibility.
  • Dedicated GPU Server: Best for agencies needing stable, high-throughput inference.

2. Install VLLM

Use Docker to simplify deployment. Run this command on your server:

docker run --gpus all -it --rm -p 8000:8000 --name vllm-container vllm/vllm:latest --model "Qwen-1.8B" --tensor-parallel-size 1 --max-model-len 2048 --enforce-eager

3. Optimize Throughput

Enhance throughput with these settings:

  • Max Batch Size: Increase batch size in API requests to process multiple inputs simultaneously.
  • Model Parallelism: Use --tensor-parallel-size if multiple GPUs are available.
  • Enforce Eager: Enable --enforce-eager to prevent errors in certain models.
  • Quantization: Apply 4-bit or 8-bit quantization to reduce memory usage and improve speed.

Performance Testing

Test throughput using tools like ab or wrk:

ab -n 1000 -c 10 -T application/json -H "Content-Type: application/json" -d '{"prompt": "Hello", "max_tokens": 50}' http://localhost:8000/generate

Alternative Solutions and Cost Optimization

If cheap Cloud GPU performance is insufficient, consider:

  • Running SLMs on ARM VPS with smaller models.
  • Using FerretDB to store data, reducing model load.

Solutions for Agencies

For agencies, best practices include:

  • Optimizing models based on real-world needs (smaller SLMs, fewer tokens).
  • Using Bitwarden Secrets Manager to secure API keys and sensitive data.
  • Applying DuckDB for faster input data processing.

Troubleshooting

  • GPU not detected: Verify Docker has GPU access. Run docker run --gpus all nvidia/cuda:11.0-base nvidia-smi.
  • Memory overflow: Reduce max_model_len or apply quantization.
  • Low throughput: Check batch size and number of concurrent requests.

References

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478