Back to articles
AI & Cloud

Building a Production LiteLLM AI Gateway with Virtual Keys, Rate Limits, and Redis Caching

This guide details how to deploy a production-grade LiteLLM AI Gateway with virtual keys, API rate limiting, and semantic Redis caching to optimize performance and cost for...

DPTCloud Engineering7 min read
Building a Production LiteLLM AI Gateway with Virtual Keys, Rate Limits, and Redis Caching

As large language models (LLMs) become more prevalent, self-hosting them on VPS for data control and cost efficiency is a strong trend. However, deploying a high-performance, secure, and scalable API system for multiple LLMs presents a significant challenge. This article provides a detailed guide on building a production-grade AI Gateway using LiteLLM, a prominent open-source platform, with three core features: virtual keys, rate limiting, and semantic caching using Redis.

Why an AI Gateway is Needed

Self-hosting LLMs like Qwen, Gemma, or IBM Granite on a VPS allows complete control over input and output data, mitigating security and privacy risks. However, when deploying multiple models or serving numerous users, you need a middleware layer to:

  • Ensure security by controlling access.
  • Implement rate limiting to prevent resource overuse.
  • Optimize performance by avoiding redundant calls to the same model.
  • Simplify API calls for client applications.

Introducing LiteLLM

LiteLLM is an open-source library that enables calling LLMs from various providers (OpenAI, Anthropic, Google, etc.) using a unified API. It supports features like automatic model switching, error handling, and, most importantly, a powerful gateway system. LiteLLM Gateway allows you to create a single access point to orchestrate requests to multiple LLMs, including self-hosted models.

Setting Up the Environment

To get started, you need a Linux-based VPS (Ubuntu/Debian). Here are the basic installation steps:

sudo apt update && sudo apt upgrade -y
sudo apt install python3 python3-pip git -y
pip3 install litellm redis

Setting Up Virtual Keys

Virtual keys allow you to create unique access keys for individual users or applications instead of using a single shared key. This enables detailed tracking and control of access.

In LiteLLM, you can set up virtual keys by creating a `config.json` file:

{
  "virtual_keys": {
    "user1_key": {
      "max_budget": 100,
      "model": "gemma-4-E4B-it-ultra-uncensored-heretic-GGUF",
      "max_tokens": 1000
    },
    "app2_key": {
      "max_budget": 500,
      "model": "qwen3.6-35b-a3b-uncensored-heretic-gguf",
      "max_tokens": 2000
    }
  }
}

In this example, `user1_key` has a maximum budget of 100 units (could be money or points), is restricted to the `gemma-4-E4B-it-ultra-uncensored-heretic-GGUF` model, and has a 1000-token output limit. You can configure other limits like API calls per minute.

Rate Limiting

Rate limiting is a critical security feature to prevent applications from being attacked or overusing resources. LiteLLM supports rate limiting via Redis.

To enable it, you need a running Redis server. Then, add the `rate_limiting` section to your `config.json`:

{
  "rate_limiting": {
    "redis_url": "redis://localhost:6379",
    "max_tokens_per_minute": 10000
  }
}

This will limit the total number of tokens processed per minute to 10,000. You can set limits for individual virtual keys by configuring `rate_limiting` within each `virtual_keys` section.

Semantic Caching

Semantic caching helps store input-output pairs from LLM requests and quickly return results for similar requests. This significantly reduces response time and processing costs.

LiteLLM supports caching via Redis. You need to install the `redis` library and configure the `config.json`:

{
  "cache": {
    "cache_type": "redis",
    "redis_url": "redis://localhost:6379",
    "ttl": 3600
  }
}

Here, `cache_type` is set to `redis`, `redis_url` points to your Redis server, and `ttl` (time to live) is 3600 seconds (1 hour). When a request is sent, LiteLLM calculates a hash of the input and checks if the result is already stored in Redis. If so, it returns the result immediately without calling the LLM model.

Deployment and Operations

To deploy the AI Gateway, run the following command on your VPS:

litellm --config config.json

The gateway will start listening on the default port (usually 4000). Now, client applications can call the API at `http://your-vps-ip:4000` with the corresponding virtual key to access the LLM models.

To ensure stability, you should use a process manager like `systemd` to keep the gateway running. You should also set up system monitoring to track CPU, RAM, and Redis cache resources.

Using DPTCloud with its NVMe Gen5 VPS, 10Gbps fiber uplink, and strong DDoS protection capabilities provides an ideal infrastructure for deploying such an AI Gateway, ensuring high performance and reliability.

Note: When using self-hosted LLMs like Gemma or Qwen, ensure compliance with license terms and data security regulations.
DPTCloud Tier III Infrastructure

Ready to deploy on PCIe Gen4 NVMe VPS & 10Gbps Uplinks?

Provision servers in under 60 seconds with edge Anti-DDoS protection and 99.99% SLA uptime.

Related Articles

View all