Building a Real-Time RAG System on a 2GB ARM VPS with LanceDB and FastEmbed

Introduction: The Challenge of Cost-Effective RAG

In the era of enterprise artificial intelligence, Retrieval-Augmented Generation (RAG) has emerged as the gold standard for querying internal company documents. However, conventional wisdom dictates that hosting a production-grade RAG pipeline requires heavy infrastructure: massive vector databases running in dedicated clusters and power-hungry GPUs to handle embeddings. For small to medium enterprises (SMEs) or internal DevOps teams, this infrastructure overhead can be financially prohibitive.

But what if you could run a production-ready, real-time RAG system on a cheap ARM-based VPS with only 2GB of RAM? Thanks to modern advancements in lightweight, embedded technologies, this is no longer a theoretical exercise. By pairing LanceDB (a serverless, server-embedded vector database) with FastEmbed (a highly optimized, lightweight embedding generation library by Qdrant), we can build an incredibly fast, highly efficient document retrieval engine that operates comfortably within strict memory constraints.

The Core Stack: Why LanceDB and FastEmbed?

To understand how we can achieve real-time search on a 2GB RAM budget, we must examine the architectural efficiency of our chosen tools.

1. LanceDB: Serverless and Disk-Centric Vector Storage

Traditional vector databases like Milvus or Qdrant (in-memory mode) require significant memory footprints because they keep indexes cached in RAM for speed. LanceDB turns this paradigm on its head. It is built on top of the Lance columnar data format, designed for high-performance AI workloads.

Zero-Overhead Architecture: LanceDB is embedded directly into your application process (like SQLite), eliminating the RAM overhead of running a separate database server daemon.
Disk-Based Indexing (IVF-PQ): LanceDB uses disk-based Approximate Nearest Neighbor (ANN) indexes. It achieves sub-millisecond query latency by reading directly from SSD/NVMe storage rather than saturating your precious 2GB of RAM.

2. FastEmbed: Light, Fast, and CPU-Optimized Embeddings

Generating text embeddings usually requires loading heavy Hugging Face transformers into memory, which can easily crash a 2GB RAM server. FastEmbed solves this by using ONNX Runtime instead of PyTorch.

Minimal Memory Footprint: By utilizing highly quantized models (such as quantized BGE-small or MiniLM), FastEmbed can generate high-quality vector embeddings using less than several hundred megabytes of RAM.
ARM Native Efficiency: ONNX Runtime is heavily optimized for CPU execution, taking full advantage of the multi-core architecture found in modern ARM64 VPS providers (like Ampere Altra instances on Oracle Cloud, AWS Graviton, or Hetzner).

Architecture Overview

Our lightweight RAG workflow follows a streamlined, linear execution path designed to maximize efficiency:

Data Ingestion: Internal documents (PDFs, Markdown, text files) are parsed and broken into smaller chunks.
Embedding Generation: FastEmbed processes these text chunks natively on the ARM CPU using a quantized model, converting text into dense vectors.
Vector Storage & Indexing: The vectors and their raw text metadata are written directly into a LanceDB table stored on the local disk.
Real-Time Querying: When a user submits a query, FastEmbed vectorizes it, and LanceDB performs a hybrid search (vector + keyword) to fetch relevant context within milliseconds. This context is then formatted and passed to a lightweight LLM API (such as OpenAI, Claude, or a self-hosted small model) to generate the final response.

Step-by-Step Implementation Guide

Let us walk through setting up this production pipeline using Python. Ensure your environment is running Python 3.10+ on your ARM VPS.

Step 1: Installing the Lightweight Dependencies

First, SSH into your ARM VPS and install the required packages. Notice how we do not need to install heavy frameworks like PyTorch or Docker containers for vector databases.

pip install lancedb fastembed pydantic

Step 2: Initializing FastEmbed and LanceDB

We write a script to initialize our embedded database and our CPU-optimized embedding engine. We will use the highly accurate yet lightweight BAAI/bge-small-en-v1.5 model.

import lancedb
from fastembed import TextEmbedding
import os

# Initialize FastEmbed with the small, quantized model
# This model takes minimal RAM and runs beautifully on ARM CPUs
embedding_model = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Connect to LanceDB (it will create a local directory as the database)
db_path = "./lancedb_data"
db = lancedb.connect(db_path)

Step 3: Creating the Document Ingestion Pipeline

LanceDB integrates seamlessly with Python generators, allowing us to stream documents and embed them on the fly without loading the entire dataset into RAM.

def prepare_documents(documents, model):
    # Extract texts for batch embedding
    texts = [doc["text"] for doc in documents]
    embeddings = list(model.embed(texts))
    
    # Yield rows with vectors and metadata
    for i, doc in enumerate(documents):
        yield {
            "vector": embeddings[i],
            "text": doc["text"],
            "metadata": doc["metadata"]
        }

# Sample internal company data
internal_docs = [
    {"text": "Our company remote work policy allows working from anywhere within the country.", "metadata": "HR-01"},
    {"text": "Production database backups are executed daily at 02:00 UTC and retained for 30 days.", "metadata": "OPS-42"},
    {"text": "Expense reports must be submitted by the 25th of each month for reimbursement.", "metadata": "FIN-09"}
]

# Create or open a table in LanceDB
table_name = "company_knowledge"
data_stream = prepare_documents(internal_docs, embedding_model)

table = db.create_table(table_name, data=data_stream, mode="overwrite")
print(f"Table '{table_name}' successfully created and indexed!")

Step 4: Executing Real-Time Queries

When a query arrives, we embed the query text using the same model and use LanceDB's highly optimized vector search capabilities to find the match.

def search_knowledge_base(query_text, model, table, top_k=2):
    # Embed the user query
    query_vector = list(model.embed([query_text]))[0]
    
    # Search the table natively using the vector
    results = table.search(query_vector).limit(top_k).to_list()
    return results

# Test a real-time query
query = "How often are production databases backed up?"
search_results = search_knowledge_base(query, embedding_model, table)

for res in search_results:
    print(f"Score: {res['_distance']:.4f} | Source: {res['metadata']} | Text: {res['text']}")

Optimizing for 2GB RAM on ARM Architectures

To guarantee absolute system stability and prevent Linux's Out-Of-Memory (OOM) killer from terminating your process, you should implement these specific host-level and code-level optimizations on your 2GB ARM VPS.

Configure a Swap File: Always allocate at least 2GB of Swap memory on your VPS SSD. While swap is slower than RAM, LanceDB's memory-mapped files (mmap) utilize it gracefully to handle temporary spikes during indexing without crashing the application.
Control Batch Sizes: When embedding documents via FastEmbed, set your batch size to a conservative number (e.g., batch_size=16 or 32). This keeps memory spikes predictable and linear.
Garbage Collection: If you are running an ingestion cron job, explicitly invoke Python's gc.collect() after processing large document batches to release unreferenced string memory immediately.

Conclusion: High Performance Doesn't Mean High Cost

By shifting away from traditional, resource-heavy client-server architectures to the embedded ecosystem of LanceDB and FastEmbed, we democratize access to advanced AI capabilities. We have successfully proven that a real-time, production-capable RAG system can run smoothly on an ARM VPS with a mere 2GB of RAM, driving infrastructure costs down to pennies a day.

This architecture is perfectly suited for internal documentation tools, customer service chatbots, or localized knowledge management bases. You no longer need thousands of dollars in cloud credits to build high-performance AI systems—just smart engineering, optimal file formats, and highly efficient execution runtimes.