Building a Real-Time RAG System with LanceDB and FastEmbed on a 2GB ARM VPS

Introduction: The Enterprise Search Dilemma on a Budget

In the modern corporate landscape, data is an organization's most valuable asset. However, as internal documentation grows exponentially—spanning PDFs, wikis, project briefs, and communication logs—retrieving actionable insights becomes a significant challenge. Retrieval-Augmented Generation (RAG) has emerged as the gold standard for solving this issue, grounding Large Language Models (LLMs) in verified internal data to eliminate hallucinations.

Yet, traditional RAG architectures often demand heavy infrastructure. Deploying heavyweight vector databases and running massive embedding models can easily overwhelm standard infrastructure, requiring expensive GPU instances or high-RAM cloud servers. For small to medium enterprises (SMEs) or internal DevOps teams operating on tight budgets, this infrastructure barrier can halt AI adoption entirely.

This comprehensive guide demonstrates how to break that barrier. By combining LanceDB (an ultra-lightweight, serverless vector database) with Qdrant's FastEmbed (a highly optimized CPU embedding library), we will build a real-time, production-ready RAG system optimized specifically for a low-cost, 2GB RAM ARM64 VPS. Here is how to achieve enterprise-grade semantic search on a fraction of the typical compute budget.

Why This Stack? Overcoming the 2GB RAM Constraint

Operating within a strict 2GB memory limit requires a radical departure from traditional AI tech stacks. Standard choices like Java-based vector databases or PyTorch-dependent embedding pipelines will instantly trigger Out-Of-Memory (OOM) errors on a lightweight ARM instance. Our optimized architecture relies on two key components:

1. LanceDB: The Serverless, Disk-Based Vector Database

Unlike traditional vector databases that require persistent, heavy daemon processes running in the background, LanceDB is built on top of the Lance columnar data format. It operates server-side but acts serverless, embedding directly into your application code (similar to SQLite). Key advantages include:

Zero Memory Footprint at Rest: It consumes no RAM when your application is idle.
Disk-Based Querying: LanceDB utilizes memory-mapped files (mmap), allowing it to perform lightning-fast vector indexing and scalar filtering directly from disk storage, making it perfect for low-memory environments.
Native ARM Support: Fully optimized for standard ARM64 architecture, leveraging neon instruction sets for accelerated matrix math.

2. FastEmbed: Light, Fast, and PyTorch-Free

Generating vector embeddings usually involves heavy deep-learning frameworks like PyTorch or Transformers, which can easily draw 1.5GB+ of RAM just to load a model. FastEmbed solves this efficiently by utilizing ONNX Runtime instead of PyTorch. It is designed from the ground up for speed and minimal resource usage, offering:

Quantized Models: It defaults to 8-bit quantized embedding models (like BGE or MiniLM), reducing memory consumption by up to 4x while maintaining over 95% of the original model's accuracy.
Minimal Dependencies: By stripping out PyTorch, the runtime memory overhead drops from gigabytes to just a few megabytes.

System Architecture Overview

To ensure real-time performance on an internal document search app, the RAG pipeline is divided into two distinct, highly optimized phases: ingestion and retrieval.

Data Ingestion Pipeline:
Document Ingestion (PDF/Markdown) → Text Chunking (Recursive Character Splitting) → FastEmbed (ONNX Generation) → LanceDB (Append-only storage with IVF-PQ indexing)

During the Retrieval & Inference Phase, the user submits a natural language query. FastEmbed converts this query into a vector on the fly, LanceDB executes a hybrid search (combining vector similarity with keyword filtering), and the relevant context is passed to a lightweight local LLM or an external cost-effective API to generate the final structured answer.

Step-by-Step Implementation

Let us walk through setting up the environment, preparing the vector database, and executing real-time semantic queries on your resource-constrained ARM VPS.

Step 1: Setting Up the ARM Environment

First, ensure your Ubuntu/Debian ARM VPS is updated and equipped with the necessary build essentials. Run the following commands in your terminal:

sudo apt update && sudo apt upgrade -y
sudo apt install python3-pip python3-venv build-essential -y

Next, isolate your project by creating a virtual environment and installing our lean dependency stack:

python3 -m venv rag_env
source rag_env/bin/activate
pip install lancedb fastembed pypdf pydantic

Step 2: Initializing FastEmbed and LanceDB

Create a main application script, app.py. We will configure FastEmbed to use a lightweight, highly accurate embedding model: BAAI/bge-small-en-v1.5 (or its multilingual equivalent if handling non-English internal docs).

import lancedb
from fastembed import TextEmbedding
import os

# Initialize the ultra-lightweight FastEmbed client
# This loads an ONNX model, keeping RAM usage strictly under ~150MB
embedding_model = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Connect to LanceDB (creates the directory locally if it doesn't exist)
db_path = "./lancedb_data"
db = lancedb.connect(db_path)

Step 3: Processing and Ingesting Internal Documents

When dealing with 2GB of RAM, you cannot load large files entirely into memory. We must stream document parsing and use generators to batch data to LanceDB.

def chunk_text(text, chunk_size=500, overlap=50):
    # Simple, efficient sliding window chunking algorithm
    chunks = []
    words = text.split()
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

# Example process for internal knowledge bases
def prepare_records(doc_id, source_name, raw_text):
    text_chunks = chunk_text(raw_text)
    
    # Generate embeddings in efficient batches using FastEmbed
    embeddings = list(embedding_model.embed(text_chunks))
    
    records = []
    for idx, (chunk, vector) in enumerate(zip(text_chunks, embeddings)):
        records.append({
            "id": f"{doc_id}_{idx}",
            "vector": vector.tolist(),
            "text": chunk,
            "metadata": {"source": source_name}
        })
    return records

Step 4: Creating the Disk-Backed Table

Now, we create or open our table in LanceDB and write the vectorized records directly to disk storage.

# Sample internal documentation data
sample_text = "Our company remote work policy allows core hours between 10 AM and 4 PM EST. Expense reports must be submitted by the 25th of each month using the internal expensify portal."
records = prepare_records(doc_id="pol_001", source_name="HR_Policy_2026.txt", raw_text=sample_text)

# Create table in LanceDB
tbl = db.create_table("internal_docs", data=records, mode="overwrite")
print(f"Successfully ingested {len(records)} chunks into LanceDB.")

Step 5: Executing Low-Latency Queries

With data indexed seamlessly on disk, searching requires minimal resource utilization. LanceDB performs vector math with high efficiency, finding relevant content in milliseconds.

def search_knowledge_base(query_text, top_k=2):
    # Vectorize the incoming user query
    query_vector = list(embedding_model.embed([query_text]))[0].tolist()
    
    # Perform the vector similarity search
    results = tbl.search(query_vector).limit(top_k).to_list()
    return results

# Test query execution
user_query = "When do I need to submit my expense reports?"
search_results = search_knowledge_base(user_query)

for res in search_results:
    print(f"Score: {res['_distance']:.4f} | Source: {res['metadata']['source']}")
    print(f"Context: {res['text']}\n---")

Production Optimization for 2GB RAM Constraints

While the code above functions perfectly out of the box, running a production system on constrained ARM hardware requires ongoing adherence to strict resource management guidelines:

Batch Sizes: When calling embedding_model.embed(), never pass hundreds of documents simultaneously. Keep processing batches between 16 and 32 items to avoid spike allocations in memory.
Garbage Collection: In long-running processes or background worker queues (like Celery), explicitly invoke Python's native gc.collect() after processing massive PDF files to instantly free unreferenced heap memory.
Index Optimization: Once your table grows beyond 50,000 chunks, explicitly invoke LanceDB's IVF-PQ (Inverted File with Product Quantization) index creation. This organizes vectors into clusters, preventing brute-force searches and maintaining flat, stable memory usage regardless of dataset scale.

Conclusion: Democratizing Enterprise-Grade AI

Deploying cutting-edge AI architectures does not require high-end corporate budgets or complex multi-node cloud systems. By strategically choosing tools optimized for modern CPU design—like LanceDB and FastEmbed—you can build high-performance, real-time RAG applications that operate within tight hardware constraints.

This setup slashes operational overhead, maintains strict data privacy within your own infrastructure, and provides lightning-fast search responses for internal business operations. Stop waiting for cloud budget approvals; leverage modern ARM architectures to deploy your optimized internal document assistant today.