Building a Lean, Production-Ready Mini Kafka Cluster using KRaft on a Single 2GB RAM VPS

Introduction: The Evolution of Apache Kafka

For years, Apache Kafka has been the gold standard for distributed event streaming. However, its traditional architecture came with a significant caveat: the mandatory dependency on Apache ZooKeeper. Managing a separate ZooKeeper ensemble added operational complexity, increased resource overhead, and created a dual-metadata management system that was notoriously difficult to maintain—especially for small-scale deployments or budget-conscious environments.

Enter KRaft (Kafka Raft metadata mode). Introduced to simplify Kafka's architecture, KRaft replaces ZooKeeper entirely by consolidating metadata management directly within Kafka itself using a consensus protocol based on Raft. This architectural shift drastically reduces memory consumption and footprint, making it entirely feasible to run a highly capable mini Kafka cluster on a single, low-cost Virtual Private Server (VPS) with just 2GB of RAM. In this guide, we will walk through the exact steps to configure, optimize, and run an ultra-lean KRaft-based Kafka cluster designed to extract maximum performance from constrained hardware.

Why KRaft is a Game-Changer for Lightweight Deployments

Before diving into the technical setup, it is crucial to understand why KRaft makes a 2GB RAM deployment viable. In the legacy ZooKeeper architecture, you essentially had to run two separate distributed systems. ZooKeeper required its own JVM heap, configuration, and disk I/O allocation. On a 2GB VPS, running both ZooKeeper and a Kafka broker left almost no headroom for the operating system cache, leading to severe memory thrashing and frequent out-of-memory (OOM) crashes.

With KRaft, the architecture is unified. A single process can act as both a Controller (handling metadata and consensus) and a Broker (handling data ingestion and storage). By merging these roles and eliminating the ZooKeeper JVM overhead, we can fine-tune a single Java Virtual Machine to fit comfortably within a 1GB memory footprint, leaving the remaining 1GB for the OS and page cache, which Kafka heavily relies on for high-throughput disk reads and writes.

Prerequisites and System Preparation

To follow this tutorial, you will need a clean VPS running a modern Linux distribution (such as Ubuntu 22.04 LTS or Debian 12) with the following specifications:

CPU: 1 or 2 vCPUs
RAM: 2 GB
Storage: SSD storage (10GB+ recommended for OS and message logs)

Step 1: Update the System and Install OpenJDK

Kafka runs on the Java platform. For a resource-constrained environment, we recommend OpenJDK 17 LTS or OpenJDK 21 LTS, as modern JVMs feature significant improvements in memory footprint and garbage collection efficiency.

sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-17-jdk-headless -y

Verify the installation by checking the Java version:

java -version

Step-by-Step Configuration of the Ultra-Lean KRaft Cluster

Step 2: Download and Extract Apache Kafka

Navigate to the /opt directory, download the latest stable Apache Kafka binaries (compiled with Scala), and extract them. Ensure you replace the version number with the current stable release.

cd /opt
sudo wget [https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz](https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz)
sudo tar -xzf kafka_2.13-3.6.1.tgz
sudo mv kafka_2.13-3.6.1 kafka
sudo useradd -r -d /opt/kafka -s /sbin/nologin kafka
sudo chown -R kafka:kafka /opt/kafka

Step 3: Optimizing KRaft Configuration for 2GB RAM

The secret to running Kafka smoothly on a 2GB VPS lies in strict resource allocation. We need to modify the KRaft properties file located at /opt/kafka/config/kraft/server.properties. This single file will configure our node to act as both a Controller and a Broker (a combined node).

Open the file with your preferred text editor:

sudo nano /opt/kafka/config/kraft/server.properties

Modify or verify the following critical parameters to optimize for low memory:

# The roles this server plays in the cluster
process.roles=broker,controller

# Unique node ID
node.id=1

# Quorum voters configuration (points to itself since it's a single-node cluster)
controller.quorum.voters=1@localhost:9093

# Listeners definition
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://your_vps_public_ip:9092
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
controller.listener.names=CONTROLLER

# Resource Optimization Settings
num.network.threads=2
num.io.threads=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

# Log Retention Policy to save disk space
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Deep Dive on Tuning: By reducing num.network.threads and num.io.threads to 2, we significantly lower the thread context-switching overhead and memory footprint. Setting the replication factors to 1 is mandatory since this is a single-node setup.

Step 4: Configuring JVM Heap Limits

By default, Kafka may try to allocate 1GB or more for its heap, which can cause the OS to kill the process via the OOM Killer when data starts flowing. We must explicitly cap the JVM memory allocation.

Create an environment variable file or add it directly to Kafka's environment script:

sudo nano /opt/kafka/config/kafka-env.sh

Add the following line to restrict the heap size to 512MB for Kafka, leaving the rest of the 2GB for OS page cache and background operations:

export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"

Initializing the KRaft Cluster

Unlike ZooKeeper where the state is stored externally, KRaft requires you to format the storage directories with a cluster UUID before starting the node.

Step 5: Generate a Cluster ID and Format Storage

Generate a unique cluster identifier using the built-in storage tool:

cd /opt/kafka
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-cluster-id)
echo "Your Cluster ID is: $KAFKA_CLUSTER_ID"

Next, format the log directories using this ID:

sudo -u kafka bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Managing Kafka as a Systemd Service

To ensure Kafka automatically starts on boot and restarts in case of a failure, we should wrap it in a systemd service file.

sudo nano /etc/systemd/system/kafka.service

Paste the following production-grade systemd unit configuration:

[Unit]
Description=Apache Kafka Distributed Message Broker (KRaft)
Documentation=[https://kafka.apache.org/](https://kafka.apache.org/)
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
EnvironmentFile=/opt/kafka/config/kafka-env.sh
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Reload the systemd daemon, enable the service, and start your lean Kafka cluster:

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka

Verify that Kafka is running successfully by checking its status:

sudo systemctl status kafka

Testing and Validation

To verify that our ultra-lean KRaft cluster is functioning correctly, let's create a test topic and send a sample message.

Create a Topic

/opt/kafka/bin/kafka-topics.sh --create --topic test-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Produce and Consume Messages

Open a producer console to send messages:

/opt/kafka/bin/kafka-console-producer.sh --topic test-events --bootstrap-server localhost:9092

Type a few messages like "Hello Kafka" and press Enter. In a separate terminal session, run the consumer client to read those messages in real-time:

/opt/kafka/bin/kafka-console-consumer.sh --topic test-events --from-beginning --bootstrap-server localhost:9092

Monitoring Memory Usage and Conclusion

Once your cluster is operational, monitor its memory footprint using the free -m or top commands. You should notice that the Kafka process consumes roughly 500-600MB of RAM, leaving plenty of overhead for the Linux kernel to utilize as a page cache. This ensures highly efficient, zero-copy data streaming operations even on a budget VPS tier.

By embracing KRaft mode and stripping away ZooKeeper, modern software engineers can comfortably run lightweight event-driven infrastructures for staging, testing, or small-scale production applications without breaking the bank. The era of bloated enterprise middleware is over—lean, efficient data streaming is here.