Building a Real-Time Log Analytics System for High-Traffic Web Apps Using Vector, Kafka, and ClickHouse on 2 VPS

Introduction: The Challenge of High-Traffic Log Analytics

In modern web infrastructure, high-traffic applications generate massive volumes of log data every second. Whether it is Nginx access logs, application runtime telemetry, or security audit trails, this data holds critical insights into system performance, user behavior, and potential operational anomalies. However, as traffic scales into millions of requests per day, traditional centralized logging stacks like the ELK (Elasticsearch, Logstash, Kibana) or LGTM (Loki, Grafana, Tempo, Mimir) stacks often become resource-intensive bottlenecks, requiring expensive, multi-node clusters just to keep up with ingestion.

For engineering teams working with budget constraints or optimized infrastructure, running a heavy logging stack is impractical. This blog post provides a comprehensive, production-ready architectural blueprint for building a Real-Time Log Analytics System using a lean, high-performance toolkit: Vector, Apache Kafka, and ClickHouse. Remarkably, we will demonstrate how to architect and deploy this robust pipeline across just two Virtual Private Servers (VPS) without compromising on throughput, durability, or analytical speed.

The Modern Analytics Stack: Why Vector, Kafka, and ClickHouse?

To achieve high throughput on minimal hardware, we must select tools optimized for performance, memory efficiency, and specific roles within the data pipeline. Our chosen stack eliminates the heavy JVM overhead often associated with traditional logging tools.

Vector (by Datadog): A ultra-fast, memory-safe log collector written in Rust. It replaces heavy agents like Logstash or Fluentd, consuming minimal CPU and RAM while handling tens of thousands of events per second per core.
Apache Kafka: The industry-standard distributed event streaming platform. Kafka acts as a highly durable, fault-tolerant message buffer, decoupling log collection from storage and ensuring data is never lost during sudden traffic spikes or downstream maintenance windows.
ClickHouse: An open-source, column-oriented DBMS designed for Online Analytical Processing (OLAP). ClickHouse compresses data aggressively and executes analytical queries across billions of rows in milliseconds, making it the perfect engine for real-time dashboards and log inspection.

Architecting for a 2-VPS Topology

Deploying an enterprise-grade pipeline on just two servers requires a strategic distribution of workloads to prevent resource contention and eliminate single points of failure for data ingestion.

VPS 1: The Edge Ingestion & Message Queue Layer

The first server handles the immediate pressure of incoming public traffic and initial data buffering. It hosts:

The high-traffic web application and its reverse proxy (e.g., Nginx).
Vector (Agent mode): Configured to scrape log files locally, parse raw strings into structured JSON, and immediately stream them out.
Apache Kafka & KRaft: Serves as the ingestion buffer, safely storing data on disk as it arrives from Vector. Utilizing Kafka's KRaft mode eliminates the need for an independent ZooKeeper cluster, conserving valuable memory.

VPS 2: The Analytical & Storage Layer

The second server is dedicated to heavy computing, long-term storage, and analytical query execution. It hosts:

Vector (Aggregator mode): Acts as a consumer that pulls batches of data from Kafka, performs any global transformations, and manages bulk writes.
ClickHouse Server: Receives structured batches from the Vector aggregator, indexes the data columnarly, and handles incoming analytical queries from business intelligence tools or dashboards.

By isolating Kafka on VPS 1 and ClickHouse on VPS 2, we ensure that intensive analytical queries running on ClickHouse do not starve the Kafka ingestion buffer of CPU or I/O cycles, maintaining system stability during traffic surges.

Step-by-Step Implementation Guide

Step 1: Configuring Vector Agent on VPS 1

First, we configure the local Vector agent on VPS 1 to tail Nginx access logs, parse them using a structured pattern, and forward them to our Kafka broker. Below is a conceptual representation of the vector.toml configuration:

[sources.nginx_logs] type = "file" include = ["/var/log/nginx/access.log"] [transforms.parse_nginx] type = "remap" inputs = ["nginx_logs"] source = ''' . = parse_nginx_log!(.message, "combined") .timestamp = parse_timestamp!(.timestamp, format: "%d/%b/%Y:%H:%M:%S %z") ''' [sinks.kafka_buffer] type = "kafka" inputs = ["parse_nginx"] bootstrap_servers = "localhost:9092" topic = "web-traffic-logs" compression = "snappy"

Using Snappy compression ensures that data transmitted over the network between VPS 1 and VPS 2 is minimized, optimizing bandwidth usage without sacrificing CPU cycles.

Step 2: Preparing the Kafka Topic

With Kafka running on VPS 1, we create a dedicated topic optimized for our log stream. Because this server acts as a buffer, we can set a strict retention policy (e.g., 24 to 48 hours) to prevent disk space exhaustion:

kafka-topics.sh --create --topic web-traffic-logs --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Using multiple partitions allows for parallel processing down the line if we decide to scale our consumer instances.

Step 3: Designing the ClickHouse Schema on VPS 2

ClickHouse requires a well-structured table using an appropriate storage engine to handle high-volume analytical workloads efficiently. We connect to ClickHouse on VPS 2 and create the target table utilizing the MergeTree engine:

CREATE TABLE sys_logs.nginx_analytics ( timestamp DateTime, remote_addr String, request_method LowCardinality(String), request_uri String, status UInt16, body_bytes_sent UInt64, http_referer String, http_user_agent String ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY (status, request_method, timestamp);

Optimization Note: Wrapping request_method in a LowCardinality data type optimizes dictionary encoding for strings with repeating values, drastically accelerating query performance and reducing storage footprints.

Step 4: Setting up the Vector Aggregator on VPS 2

The final link in our data pipeline is the Vector aggregator running on VPS 2. This instance pulls events from VPS 1's Kafka broker and streams them into ClickHouse using optimized bulk inserts:

[sources.kafka_in] type = "kafka" bootstrap_servers = "VPS1_IP_ADDRESS:9092" topics = ["web-traffic-logs"] group_id = "vector-clickhouse-consumer" [sinks.clickhouse_out] type = "clickhouse" inputs = ["kafka_in"] endpoint = "[http://127.0.0.1:8123](http://127.0.0.1:8123)" database = "sys_logs" table = "nginx_analytics" batch.max_events = 10000 batch.timeout_secs = 5

Configuring a batch.max_events threshold ensures that Vector batches rows together before sending them to ClickHouse. Never insert rows into ClickHouse one by one; batching is critical to maintaining low disk I/O write amplification and ensuring system stability.

Performance Optimization & Best Practices

Operating a high-throughput system on minimal hardware leaves little room for inefficient configurations. Implement these operational strategies to guarantee smooth real-time analytics execution:

Tune ClickHouse Batch Sizes: ClickHouse thrives on large block writes. Ensure your Vector aggregator batches contain between 5,000 and 20,000 rows, or flushes every 2 to 5 seconds.
Leverage Kafka Compression: Enable snappy or lz4 compression at the producer level (Vector Agent). This significantly lowers network payloads between your two servers.
Monitor Memory Limits: Limit Kafka's JVM heap size to fit comfortably within VPS 1's memory allocation without triggering Linux OOM (Out of Memory) killers. Since Kafka relies heavily on the OS page cache for performance, a small heap size (e.g., 1GB to 2GB) is often sufficient.
Index Strategy: Choose your ClickHouse ORDER BY key carefully. Sort by columns frequently used in your dashboard filters (such as status codes, endpoints, or timestamps) to ensure query scanners skip irrelevant data blocks completely.

Conclusion: Enterprise Capabilities on a Lean Budget

By bypassing resource-heavy legacy logging architectures and combining the performance profiles of Vector, Kafka, and ClickHouse, engineering teams can successfully run an enterprise-grade real-time log analytics system on just two cost-effective VPS instances. This topology guarantees structured real-time metrics, system durability during high traffic surges, and blazing-fast analytical query capabilities for your production web applications.