Scaling on a Budget: Multi-Region Anycast Layer 4 Load Balancing with Envoy and Keepalived

Introduction to Modern Traffic Distribution

In today's hyper-connected digital economy, application downtime and high latency translate directly into lost revenue and diminished user trust. High Availability (HA) and geographical redundancy are no longer luxury configurations reserved exclusively for enterprises with massive infrastructure budgets. Traditionally, achieving multi-region failover and localized latency optimization required expensive cloud load balancers or proprietary hardware appliances.

However, by combining powerful open-source software with strategic network routing, it is entirely possible to construct a robust, production-ready Multi-Region Anycast Layer 4 Load Balancing architecture on top of budget Virtual Private Servers (VPS). This technical guide explores how to pair Envoy Proxy and Keepalived to deliver enterprise-grade performance, high availability, and seamless scalability at a fraction of the traditional cost.

Understanding the Architectural Components

Before diving into the configuration, it is critical to understand how the individual technologies interact to create a unified, resilient system across multiple geographical zones.

1. Anycast Routing: The Foundation of Global Redundancy

Unlike standard Unicast routing, where a single IP address maps to a single physical machine, Anycast routing allows multiple geographically dispersed servers to share the exact same IP address. The internet's Border Gateway Protocol (BGP) automatically routes client traffic to the nearest topology path (usually the closest data center). If a specific region experiences an outage, BGP routes traffic away from the failed node to the next closest active location, ensuring near-instantaneous global failover without manual DNS intervention.

2. Envoy Proxy: High-Performance Layer 4 Load Balancing

While frequently deployed as a Layer 7 reverse proxy and service mesh, Envoy is an exceptionally efficient Layer 4 (TCP/UDP) load balancer. Operating at the transport layer, Envoy handles raw TCP connections with minimal overhead, utilizing advanced threading models and non-blocking I/O to distribute incoming traffic across downstream application backends based on configurable algorithms like round-robin, least-connections, or random choice.

3. Keepalived: Local High Availability and Health Checking

Within a single data center region, relying on a single Envoy instance creates a single point of failure (SPOF). Keepalived resolves this by implementing the Virtual Router Redundancy Protocol (VRRP). It monitors the health of the local Envoy instances. If the primary load balancer fails, Keepalived dynamically shifts a local Virtual IP (VIP) to a backup node within milliseconds, providing localized redundancy before a regional BGP failover is ever triggered.

Designing the Multi-Region Architecture

To implement this setup effectively on budget VPS infrastructure, we establish a standardized node blueprint across at least two distinct geographical regions (e.g., US-East and EU-West). Each region contains:

• Two Load Balancer Nodes: Configured with Keepalived for local high availability and Envoy Proxy for traffic distribution.
• BGP Daemon: (Such as Bird or ExaBGP) configured on the VPS nodes to announce the shared Anycast IP address to the upstream provider's routers.
• Application Backends: The actual web or application servers processing the traffic.

Note: To implement true Anycast on budget VPS hosts, you must select providers that support Custom BGP Announcements or Virtual Cross-Connects (e.g., Vultr, BuyVM, or Hetzner Cloud in specific zones) allowing you to announce your own IP prefix (/24 for IPv4 or /48 for IPv6).

Step-by-Step Configuration Guide

Step 1: Implementing Local HA with Keepalived

First, we configure Keepalived on the load balancer nodes within a single region to manage the localized Virtual IP. This ensures that if an individual VPS undergoes maintenance or crashes, local operations continue uninterrupted.

On the primary load balancer node (lb-node1), create the following configuration in /etc/keepalived/keepalived.conf:

vrrp_script check_envoy {
    script "/usr/bin/pgrep envoy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass Secr3tPass!
    }
    virtual_ipaddress {
        192.168.1.100/24
    }
    track_script {
        check_envoy
    }
}

On the backup load balancer node (lb-node2), the configuration remains identical, except the state is set to BACKUP and the priority is adjusted to 100. This setup guarantees that lb-node1 takes precedence as long as the Envoy process is active.

Step 2: Configuring Envoy Proxy for Layer 4 Distribution

With localized IP redundancy handled by Keepalived, Envoy must be configured to ingest raw TCP traffic on the Virtual IP and efficiently distribute it to backend nodes. Create the envoy.yaml configuration structure as follows:

static_resources:
  listeners:
  - name: tcp_edge_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": [type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy](https://type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy)
          stat_prefix: ingress_tcp
          cluster: application_backend_cluster
  clusters:
  - name: application_backend_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: application_backend_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 10.0.0.10
                port_value: 8080
        - endpoint:
            address:
              socket_address:
                address: 10.0.0.11
                port_value: 8080
    health_checks:
    - timeout: 1s
      interval: 5s
      unhealthy_threshold: 3
      healthy_threshold: 2
      tcp_health_check: {}

This optimized configuration creates a high-performance Layer 4 TCP proxy listening globally across all interfaces on port 80. It utilizes a strict Round Robin load balancing policy to stream connections to backend nodes operating at 10.0.0.10 and 10.0.0.11, while continuously validating connection health via TCP heartbeats.

Step 3: Integrating Multi-Region Anycast via BGP

To elevate this architecture from a regional high-availability setup to a globally optimized multi-region deployment, you must link your Keepalived Virtual IP status to a local BGP daemon like Bird. When Keepalived holds the MASTER state, a script signals Bird to announce your global Anycast IP prefix to your VPS provider's upstream routers.

When both the US and EU regions actively announce the same IP prefix, upstream internet service providers (ISPs) utilize path routing optimization to send European users to the EU VPS cluster and American users to the US VPS cluster. If the entire EU datacenter goes offline, or if local Keepalived instances fail on both nodes, the BGP session drops, causing global routers to seamlessly reroute traffic to the US datacenter within seconds.

Key Maintenance and Optimization Strategies

Operating a distributed, self-managed load-balancing layer requires strict adherence to monitoring and system optimization principles:

1. Tune Linux TCP Stack Parameters: Modify /etc/sysctl.conf to support massive concurrent connection volumes by increasing maximum open files (fs.file-max), expanding local port ranges (net.ipv4.ip_local_port_range), and enabling fast socket reuse (net.ipv4.tcp_tw_reuse).
2. Implement Robust Health Check Fail-safes: Ensure that active health checks between Envoy and backend applications do not overwhelm application pools. Use lightweight, dedicated health endpoints rather than heavy database-driven pages.
3. Monitor BGP Convergence Times: Always validate how quickly your VPS infrastructure provider updates routes during a simulation failure. Convergence times under 10-30 seconds are ideal for maintaining uninterrupted user experiences.

Conclusion: Enterprise Capabilities on a Startup Budget

Building a multi-region, Anycast-enabled architecture does not require a blank check for premium cloud vendors. By combining the infrastructure flexibility of budget-friendly VPS providers with the absolute reliability of Keepalived and the blistering speed of Envoy Proxy, engineered systems can scale smoothly across regions while staying resilient against hardware faults and regional outages. This design provides structural control, exceptional performance overhead, and predictable infrastructure spending for scaling digital platforms.