Optimizing nftables with IPSet on VPS: A High-Performance Strategy to Block Botnet Brute-Force Attacks

Introduction: The Growing Threat of Distributed Brute-Force Attacks

For modern enterprises and system administrators, Virtual Private Servers (VPS) are indispensable assets. However, their public-facing nature makes them constant targets for malicious actors. Among the most pervasive threats are automated botnets executing coordinated brute-force attacks against critical services like SSH, FTP, and custom API endpoints.

Traditional defense mechanisms, such as standard iptables rules combined with Fail2ban, often fall short under the sheer volume of a distributed attack. When thousands of unique IP addresses assault a server simultaneously, traditional firewalls evaluate rules linearly, causing a massive spike in CPU utilization and threatening overall system stability. This article explores an advanced, high-performance solution: optimizing the modern nftables framework using set structures to neutralize botnet brute-force attacks effectively without compromising VPS performance.

The Architecture of Efficiency: Why Traditional Firewalls Fail

To understand the necessity of optimization, we must analyze how Linux netfilter processes packets. Traditional firewalls read rules from top to bottom. If your defense system dynamically adds thousands of attacking IPs to a standard linear ruleset, every single incoming packet must be checked against that entire list until a match is found.

The Overhead of Linear Searching

Imagine an attacker deploying a botnet with 5,000 compromised nodes. Under a standard iptables or basic nftables configuration, a legitimate packet might undergo up to 5,000 sequential evaluations before being allowed through. This behavior induces severe latency and can lead to a self-inflicted Denial of Service (DoS) state, where the firewall consumes 100% of the VPS's CPU resources just processing rule evaluations.

The Modern Alternative: nftables and Native Sets

Introduced to replace iptables, nftables brings a more efficient infrastructure, utilizing a localized virtual machine execution environment. More importantly, it natively integrates the concept of sets (analogous to the standalone IPSet utility used with iptables). Instead of evaluating rules linearly, nftables sets utilize highly optimized hash tables. Regardless of whether the set contains 10 or 10,000 IP addresses, the lookup time remains virtually constant—an $O(1)$ complexity operation. This makes it the ideal weapon against distributed botnets.

Step-by-Step Implementation: Configuring nftables with Sets

Transitioning to an optimized nftables configuration requires a structured approach. Below is a comprehensive guide to implementing a high-performance firewall architecture on a Linux-based VPS.

Step 1: Cleaning the Slate and Installing nftables

Before deploying the new configuration, ensure that legacy firewalls do not conflict with your new setup. Disable and mask iptables-based services:

systemctl stop ufw firewalld iptables
systemctl mask ufw firewalld iptables

Next, install and enable the nftables service:

apt-get update && apt-get install -y nftables
systemctl enable nftables

Step 2: Designing the Configuration Architecture

We will construct a configuration file (typically located at /etc/nftables.conf) that defines a standard IPv4/IPv6 unified table, handles established connections efficiently, and implements a dynamic, self-expiring blacklist set.

Review the optimized structure below:

flush ruleset

table inet filter {
    # Define a high-performance set for blacklisted IPs
    set botnet_blacklist {
        type ipv4_addr
        flags dynamic, timeout
        timeout 24h
        size 65536
    }

    # Meter to track connection rates per source IP
    set ssh_meters {
        type ipv4_addr
        flags dynamic
        size 65536
    }

    chain input {
        type filter hook input priority 0; policy drop;

        # 1. Allow loopback traffic
        iifname "lo" accept

        # 2. Stateful tracking: Allow established and related traffic
        ct state established,related accept
        ct state invalid drop

        # 3. Drop blacklisted IPs instantly via $O(1)$ lookup
        ip saddr @botnet_blacklist counter drop

        # 4. ICMP (Ping) management
        ip protocol icmp accept

        # 5. Protected SSH Service with Rate Limiting
        tcp dport 22 choice_architect {
            # Add to blacklist if exceeding 5 connections per minute
            update @ssh_meters { ip saddr limit rate over 5/minute } update @botnet_blacklist { ip saddr } counter drop
            accept
        }
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Analyzing the Optimization Elements

Let us break down why the above configuration provides superior protection and efficiency:

• Early Drop Execution: The blacklist evaluation occurs immediately after allowing established connections. This ensures that blocked botnet nodes are rejected at the absolute earliest stage of packet processing, saving valuable memory and CPU cycles.
• Dynamic Timeouts: The timeout 24h flag ensures the firewall automatically purges old entries. This prevents the set from growing indefinitely, keeping the memory footprint minimal and eliminating manual maintenance.
• Stateful Efficiency: By accepting established,related connections first, the firewall bypasses complex rules for ongoing, legitimate sessions, maximizing overall throughput.

Integrating Log Analysis Tools: Fail2ban with nftables

While native rate-limiting within nftables is highly effective for basic protocol floods, sophisticated brute-force attacks often operate slowly to bypass network-layer limits. To counter this, integrating an application-layer log analyzer like Fail2ban with an nftables backend provides complete security coverage.

Instead of relying on legacy iptables actions, Fail2ban can be configured to directly insert malicious IPs into our pre-defined nftables sets. Modify your /etc/fail2ban/jail.local file to enforce this synergy:

[DEFAULT]
banaction = nftables[type=set, set=botnet_blacklist]

[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s
maxretry = 3
bantime  = 86400

With this integration, Fail2ban scans application logs for failed authentication patterns and offloads the blocking mechanism to the highly efficient nftables hash set, combining deep log inspection with wire-speed packet filtering.

Monitoring and Maintenance

An optimized firewall is not a "set-and-forget" asset. Monitoring ensures your rules perform under real-world stress. You can check the real-time status of your blacklist and track how many botnet IPs have been successfully thwarted using the following commands:

1. View Set Elements: Execute nft list set inet filter botnet_blacklist to view all currently banned IP addresses along with their remaining time-to-live (TTL).
2. Inspect Rule Counters: By adding the counter keyword to your rules, you can observe exactly how many gigabytes of malicious traffic have been dropped, providing clear data on attack vectors.

Conclusion: Securing Your Enterprise Edge

Protecting a VPS from modern, distributed botnet brute-force attacks requires moving away from outdated, linear packet filtering models. By implementing nftables combined with structured, high-performance sets, you shift your network defense from resource-intensive sequential scanning to predictable, high-speed $O(1)$ lookups. This architecture guarantees that even under a massive distributed attack, your server remains responsive, secure, and operational. Implement these changes today to secure your infrastructure against automated threats effectively.