VPS Network Security: Dropping Hacker Port Scans at the Kernel Level Using NFTables Raw Tables

Introduction: The Invisible Threat of Automated Reconnaissance

In the modern cloud ecosystem, any Virtual Private Server (Server) deployed with a public IP address becomes an immediate target. Within minutes of provisioning, automated bots and malicious actors initiate continuous port scanning campaigns. These scans are not merely nuisances; they represent the critical reconnaissance phase of a cyber attack, designed to map out your infrastructure, identify open ports, and fingerprint running services for known vulnerabilities.

Traditional firewalls often handle these threats at higher layers of the network stack, processing packets after the operating system has already expended computational resources. For high-traffic environments or servers under intense scanning stress, this traditional approach can lead to resource exhaustion. To achieve true resilience, security engineers must move the defense mechanism lower down the stack. This technical guide explores how to implement kernel-level mitigation against port scanning utilizing nftables and its highly efficient raw table capabilities.

Understanding the Mechanics of Port Scanning

Before implementing a defense strategy, it is crucial to understand what we are defending against. Attackers primarily use tools like Nmap, Masscan, or custom zmap scripts to send crafted TCP or UDP packets to a range of ports on your VPS. The most common technique is the TCP SYN scan (often called half-open scanning).

The Mechanism: The attacker sends a SYN packet to a target port.
The Open Port Response: If the port is open, the server responds with a SYN-ACK. The attacker then drops the connection, knowing a service is listening.
The Closed Port Response: If the port is closed, the operating system kernel responds with a RST (Reset) packet.

By analyzing these responses, hackers map your attack surface. Furthermore, advanced scans manipulate TCP flags (such as FIN, XMAS, and NULL scans) to bypass naive firewall rules. Our goal is to detect these anomalies and silently drop the packets at the earliest possible stage in the Linux kernel network stack.

Why Traditional Firewalls Fall Short (And Why Netfilter Architecture Matters)

Many administrators rely on high-level tools like UFW (Uncomplicated Firewall) or Firewalld, which act as frontends for iptables or nftables. While effective for basic access control, they typically place rules in the standard filter table, operating during the INPUT hook stage.

By the time a packet reaches the standard INPUT hook, the Linux kernel has already allocated socket buffers (sk_buff), performed route lookups, and expended precious CPU cycles. Under a massive, distributed port scan or a SYN flood, this architecture can lead to a Denial of Service (DoS) purely from kernel overhead.

The solution lies in the Netfilter Prerouting hook, specifically within a raw table type. By intercepting packets at the absolute entry point of the network stack—before connection tracking (conntrack) takes place—we can evaluate and drop malicious packets with near-zero performance penalties.

The Power of NFTables and the Raw Table

Introduced to replace iptables, nftables provides a more streamlined syntax, better performance, and enhanced modularity. One of its greatest architectural advantages is the ability to create tables that hook into the prerouting chain with a very high priority (meaning it executes early).

By configuring an nftables policy with a priority of -300 (raw bypass), we process packets before the stateful connection tracking engine handles them. If a packet matches the signature of a port scanner, it is summarily executed via the drop statement. The packet ceases to exist, no RST or ICMP unreachable packets are transmitted back, and the hacker's scanner simply registers a timeout, effectively making your VPS appear completely dark and nonexistent.

Step-by-Step Configuration: Deploying Kernel-Level Defenses

Let us walk through the practical implementation of creating a robust, low-level port scan mitigation configuration using nftables. Ensure you have root privileges on your Linux VPS (Debian 11+, Ubuntu 22.04+, or RHEL 9+ are recommended).

Step 1: Install and Enable NFTables

First, ensure that the traditional iptables service is disabled to prevent conflicts, and install the nftables package:

# Debian/Ubuntu systems
sudo apt update
sudo apt install nftables -y

# Enable the service to run at boot
sudo systemctl enable nftables

Step 2: Designing the Raw Mitigation Table

We will create a specific configuration file. Instead of modifying the monolithic /etc/nftables.conf directly, it is best practice to use an include directory or cleanly structure your script. Let's define a rule architecture that targeting invalid TCP flags, which are characteristic of aggressive scanner fingerprinting.

Create or edit your nftables ruleset to include the following infrastructure:

table ip mitigator {
    set scanned_ports { type inet_service; flags constant; elements = { 22, 80, 443 } }
    set blacklisted_ips { type ipv4_addr; flags dynamic, timeout; timeout 1h; }

    chain early_bypass {
        type filter hook prerouting priority -300; policy accept;

        # Drop packets from already blacklisted scanners instantly
        ip saddr @blacklisted_ips drop

        # TCP Flag Anomalies (Common in stealth port scanning)
        tcp flags & (fin|syn|rst|psh|ack|urg) == 0 drop
        tcp flags & (fin|syn) == fin|syn drop
        tcp flags & (syn|rst) == syn|rst drop
        tcp flags & (fin|rst) == fin|rst drop
        tcp flags & (fin|ack) == fin drop
        tcp flags & (urg|psth|fin) == urg|psth|fin drop
    }
}

Step 3: Implementing Rate Limiting and Dynamic Blacklisting

To explicitly stop sequential port scans (where an attacker hits multiple ports in rapid succession), we can track connections to unallocated ports. If an IP addresses hits ports outside our allowed services at an anomalous rate, we dynamically add them to the blacklisted_ips set directly inside the kernel execution path.

chain port_scan_detector {
        type filter hook input priority -150; policy accept;

        # If the destination port is NOT in our allowed list, track rate
        tcp dport != @scanned_ports {
            update @blacklisted_ips { ip saddr timeout 1h } 
            log prefix "[PORT SCAN DETECTED] " flags all 
            drop
        }
    }

With this configuration, the moment an automated tool attempts to probe ports that are not explicitly defined in your scanned_ports set, its source IP address is committed to a kernel memory set for 1 hour. Any subsequent packet sent by that IP address will be dropped instantly at the prerouting -300 hook, completely freeing your upper system architecture from processing the hostile traffic.

Testing the Configuration and Verifying Kernel Performance

Once you apply the rules using nft -f /etc/nftables.conf, it is vital to test the resilience of your setup. From an external machine, you can simulate an aggressive port scan using Nmap:

nmap -sS -Pn -p 1-1000 your_vps_ip

During the scan, monitor the kernel behavior and verify that the mitigation rules are executing perfectly. You can view the live contents of your dynamic blacklist set directly from the command line:

sudo nft list set ip mitigator blacklisted_ips

You will see the attacker's IP address populated within the set alongside a ticking countdown timer. If you monitor system resources using tools like htop or top during the scan, you will notice that CPU usage remains flat, validating that the raw table drop architecture successfully avoided processing overhead.

Conclusion: Proactive Architecture Over Reactive Defenses

Securing a VPS requires moving away from reactive security structures. Waiting for an application or a high-level firewall to reject an attacker's connection handshake is an inefficient use of system resources that exposes your software stack to potential zero-day vulnerabilities or denial-of-service vectors.

By utilizing the raw table capabilities of nftables, you effectively drop network scans at the absolute lowest software threshold within the Linux operating system. The packets are dropped silently, saving valuable CPU cycles, masking your server from automated reconnaissance tools, and maintaining peak operational performance for your legitimate users. Implement these rules today to ensure your cloud infrastructure remains secure, performant, and invisible to malicious scanners.