DPTCloud
Back to articles
Technology Insight

Hardening VPS Security: Dropping Hacker Port Scans at the Kernel Level with NFTables Raw Tables

June 4, 2026

Introduction: The Invisible Threat of Automated Port Scanning

Every minute of every day, millions of automated bots crisscross the internet, probing IP addresses for open ports. For a Virtual Private Server (VPS), this constant background radiation of the internet isn't just a nuisance; it is the reconnaissance phase of a potential cyberattack. Hackers use port scanning to map your attack surface, identify running services, and fingerprint your operating system.

While traditional firewalls like UFW or basic Iptables rules can block unauthorized access, they often do so too late in the network stack. By the time a packet is rejected, your system has already spent precious CPU cycles processing it. For a high-traffic VPS or a small instance under a heavy distributed scan, this overhead can lead to degraded performance or even a self-inflicted Denial of Service (DoS). To truly secure your infrastructure, you need to stop hackers before they even reach the upper layers of your operating system. The answer lies at the Linux kernel level using NFTables Raw Tables.

Understanding the Problem: The Cost of Late-Stage Filtering

To appreciate why kernel-level filtering is superior, we must look at how the Linux kernel handles incoming network packets. In a standard setup, when a TCP SYN packet arrives at your network interface card (NIC), the kernel allocates a memory buffer called an sk_buff. The packet then travels upward through the network stack, passing through various hooks before finally reaching the routing decision and standard firewall rules (like the filter table).

Standard firewall configurations protect your services, but they do not protect your system resources from the sheer volume of modern scanning botnets.

If a hacker launches a high-velocity port scan using tools like Masscan or ZMap, your VPS is forced to allocate millions of sk_buff structures only to discard them moments later. This architecture shifts the bottleneck from network bandwidth to CPU utilization. To prevent this, we must intercept and drop malicious packets at the absolute entry point of the Linux network subsystem.

The Solution: NFTables and the Netdev Hook

NFTables, the modern successor to Iptables, introduces a highly efficient architecture for packet classification. One of its most powerful features is the support for the netdev family and the ingress hook.

Operating at the netdev level means your firewall rules are executed immediately after the NIC driver passes the packet to the kernel, well before the traditional prerouting layer. By implementing what is conceptually known as a "raw table" in this layer, we can evaluate packets using minimal expressions and drop port scans instantly. At this stage, the packet is processed so early that the kernel hasn't spent significant resources on it, allowing your VPS to sustain massive scanning volumes without a spike in CPU load.

Step-by-Step Guide: Building the Kernel-Level Defense

Let us walk through the process of creating a robust, high-performance NFTables configuration designed to neutralize port scanners at the kernel level. We will focus on blocking common scanning techniques, such as stealth TCP SYN scans, NULL scans, and XMAS scans.

Step 1: Prerequisites and Identifying the Network Interface

Before applying the rules, you must identify the primary network interface of your VPS. Run the following command in your terminal:

ip link show

Look for your public interface name, which is typically something like eth0, ens3, or enp0s3. We will use eth0 as our placeholder for this guide.

Step 2: Creating the NFTables Configuration Script

We will create a dedicated configuration file that defines our raw netdev table. Open or create the file at /etc/nftables.conf and structure it as follows:

#!/usr/sbin/nft -f

flush ruleset

table netdev filter_raw {
    chain ingress_hardening {
        type filter hook ingress device "eth0" priority -500; policy accept;

        # Drop invalid TCP flags (Stealth Scans)
        tcp flags & (syn|rst) == syn|rst drop
        tcp flags & (syn|fin) == syn|fin drop
        tcp flags & (fin|rst) == fin|rst drop
        tcp flags & (fin|syn|rst|psh|ack|urg) == 0 drop
        tcp flags & (fin|syn|rst|psh|ack|urg) == fin|syn|rst|psh|ack|urg drop

        # Drop fragments
        ip frag-off & 0x3fff != 0 drop
    }
}

Step 3: Analyzing the Technical Logic

Let's break down exactly what this configuration does at the kernel level:

  • The Ingress Hook: The line type filter hook ingress device "eth0" priority -500; attaches our chain directly to the early entry point of eth0. The priority of -500 ensures this runs before almost any other subsystem in the kernel.
  • Stealth Scan Mitigation: Hackers rarely scan ports using standard TCP connections. Instead, they manipulate TCP flags to see how the OS responds.
    • syn|rst and syn|fin combinations are fundamentally illegal according to RFC specifications; dropping them stops scanners from detecting closed ports.
    • A flag value of 0 represents a NULL scan, while all flags turned on represents an XMAS scan. Both are instantly dropped here.
  • Fragment Dropping: Attackers often fragment packets intentionally to bypass deep packet inspection. By dropping fragmented packets (ip frag-off) at the ingress stage, we eliminate this evasion vector entirely.

Advanced Defense: Dynamic Blacklisting with Sets

While dropping anomalous packets is effective, aggressive scanners will continue to probe your valid ports (like port 22 for SSH or port 443 for HTTPS) to look for application vulnerabilities. To counter this, we can combine our early-stage raw table with an automated blacklisting mechanism using NFTables sets.

NFTables allows us to create a dynamic, time-limited list of malicious IP addresses. If an IP attempts to connect to a restricted port or exceeds a specific connection rate, it is automatically added to a blacklist set. We can then reference this set at the netdev ingress hook to drop all subsequent traffic from that IP instantly, bypassing the rest of the network stack entirely for the duration of the ban.

Implementing the Dynamic Blacklist

Modify your configuration to include a dynamic IPv4 blacklist set with a timeout of 24 hours:

table inet filter_standard {
    set scanner_blacklist {
        type ipv4_addr
        flags dynamic, timeout
        timeout 24h
    }

    chain input {
        type filter hook input priority 0; policy accept;

        # Check against blacklist
        ip saddr @scanner_blacklist drop

        # Example: Limit SSH connection rate, ban if exceeded
        tcp dport 22 meter ssh_limit { ip saddr ct count over 5 } add @scanner_blacklist { ip saddr } drop
    }
}

By integrating this with our raw table, any IP that triggers the ssh_limit gets pushed into the scanner_blacklist. You can then add a rule at the top of your netdev ingress chain to drop packets from @scanner_blacklist before they even reach the standard input chain, maximizing performance under sustained brute-force or scanning conditions.

Verification and Monitoring

Once you have saved your configuration, enable and restart the NFTables service to apply the kernel-level protections:

systemctl enable nftables
systemctl restart nftables

To verify that your raw tables are actively intercepting and dropping hacker reconnaissance traffic, you can view the ruleset statistics in real-time. Use the following command to see packet and byte counters for your chains:

nft list ruleset

If you wish to watch the automated mitigation in action, you can append a counter to your drop rules (e.g., tcp flags & (syn|rst) == syn|rst counter drop). This will allow you to see exactly how many hundreds of thousands of malicious probes your kernel is silently discarding while your VPS remains completely stable and responsive.

Conclusion: Achieving True Network Invisibility

Securing a VPS requires moving away from reactive firewall strategies toward proactive, architecture-aware defense. By implementing NFTables Raw Tables at the netdev ingress layer, you shift the battlefield from the user space and upper kernel stacks straight to the network entry point.

This configuration effectively renders your server invisible to automated scanning suites. Illegal TCP flag manipulation drops without consuming measurable system resources, and aggressive scanners are blocked before they can ever strain your applications. In an era where automated botnets scan the entire IPv4 space multiple times a day, this level of optimization isn't just an elite luxury—it is a fundamental best practice for enterprise-grade VPS security.