Kernel Hardening and Infrastructure-Level DDoS Protection for VPS

In the volatile internet environment of 2026, a VPS (Virtual Private Server) with default settings is like a house with its doors wide open in a busy city. Distributed Denial of Service (DDoS) attacks are no longer just simple floods; they have evolved into sophisticated techniques that exploit deep vulnerabilities in the TCP/IP stack. This article will guide you through "Hardening" the Linux Kernel and establishing low-level defense layers to protect your system against large-scale attacks.

1. Kernel Hardening via Sysctl Parameters

The Linux Kernel provides an interface through sysctl to adjust network parameters in real-time. By fine-tuning these values, we can optimize how the operating system handles "malicious" packets such as SYN Floods or ICMP Floods.

• TCP SYN Cookies: During a SYN Flood attack, your connection queue can become saturated. Activating SYN Cookies allows the server to continue accepting connections without immediately reserving state in memory.
• IP Spoofing Protection: Prevents the forging of source IP addresses by implementing Reverse Path Filtering.
• TCP Timestamps: Disable timestamps if not strictly necessary to avoid leaking system uptime or being targeted by sequence-based attacks.

// Interface simulating Sysctl configuration for security systems
interface KernelParam {
    key: string;
    value: number | string;
    description: string;
}

const securityParams: KernelParam[] = [
    { key: "net.ipv4.tcp_syncookies", value: 1, description: "Enable SYN Flood protection" },
    { key: "net.ipv4.conf.all.rp_filter", value: 1, description: "Anti-IP Spoofing" },
    { key: "net.ipv4.tcp_max_syn_backlog", value: 4096, description: "Increase SYN backlog queue" }
];

function applySysctl(params: KernelParam[]): void {
    params.forEach(p => {
        console.log(`Applying: sysctl -w ${p.key}=${p.value} # ${p.description}`);
    });
}

applySysctl(securityParams);
    

2. Managing TCP Connections and Resource Limits

A DDoS attack often attempts to exhaust memory resources by maintaining thousands of connections in TIME_WAIT or FIN_WAIT states. Reducing the timeout duration for these connections is absolutely critical.

If you leave the default tcp_fin_timeout at 60 seconds, an attacker only needs to send a few thousand packets per minute to completely freeze the VPS network stack. Consider reducing this value to approximately 15-20 seconds.

// Logic for checking and optimizing connection time-to-live
const networkOptimization = {
    tcpFinTimeout: 15,
    tcpKeepaliveTime: 600,
    tcpMaxTwBuckets: 1440000
};

function checkOptimization(current: number, target: number): boolean {
    if (current > target) {
        console.log("Reduction needed to free up RAM resources faster.");
        return true;
    }
    return false;
}

checkOptimization(60, networkOptimization.tcpFinTimeout); 
    

3. NFTables: The Powerful Successor to Iptables

By 2026, NFTables has fully replaced Iptables due to its superior performance and flexible syntax. NFTables allows you to create "Sets" of IP addresses to filter traffic with extremely low latency.

Feature	Iptables (Legacy)	NFTables (Recommended)
Performance	Degrades as rule count increases	Stable thanks to JIT virtual machine
Rule Organization	Linear chains	Hierarchical, supports Maps and Sets
DDoS Mitigation	Difficult to configure complex Rate Limits	Supports highly detailed packet limiting

4. Implementing Infrastructure-Level Rate Limiting

Rate Limiting is the technique of restricting the number of requests from a specific IP address within a certain timeframe. This prevents Brute-force attacks and small-scale Application Layer (L7) DDoS attacks directly from the network layers (L3/L4).

Use NFTables to limit each IP to a maximum of 10 new connection attempts per second on ports 80/443:

// Simulating data structure for Rate Limit rules
interface RateLimitRule {
    port: number;
    maxRequests: number;
    interval: string;
    action: "drop" | "reject" | "accept";
}

const httpProtection: RateLimitRule = {
    port: 443,
    maxRequests: 20,
    interval: "1s",
    action: "drop"
};

console.log(`Protecting port ${httpProtection.port}: Exceeding ${httpProtection.maxRequests} req/${httpProtection.interval} results in ${httpProtection.action}`);
    

5. Defending Against UDP Amplification Attacks

UDP is a stateless protocol often exploited to amplify traffic (e.g., via DNS or NTP). On a standard web VPS, you rarely need to open UDP ports except for DNS resolution. The golden rule is: If you don't use it, close it.

Dropping all UDP traffic or limiting UDP packet size prevents your VPS from becoming a victim or a "reflector" in amplification attacks orchestrated from other hijacked servers.

6. Tinh chỉnh Connection Hash Table (ConnTrack Tuning)

The Linux conntrack module tracks the state of all connections. During a DDoS attack, this table often overflows ("table full"), causing the server to reject even legitimate connections. We need to increase the table size and optimize the hash memory.

// Calculating ConnTrack size based on system RAM
function calculateMaxConntrack(ramGB: number): number {
    // Estimation: 1GB RAM can handle approx 65,536 entries
    const baseEntries = 65536;
    return ramGB * baseEntries;
}

const vpsRam = 8; // VPS with 8GB RAM
const maxEntries = calculateMaxConntrack(vpsRam);
console.log(`Optimal config: net.netfilter.nf_conntrack_max = ${maxEntries}`);
    

7. Utilizing Remote Traffic Filtering (Anycast Networks)

No matter how well you harden your kernel, if an attack bandwidth reaches 100Gbps while your VPS port is only 1Gbps, the system will go down. The solution is to combine infrastructure security with reverse proxy services like Cloudflare or specialized Anti-DDoS solutions at the Datacenter level.

In this setup, your VPS only accepts traffic from the security provider's IP range. Any traffic sent directly to the VPS's original IP is DROPped immediately at the edge.

8. Conclusion: The 3-Layer Security Protocol

To protect your VPS sustainably, follow this 3-layer protocol:

1. Layer 1 (Kernel): Tune sysctl to handle the network stack as efficiently as possible.
2. Layer 2 (Firewall): Use NFTables to filter bad IPs and limit connection rates.
3. Layer 3 (Monitoring): Use tools like fail2ban or Netdata to detect early warning signs of an anomaly.

Remember that security is a continuous process. A system hardened well today still requires updates and constant monitoring tomorrow!