Back to articles
AI & Cloud

Protect VPS from AI Crawler Bots with BunkerWeb and Advanced CrowdSec Rules 2026

This guide provides a detailed walkthrough to protect your VPS from AI crawler bots like GPTBot and ChatGPT using BunkerWeb and advanced CrowdSec rules, helping block unwanted...

DPTCloud Engineering4 min read
Protect VPS from AI Crawler Bots with BunkerWeb and Advanced CrowdSec Rules 2026

This guide provides a detailed walkthrough to protect your VPS from AI crawler bots like GPTBot, ChatGPT, and other automated search bots using BunkerWeb and advanced CrowdSec rules. It helps block unwanted access, reduce server load, and safeguard sensitive data. The guide covers installation, configuration, testing, and performance optimization, suitable for Ubuntu/Debian VPS users running AI services, APIs, or sensitive websites.

Overview of Threats from AI Crawler Bots

AI crawler bots such as GPTBot (Google), ChatGPT, Perplexity AI, and automated data extraction tools are increasingly common. These bots frequently access websites or APIs to collect data, analyze content, or index for large language models. Their activity not only consumes server resources but can also lead to data leaks, policy violations, or degraded service performance.

Particularly, VPS systems running AI services like API gateways, self-hosted LLMs, or live streaming platforms are prime targets due to high traffic and sensitive data. To protect such systems, a strong defense strategy combining BunkerWeb (a lightweight, open-source web application firewall) and CrowdSec (a community-driven bot detection platform) is essential.

Solution: BunkerWeb + Advanced CrowdSec

This solution combines two powerful tools:

  • BunkerWeb: An open-source, lightweight WAF that supports custom rule definitions, IP blocking, user-agent filtering, and behavior-based access control.
  • CrowdSec: A community-driven bot detection platform that maintains an up-to-date database of malicious IPs and can integrate with Nginx, Apache, or any server via agents.

By combining BunkerWeb and CrowdSec, a multi-layered defense is created: BunkerWeb handles initial HTTP requests, while CrowdSec analyzes bot behavior and provides automatic blocking rules.

Installation and Configuration

System requirements:

  • Ubuntu 22.04/20.04 or Debian 11/12
  • At least 2GB RAM and 2 vCPU VPS
  • Stable internet connection

1. Install BunkerWeb:

curl -s https://raw.githubusercontent.com/evilsocket/bunkerweb/master/install.sh | sudo bash

2. Configure BunkerWeb:

  • Open the admin panel at http://your-vps-ip:8080
  • Log in with default credentials: admin / admin
  • In the "Rules" section, create a rule to block specific user-agents:
if (user_agent ~ "GPTBot|ChatGPT|PerplexityAI|Python-urllib") then block

3. Install CrowdSec:

curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash\nsudo apt install crowdsec

4. Integrate CrowdSec with Nginx:

sudo cscli hub update\nsudo cscli cscli add-scenarios crowdsec/ai-crawler-bot\nsudo cscli cscli add-banlist crowdsec/ai-crawler-bot

5. Configure BunkerWeb to use CrowdSec:

  • In BunkerWeb, go to "Settings" → "Security" → "Integrations" → "CrowdSec"
  • Enter the CrowdSec API key and URL
  • Enable "Enable CrowdSec integration"

Testing and Optimization

1. Test effectiveness:

  • Send a fake request from your local machine with an AI crawler user-agent:
  • Run: curl -H "User-Agent: GPTBot/1.0" http://your-vps-ip
  • Check in BunkerWeb or CrowdSec to verify if the IP is blocked.

2. Optimize rules:

  • Refine BunkerWeb rules to block only specific bots and avoid false positives.
  • Update CrowdSec rules regularly: sudo cscli hub update
  • Set short ban durations (e.g., 1 hour) to prevent blocking legitimate IPs.

Alternative Solutions and Notes

If BunkerWeb is not suitable, consider Nginx with mod_security or Cloudflare WAF. However, BunkerWeb is lighter and easier to configure. CrowdSec benefits from rapid community updates but requires monitoring to avoid false blocking.

Assumption: A VPS running an AI API on Nginx receives frequent AI bot traffic. Without a WAF, CPU usage spikes, slowing response times. With BunkerWeb + CrowdSec, these bots are blocked early, reducing unnecessary traffic by 60–80%.

References:

Related infrastructure & engineering guides: High-RAM Cloud VPS for Self-Hosted LLMs & API Gateways

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478