Protect VPS from AI Crawler Bots with BunkerWeb and Advanced CrowdSec Rules 2026
This guide provides a detailed walkthrough to protect your VPS from AI crawler bots like GPTBot and ChatGPT using BunkerWeb and advanced CrowdSec rules, helping block unwanted...
This guide provides a detailed walkthrough to protect your VPS from AI crawler bots like GPTBot, ChatGPT, and other automated search bots using BunkerWeb and advanced CrowdSec rules. It helps block unwanted access, reduce server load, and safeguard sensitive data. The guide covers installation, configuration, testing, and performance optimization, suitable for Ubuntu/Debian VPS users running AI services, APIs, or sensitive websites.
Overview of Threats from AI Crawler Bots
AI crawler bots such as GPTBot (Google), ChatGPT, Perplexity AI, and automated data extraction tools are increasingly common. These bots frequently access websites or APIs to collect data, analyze content, or index for large language models. Their activity not only consumes server resources but can also lead to data leaks, policy violations, or degraded service performance.
Particularly, VPS systems running AI services like API gateways, self-hosted LLMs, or live streaming platforms are prime targets due to high traffic and sensitive data. To protect such systems, a strong defense strategy combining BunkerWeb (a lightweight, open-source web application firewall) and CrowdSec (a community-driven bot detection platform) is essential.
Solution: BunkerWeb + Advanced CrowdSec
This solution combines two powerful tools:
- BunkerWeb: An open-source, lightweight WAF that supports custom rule definitions, IP blocking, user-agent filtering, and behavior-based access control.
- CrowdSec: A community-driven bot detection platform that maintains an up-to-date database of malicious IPs and can integrate with Nginx, Apache, or any server via agents.
By combining BunkerWeb and CrowdSec, a multi-layered defense is created: BunkerWeb handles initial HTTP requests, while CrowdSec analyzes bot behavior and provides automatic blocking rules.
Installation and Configuration
System requirements:
- Ubuntu 22.04/20.04 or Debian 11/12
- At least 2GB RAM and 2 vCPU VPS
- Stable internet connection
1. Install BunkerWeb:
curl -s https://raw.githubusercontent.com/evilsocket/bunkerweb/master/install.sh | sudo bash
2. Configure BunkerWeb:
- Open the admin panel at
http://your-vps-ip:8080 - Log in with default credentials: admin / admin
- In the "Rules" section, create a rule to block specific user-agents:
if (user_agent ~ "GPTBot|ChatGPT|PerplexityAI|Python-urllib") then block
3. Install CrowdSec:
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash\nsudo apt install crowdsec
4. Integrate CrowdSec with Nginx:
sudo cscli hub update\nsudo cscli cscli add-scenarios crowdsec/ai-crawler-bot\nsudo cscli cscli add-banlist crowdsec/ai-crawler-bot
5. Configure BunkerWeb to use CrowdSec:
- In BunkerWeb, go to "Settings" → "Security" → "Integrations" → "CrowdSec"
- Enter the CrowdSec API key and URL
- Enable "Enable CrowdSec integration"
Testing and Optimization
1. Test effectiveness:
- Send a fake request from your local machine with an AI crawler user-agent:
- Run:
curl -H "User-Agent: GPTBot/1.0" http://your-vps-ip - Check in BunkerWeb or CrowdSec to verify if the IP is blocked.
2. Optimize rules:
- Refine BunkerWeb rules to block only specific bots and avoid false positives.
- Update CrowdSec rules regularly:
sudo cscli hub update - Set short ban durations (e.g., 1 hour) to prevent blocking legitimate IPs.
Alternative Solutions and Notes
If BunkerWeb is not suitable, consider Nginx with mod_security or Cloudflare WAF. However, BunkerWeb is lighter and easier to configure. CrowdSec benefits from rapid community updates but requires monitoring to avoid false blocking.
Assumption: A VPS running an AI API on Nginx receives frequent AI bot traffic. Without a WAF, CPU usage spikes, slowing response times. With BunkerWeb + CrowdSec, these bots are blocked early, reducing unnecessary traffic by 60–80%.
References:
- Comprehensive Guide to Deploying AI Video Translator with Automatic Dubbing on Ubuntu VPS
- Comprehensive Guide to Building a 24/7 Live Stream AI Avatar System on Linux VPS with Linly Talker and WebRTC
Related infrastructure & engineering guides: High-RAM Cloud VPS for Self-Hosted LLMs & API Gateways
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
