Building an Automated Anti-Scraping System: Combining CrowdSec and OpenResty on a VPS to Block Bad Bots
Introduction: The Growing Threat of Malicious Web Scraping
In the modern digital economy, data is one of the most valuable assets a business possesses. However, this value also makes it a prime target for automated exploitation. While legitimate bots like Googlebot are essential for search engine visibility, a massive influx of 'Bad Bots' operates in the shadows. These malicious scripts continuously scrape proprietary data, harvest pricing intelligence, steal copyrighted content, and aggressively drain server resources.
For businesses hosting applications on Virtual Private Servers (VPS), standard rate limiting or static IP blacklisting is no longer sufficient. Sophisticated scraping networks utilize distributed residential proxies to bypass traditional defenses. To protect your digital infrastructure without degrading the experience of legitimate users, you need a dynamic, real-time intrusion prevention system. This guide provides a comprehensive framework for building an automated anti-scraping system by combining OpenResty—a high-performance web platform built on Nginx—with CrowdSec, a modern, collaborative security engine.
Understanding the Architecture: Why OpenResty and CrowdSec?
Before diving into the technical implementation, it is crucial to understand why this specific combination of open-source tools provides an elite defense mechanism against automated threats.
- OpenResty (The Gatekeeper): OpenResty integrates the standard Nginx core with powerful Lua modules. This allows developers to execute lightweight Lua scripts directly inside the Nginx worker processes. By handling access control at the web server level before requests ever reach your application backend (such as PHP, Node.js, or Python), you achieve maximum throughput and minimal latency.
- CrowdSec (The Brain): CrowdSec is a behavior-based security engine that analyzes server logs in real-time to detect anomalous patterns. Unlike traditional tools like Fail2ban, which rely purely on local regex matching, CrowdSec utilizes a decentralized, crowdsourced intelligence network. When a bad bot is blocked on one server, its IP reputation is shared across the entire CrowdSec ecosystem, protecting your VPS proactively.
By leveraging an OpenResty Lua plugin connected to the CrowdSec Local API (LAPI), we can inspect every incoming request against a real-time database of malicious actors, blocking them instantly at the network edge with negligible CPU overhead.
Step 1: Installing and Configuring OpenResty
To begin, we must replace standard Nginx with OpenResty on your VPS. OpenResty provides the necessary Lua ecosystem natively, eliminating the need to compile complex Nginx modules from scratch.
1.1 Add the Official OpenResty Repository
Log in to your VPS via SSH as a privileged user and execute the following commands to add the repository (example based on Ubuntu/Debian systems):
sudo apt-get update sudo apt-get install -y wget gnupg software-properties-common wget -qO - [https://openresty.org/package/pubkey.gpg](https://openresty.org/package/pubkey.gpg) | sudo apt-key add - sudo add-apt-repository -y "deb [http://openresty.org/package/debian](http://openresty.org/package/debian) $(lsb_release -sc) openresty"
1.2 Install the Package
Update your package lists and install OpenResty along with the standard development tools:
sudo apt-get update sudo apt-get install -y openresty
Once installed, verify that OpenResty is running successfully by checking its systemd service status: sudo systemctl status openresty.
Step 2: Deploying the CrowdSec Security Engine
Next, we will install the CrowdSec daemon on the same VPS. CrowdSec will monitor OpenResty's access logs to detect bad behaviors such as directory brute-forcing, aggressive crawling, or application scanning.
2.1 Install CrowdSec via Official Script
Execute the official installation script to configure repositories and download the security engine:
curl -s [https://install.crowdsec.net](https://install.crowdsec.net) | sudo bash sudo apt-get install crowdsec -y
2.2 Install the Nginx/OpenResty Collection
CrowdSec uses "collections"—pre-configured sets of parsers and scenarios—to understand specific log formats. Install the Nginx collection to immediately enable detection capabilities for web threats:
sudo cscli collections install crowdsecurity/nginx sudo systemctl restart crowdsec
At this stage, CrowdSec is actively reading your server logs and calculating risk scores for incoming traffic based on localized behavior rules.
Step 3: Integrating OpenResty with the CrowdSec LAPI Bouncer
To actively block bad bots rather than just logging their existence, we must bridge OpenResty and CrowdSec. This is achieved using the CrowdSec OpenResty Remediation Component (Bouncer), which utilizes Lua to query CrowdSec's Local API before serving any web page.
3.1 Install the OpenResty Bouncer
Download and install the dedicated Lua bouncer package for OpenResty:
sudo apt-get install lua-resty-http sudo apt-get install crowdsec-openresty-bouncer
3.2 Generate an API Key
Generate a secure Local API key so that the OpenResty bouncer can securely communicate with the CrowdSec daemon:
sudo cscli bouncers add openresty-bouncer-vps
Note: Copy the generated API token immediately; you will need to paste it into the configuration file in the next step.
3.3 Configure the Lua Plugin
Open the bouncer configuration file located at /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf and update the following directives:
API_URL=[http://127.0.0.1:8080](http://127.0.0.1:8080) API_KEY=YOUR_GENERATED_API_KEY BOUNCING_ON_TYPE=ban
Step 4: Crafting Custom Scenarios to Detect Aggressive Scraping
While CrowdSec includes general web protection, scrapers often operate just below standard rate limits. To catch dedicated bad bots, we need to implement a custom rule specifically targeting rapid, non-human content harvesting.
4.1 Create a Custom Anti-Scraping Scenario
Create a new configuration file at /etc/crowdsec/scenarios/vps-anti-scraping.yaml:
type: leaky_bucket name: company/vps-anti-scraping description: "Detect aggressive data scraping patterns" filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.static_image == 'false'" leakspeed: 2/s capacity: 30 groupeby: evt.Meta.source_ip blackhole: 5m labels: remediation: true scope: ip
This configuration utilizes a leaky bucket algorithm. If an IP address requests more than 30 pages within a highly condensed window (leaking at 2 requests per second), the scenario triggers. The malicious IP is instantly categorized for remediation, and the OpenResty bouncer will immediately serve a 403 Forbidden or a CAPTCHA wall to that specific user.
Apply the new rules by restarting the service: sudo systemctl restart crowdsec.
Step 5: Testing and Monitoring the Defense Matrix
To ensure your automated system works flawlessly without disrupting legitimate corporate traffic or search engines, comprehensive verification is necessary.
5.1 Simulating a Scraping Attack
You can simulate an automated scraping attack from a separate machine using an aggressive benchmarking tool like ApacheBench (ab) or a custom Python script:
ab -n 200 -c 10 http://your-vps-ip/index.html
5.2 Inspecting the Blocks
Return to your VPS terminal and execute the CrowdSec command-line interface tool to verify that the attacking IP has been systematically banned:
sudo cscli decisions list
The output will display a structured table showing the offending IP, the specific scenario triggered (e.g., company/vps-anti-scraping), the remediation type applied (ban), and the exact time remaining on the expiration timer.
Conclusion: Proactive Enterprise Security on a Budget
Implementing a manual IP blocklist is a reactive, losing battle against modern web automation. By uniting the raw performance of OpenResty with the crowdsourced, behavioral intelligence of CrowdSec, you establish an enterprise-grade web application firewall (WAF) directly on your VPS. This architecture guarantees that data scraping attempts are minimized, server resource utilization remains predictable, and proprietary business insights remain securely protected from unauthorized automated harvesting. Continuous optimization of your custom scenarios will ensure that your defense system remains resilient against the evolving tactics of malicious actors.