DPTCloud
Back to articles
Technology Insight

Scaling Web Scraping: How to Build a Distributed Browserless Cluster Across Multiple VPS for SPA Data Extraction

June 3, 2026

Introduction: The Evolution of Web Scraping and the SPA Challenge

Data has become the primary fuel driving modern business intelligence, competitive analysis, and machine learning models. However, gathering this data has become increasingly complex. The widespread adoption of Modern Web Development frameworks like React, Angular, and Vue.js has shifted the web landscape toward Single Page Applications (SPAs). Unlike traditional static websites that deliver fully rendered HTML from the server, SPAs rely heavily on client-side JavaScript execution to fetch and render content dynamically.

For automated data extraction systems, this architectural shift renders traditional, lightweight scraping tools like BeautifulSoup or Axios largely ineffective. To extract data from an SPA reliably, you must execute the underlying JavaScript. This requires a headless browser environment (such as Chromium managed via Puppeteer or Playwright). While powerful, running instances of full browsers is notoriously resource-intensive, consuming substantial CPU and memory. Scaling this operation to scrape millions of pages requires moving away from single-machine setups toward a distributed, automated Browserless Cluster deployed across multiple Virtual Private Servers (VPS). This guide provides a comprehensive technical blueprint for architecting such a system.

1. Architectural Overview of a Distributed Browserless Cluster

To build a resilient and scalable scraping infrastructure, we must separate the data extraction logic from the browser execution environment. A decentralized architecture typically consists of three primary layers:

  • The Scraper Control Plane (Client Node): The central application that orchestrates the scraping workflows, manages target URL queues, and processes the extracted data payloads.
  • The Load Balancing Layer: A reverse proxy (such as Nginx, HAProxy, or Traefik) that distributes incoming WebSocket or HTTP requests from the control plane evenly across the available worker nodes.
  • The Worker Pool (Browserless Nodes): Multiple cost-effective VPS instances running Dockerized Browserless containers. Each container manages a pool of isolated headless Chromium instances ready to execute JavaScript and render target SPAs.
Architectural Note: By decouplng the execution environment, if a specific browser instance crashes due to a memory leak or a heavy website, it does not bring down your core scraping application. The load balancer simply routes the next request to a healthy node.

2. Preparing and Provisioning the VPS Worker Nodes

To optimize costs and performance, you should deploy multiple medium-tier VPS instances rather than one massive, expensive server. This strategy minimizes blast radiuses and circumvents IP-rate limiting by distributing traffic across different networks.

System Prerequisites

Each VPS worker node should run a stable Linux distribution, preferably Ubuntu 22.04 LTS or higher, and have at least 2 vCPUs and 4GB of RAM. Headless browsers require sufficient memory allocation to prevent out-of-memory (OOM) crashes during concurrent scraping operations.

Installing Docker and Docker Compose

Execute the following standard commands on each worker VPS to update the system packages and install the Docker engine:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL [https://download.docker.com/linux/ubuntu/gpg](https://download.docker.com/linux/ubuntu/gpg) | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] [https://download.docker.com/linux/ubuntu](https://download.docker.com/linux/ubuntu) $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-compose

3. Deploying Browserless on Worker Nodes

Browserless is a specialized, open-source tool designed to run headless browsers inside Docker containers, optimized for high-performance automation. It includes built-in queue management, session handling, and resource monitoring.

Creating the Configuration File

On each VPS worker node, create a docker-compose.yml file to define and configure the Browserless container service. It is critical to enforce authentication to prevent unauthorized parties from using your infrastructure.

version: '3.8'
services:
  browserless:
    image: browserless/chrome:latest
    ports:
      - "3000:3000"
    environment:
      - TOKEN=YourSecureClusterToken123
      - CONCURRENT_USER_SESSIONS=10
      - MAX_QUEUE_COUNT=20
      - PRE_BOOT_CHROME=true
      - DEMO_MODE=false
    restart: always

Key configuration parameters explained:

  • TOKEN: The secret key required by your scraping script to connect to the node.
  • CONCURRENT_USER_SESSIONS: Limits the maximum number of parallel browser tabs open simultaneously. Adjust this based on your VPS RAM capacity (roughly 350-400MB per session).
  • PRE_BOOT_CHROME: Keeps warm Chromium instances ready in the background, significantly reducing initial connection latency.

Launch the service by running sudo docker-compose up -d. Verify the installation by navigating to http://your_vps_ip:3000 in your browser to view the Browserless management interface.

4. Setting Up the Central Load Balancer

Once all worker VPS nodes are running Browserless, a centralized load balancer must be deployed to distribute incoming automated testing and scraping connections. Nginx is highly recommended due to its excellent performance with WebSockets, which Browserless relies heavily on.

Provision a separate lightweight VPS for Nginx and configure the /etc/nginx/nginx.conf file as follows:

http {
    upstream browserless_cluster {
        least_conn;
        server vps_worker_1_ip:3000;
        server vps_worker_2_ip:3000;
        server vps_worker_3_ip:3000;
    }

    server {
        listen 80;
        server_name cluster.yourdomain.com;

        location / {
            proxy_pass http://browserless_cluster;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_read_timeout 600s;
            proxy_send_timeout 600s;
        }
    }
}

Using the least_conn; directive ensures that Nginx always routes new scraping tasks to the worker node with the fewest active connections, preventing individual VPS instances from becoming bottlenecks.

5. Executing and Optimizing the Scraping Script

With your distributed cluster active, you can connect your automation scripts using standard libraries like Playwright or Puppeteer. Instead of launching a local browser instance, configure your framework to connect to the load balancer endpoint via WebSockets.

Sample Node.js Implementation with Playwright

The following script connects to your cluster, handles dynamic SPA rendering by waiting for network idle states, and extracts the required data elements:

const { chromium } = require('playwright');

(async () => {
    const clusterUrl = 'ws://cluster.yourdomain.com?token=YourSecureClusterToken123';
    
    // Connect directly to the distributed Browserless cluster
    const browser = await chromium.connectOverCDP(clusterUrl);
    const context = await browser.newContext();
    const page = await context.newPage();

    try {
        // Navigate to the target Single Page Application
        await page.goto('[https://example-spa-target.com/products](https://example-spa-target.com/products)', {
            waitUntil: 'networkidle',
            timeout: 60000
        });

        // Wait explicitly for dynamic content components to inject into the DOM
        await page.waitForSelector('.product-grid-item');

        // Extract the rendered data array
        const products = await page.evaluate(() => {
            return Array.from(document.querySelectorAll('.product-grid-item')).map(item => ({
                title: item.querySelector('.title').innerText,
                price: item.querySelector('.price').innerText
            }));
        });

        console.log(`Successfully extracted ${products.length} items.`, products);
    } catch (error) {
        console.error('Extraction failed:', error.message);
    } finally {
        await browser.close();
    }
})();

6. Enterprise Best Practices: Security, Throttling, and Monitoring

Operating a production-grade scraping cluster requires strict operational discipline to guarantee longevity and compliance.

  • Resource Isolation and Auto-Reboot: Headless Chromium instances inevitably suffer from memory degradation over prolonged periods. Configure a cron job or Docker policies to restart your Browserless containers nightly to clear residual caches.
  • Proxy Rotation and Anti-Bot Bypass: While your cluster manages system resources, target SPAs often use anti-bot systems (like Cloudflare or Akamai). Integrate an external proxy rotation service at the browser level or pass a custom proxy string within your connection URL (e.g., &--proxy-server=[http://your-proxy.com](http://your-proxy.com)).
  • Network Security: Use firewall tools like UFW on your worker nodes to ensure they only accept incoming traffic originating from the static IP of your Load Balancer and your internal development systems.

Conclusion

Transitioning from local runtime instances to a distributed Browserless Cluster on multiple VPS transforms your web scraping from an fragile execution environment into an industrial-strength, enterprise-grade data collection pipeline. By abstracting away the heavy operational lifting of headless browser resource management, your engineering teams can focus on extracting real business value from complex Single Page Applications reliably and cost-effectively.

Scaling Web Scraping: How to Build a Distributed Browserless Cluster Across Multiple VPS for SPA Data Extraction | DPTCloud