Scaling Automation: Building a Headless Chrome Cluster with Selenoid on VPS
Introduction: The Challenge of Enterprise Automation Scaling
In modern agile software development, continuous integration and continuous deployment (CI/CD) pipelines rely heavily on automated end-to-end (E2E) testing. As application complexity grows, the volume of required test suites scales exponentially. Running hundreds of Selenium or Playwright tests sequentially creates a massive bottleneck, delaying deployments and feedback loops.
Traditionally, teams turned to managed cloud grids like Sauce Labs or BrowserStack, or managed internal Selenium Grids. However, these solutions often introduce prohibitive costs or significant infrastructure overhead. A managed Selenium Grid is notoriously fragile, frequently suffering from memory leaks, zombie browser processes, and inconsistent state between test runs.
To overcome these challenges, enterprise engineering teams are increasingly adopting Selenoid—a lightweight, efficient implementation of the Selenium hub utilizing Docker containers to launch browsers. This guide provides a comprehensive, step-by-step technical blueprint for configuring a Headless Chrome Cluster with Selenoid on a Virtual Private Server (VPS) to power large-scale, enterprise-ready automation testing.
---Why Selenoid and Headless Chrome on a VPS?
Before diving into the implementation, it is vital to understand why the combination of Selenoid, Headless Chrome, and a VPS offers a superior architecture for automation engineering.
- Isolated Environments: Selenoid spins up a fresh Docker container for every single test session and terminates it immediately afterward. This guarantees total isolation, preventing data contamination, cookies, or cached states from affecting subsequent tests.
- Resource Efficiency: Traditional browsers require a graphical user interface (GUI) desktop environment (like Xvfb). Operating Chrome in headless mode eliminates the UI rendering overhead, reducing CPU and RAM consumption by up to 60%. This allows you to achieve significantly higher concurrency on identical hardware.
- Cost Effectiveness: Deploying this architecture on a high-performance VPS (such as DigitalOcean, Linode, or AWS EC2) costs a fraction of the price of proprietary test cloud subscriptions, granting you complete control over your computing resources and data privacy.
- Rich Tooling and Monitoring: Selenoid provides built-in support for real-time video recording, VNC console access for live debugging, and comprehensive logging out of the box.
Prerequisites and Environment Preparation
To follow this guide, you will need a clean VPS running a modern Linux distribution (Ubuntu 22.04 LTS or 24.04 LTS recommended). Ensure your VPS meets the minimum hardware requirements based on your desired concurrency:
Hardware Estimation Rule: As a rule of thumb, allocate approximately 0.5 CPU cores and 500MB to 700MB of RAM per concurrent Headless Chrome instance. For a cluster running 20 parallel browsers, a VPS with 8 vCPUs and 16GB RAM is ideal.
Step 1: System Update and Docker Installation
First, connect to your VPS via SSH and update the package repository to ensure all system dependencies are current:sudo apt update && sudo apt upgrade -y
Next, install Docker and Docker Compose, which are required to orchestrate the Selenoid containers and configuration:sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL [https://download.docker.com/linux/ubuntu/gpg](https://download.docker.com/linux/ubuntu/gpg) | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] [https://download.docker.com/linux/ubuntu](https://download.docker.com/linux/ubuntu) $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
Verify that Docker is running correctly by executing:sudo systemctl status docker---
Configuring the Selenoid Cluster
Selenoid relies on a simple JSON configuration file named browsers.json to map requested browser versions to specific Docker images. We will configure it to use optimized, headless-ready Chrome images provided by the Aerokube team.
Step 2: Create the Selenoid Configuration Directory
Establish a dedicated directory structure on your VPS to house the configuration files:
sudo mkdir -p /etc/selenoidStep 3: Define the browsers.json File
Create and edit the browsers.json file:
sudo nano /etc/selenoid/browsers.jsonPaste the following configuration into the file. This defines the Chrome versions available to your automation scripts:{
"chrome": {
"default": "120.0",
"versions": {
"120.0": {
"image": "selenoid/vnc:chrome_120.0",
"port": "4444",
"path": "/"
},
"121.0": {
"image": "selenoid/vnc:chrome_121.0",
"port": "4444",
"path": "/"
}
}
}
}
Note: Even though we specify the VNC images, we will pass explicit capabilities in our test scripts to force Headless execution, preserving the option to enable VNC debugging when troubleshooting localized environment issues.
Step 4: Pulling the Required Browser Images
To prevent execution delays during your initial test runs, pre-pull the required Docker images onto your VPS:
sudo docker pull selenoid/vnc:chrome_120.0
sudo docker pull selenoid/vnc:chrome_121.0
sudo docker pull aerokube/selenoid:latest-releaseOrchestrating Selenoid with Docker Compose
Using Docker Compose allows us to manage Selenoid alongside its companion UI (Selenoid UI) effortlessly, streamlining monitoring and cluster management.
Step 5: Writing the docker-compose.yml
Create a docker-compose.yml file in your preferred working directory (e.g., ~/selenoid-cluster/):
mkdir ~/selenoid-cluster && cd ~/selenoid-cluster
nano docker-compose.ymlPopulate the file with the following multi-service configuration:version: '3.8'
services:
selenoid:
image: "aerokube/selenoid:latest-release"
container_name: selenoid
ports:
- "4444:4444"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/etc/selenoid:/etc/selenoid:ro"
- "/root/selenoid/video:/opt/selenoid/video"
- "/root/selenoid/logs:/opt/selenoid/logs"
environment:
- TZ=UTC
command: ["-conf", "/etc/selenoid/browsers.json", "-limit", "20", "-video-output-dir", "/opt/selenoid/video", "-log-output-dir", "/opt/selenoid/logs"]
restart: always
selenoid-ui:
image: "aerokube/selenoid-ui:latest-release"
container_name: selenoid-ui
ports:
- "8080:8080"
command: ["--selenoid-uri", "http://selenoid:4444"]
depends_on:
- selenoid
restart: always
Crucial parameters configured in this file include:
/var/run/docker.sockmapping: Allows Selenoid to communicate with the host's Docker daemon to spawn sibling browser containers.-limit 20: Restricts the maximum number of concurrent browser containers allowed to run simultaneously, preventing resource exhaustion on the VPS.- Selenoid UI: Exposes a clean web dashboard on port 8080 to visualize real-time cluster utilization.
Step 6: Launching the Cluster
Initialize and start your cluster services in detached mode:
sudo docker compose up -dVerify that both containers are active and healthy:
sudo docker compose psOptimizing for Large-Scale Execution
To run a truly enterprise-scale automation framework processing thousands of test cases, default Linux configurations must be optimized to manage high network throughput and file handle consumption.
Adjusting File Descriptors and Kernel Limits
Each concurrent browser and network connection opens file descriptors. Modify the system configuration to handle heavy loads:
sudo nano /etc/security/limits.confAdd the following lines at the bottom of the file:* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535
Apply these network optimizations in /etc/sysctl.conf to improve TCP socket reuse rates under load:
sudo nano /etc/sysctl.confAppend the following kernel parameters:
net.core.somaxconn = 1024
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65000Apply the changes immediately by executing: sudo sysctl -p.
Connecting Your Test Automation Framework
With your headless cluster fully operational, configure your automation framework to point to the Selenoid hub address: http://.
Below is an enterprise-pattern implementation using Python with Selenium, optimized specifically for Headless Chrome execution within Selenoid:from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def create_remote_headless_driver():
chrome_options = Options()
# Strict Headless Configuration
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--disable-gpu")
# Selenoid Specific Capabilities
selenoid_capabilities = {
"browserName": "chrome",
"browserVersion": "121.0",
"selenoid:options": {
"enableVNC": False, # Set to True for live debugging
"enableVideo": False, # Set to True to record execution
"name": "Regression Test Suite - Account Validation"
}
}
chrome_options.set_capability("cloud:options", selenoid_capabilities)
# Point to the VPS Selenoid Hub
vps_hub_url = "http://---
Securing the Cluster
Exposing ports 4444 and 8080 completely unrestricted to the public internet presents significant security risks. It is imperative to restrict access using a firewall or reverse proxy.
UFW Firewall Configuration
We recommend restricting access to the Selenoid hub exclusively to the IP addresses of your CI/CD runners (e.g., Jenkins, GitHub Actions runners, or GitLab CI nodes):
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
# Allow a specific CI/CD Server IP to access Selenoid Hub
sudo ufw allow from to any port 4444
# Enable the Firewall
sudo ufw enable Conclusion and Next Steps
By migrating from traditional execution models to a Headless Chrome Cluster managed via Selenoid on a VPS, you build an enterprise-grade testing infrastructure capable of handling extensive parallel test suites. This architecture drastically reduces execution times from hours to minutes, improves stability, and maintains a predictable, cost-contained infrastructure footprint.
As next steps to advance your grid maturity, consider setting up an Nginx Reverse Proxy with Basic Authentication to protect the Selenoid UI dashboard, and integrating Prometheus and Grafana metrics to monitor your container cluster performance in real-time.