Building an Automated Web Archiver with ArchiveBox on VPS
Detailed guide to building an automated website archiving system with ArchiveBox on VPS, covering installation, configuration, automation, and security.
Image credit: Ben_Kerckx · Pixabay Content License
Overview of Automated Web Archiver with ArchiveBox
ArchiveBox is an open-source tool that enables you to automatically and reliably archive web content. It captures HTML, images, videos, and related files, creating a comprehensive archive of websites. This ensures your content remains preserved against changes or deletions on original websites.
In this guide, we will build an automated website storage system on a VPS using ArchiveBox. The system will automatically collect and store web pages on a schedule, providing fast and secure access to archived content.
Requirements and Initial Setup
To begin, you need a Linux-based VPS (Ubuntu 20.04 or 22.04 is recommended). The specific requirements include:
- RAM: at least 4GB
- Storage: 50GB or more (depending on the number of websites to archive)
- Internet bandwidth: sufficient to download web content
First, update your system:
sudo apt update && sudo apt upgrade -y
Install Docker and Docker Compose:
sudo apt install docker.io docker-compose -y
Configuring and Deploying ArchiveBox
Create a new directory for ArchiveBox configuration:
mkdir archivebox && cd archivebox
Create a docker-compose.yml file with the following content:
version: '3.8'
services:
archivebox:
image: archivebox/archivebox:latest
container_name: archivebox
volumes:
- ./data:/archivebox/data
ports:
- "8000:8000"
command: archivebox server --host 0.0.0.0 --port 8000 --allow-root
restart: unless-stopped
Start the service:
docker-compose up -d
After starting, access http://your-vps-ip:8000 to verify the ArchiveBox management interface.
Automating Website Archiving
To automate archiving, use cron jobs. Create a script file archive.sh:
#!/bin/bash
docker-compose exec archivebox archivebox add https://example.com --depth=1 --timeout=300
Make the script executable:
chmod +x archive.sh
Add the command to cron to run daily:
0 2 * * * /path/to/archive.sh
Replace /path/to/archive.sh with the actual path to the script.
Security and Maintenance
To secure the system, consider the following:
- Use HTTPS with Let's Encrypt to secure connections.
- Configure UFW firewall to allow only ports 80 and 443.
- Perform regular backups of stored data.
For maintenance, monitor storage usage and update ArchiveBox regularly:
docker-compose pull archivebox
Testing and Verification
To verify the system works correctly, follow these steps:
- Access the web interface and check if the website has been added.
- Run
docker-compose logs archiveboxto review error logs. - Test accessing an archived page to ensure it functions.
Troubleshooting
If issues arise, check the following:
- ArchiveBox container logs.
- Correct mounting of the data directory.
- Firewall settings on the VPS.
For further assistance, refer to the official ArchiveBox documentation at https://archivebox.io/docs/.
References
Recommended NVMe Cloud VPS Plans to Deploy This Setup
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-3E
10% VAT included
- CPU: 2 vCPU
- RAM: 4 GB
- SSD: 50 GB NVMe
