Back to articles
DevOps & Systems

Building an Automated Web Archiver with ArchiveBox on VPS

Detailed guide to building an automated website archiving system with ArchiveBox on VPS, covering installation, configuration, automation, and security.

DPTCloud Engineering4 min read
Building an Automated Web Archiver with ArchiveBox on VPS

Image credit: Ben_Kerckx · Pixabay Content License

Overview of Automated Web Archiver with ArchiveBox

ArchiveBox is an open-source tool that enables you to automatically and reliably archive web content. It captures HTML, images, videos, and related files, creating a comprehensive archive of websites. This ensures your content remains preserved against changes or deletions on original websites.

In this guide, we will build an automated website storage system on a VPS using ArchiveBox. The system will automatically collect and store web pages on a schedule, providing fast and secure access to archived content.

Requirements and Initial Setup

To begin, you need a Linux-based VPS (Ubuntu 20.04 or 22.04 is recommended). The specific requirements include:

  • RAM: at least 4GB
  • Storage: 50GB or more (depending on the number of websites to archive)
  • Internet bandwidth: sufficient to download web content

First, update your system:

sudo apt update && sudo apt upgrade -y

Install Docker and Docker Compose:

sudo apt install docker.io docker-compose -y

Configuring and Deploying ArchiveBox

Create a new directory for ArchiveBox configuration:

mkdir archivebox && cd archivebox

Create a docker-compose.yml file with the following content:

version: '3.8'
services:
  archivebox:
    image: archivebox/archivebox:latest
    container_name: archivebox
    volumes:
      - ./data:/archivebox/data
    ports:
      - "8000:8000"
    command: archivebox server --host 0.0.0.0 --port 8000 --allow-root
    restart: unless-stopped

Start the service:

docker-compose up -d

After starting, access http://your-vps-ip:8000 to verify the ArchiveBox management interface.

Automating Website Archiving

To automate archiving, use cron jobs. Create a script file archive.sh:

#!/bin/bash
docker-compose exec archivebox archivebox add https://example.com --depth=1 --timeout=300

Make the script executable:

chmod +x archive.sh

Add the command to cron to run daily:

0 2 * * * /path/to/archive.sh

Replace /path/to/archive.sh with the actual path to the script.

Security and Maintenance

To secure the system, consider the following:

  • Use HTTPS with Let's Encrypt to secure connections.
  • Configure UFW firewall to allow only ports 80 and 443.
  • Perform regular backups of stored data.

For maintenance, monitor storage usage and update ArchiveBox regularly:

docker-compose pull archivebox

Testing and Verification

To verify the system works correctly, follow these steps:

  • Access the web interface and check if the website has been added.
  • Run docker-compose logs archivebox to review error logs.
  • Test accessing an archived page to ensure it functions.

Troubleshooting

If issues arise, check the following:

  • ArchiveBox container logs.
  • Correct mounting of the data directory.
  • Firewall settings on the VPS.

For further assistance, refer to the official ArchiveBox documentation at https://archivebox.io/docs/.

References

DPTCloud Tier III Infrastructure

Recommended NVMe Cloud VPS Plans to Deploy This Setup

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02

VPS-2E

94.600 ₫ / mo

10% VAT included

  • CPU: 2 vCPU
  • RAM: 2 GB
  • SSD: 30 GB NVMe
ECO · vn-02Recommended

VPS-3E

170.500 ₫ / mo

10% VAT included

  • CPU: 2 vCPU
  • RAM: 4 GB
  • SSD: 50 GB NVMe
BASIC · vn-01Recommended

VPS-05

457.600 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478