Building a High-Availability Distributed Object Storage System with Garage S3 on a Budget VPS Cluster

Introduction to Decentralized Storage for Modern Enterprise

In the contemporary digital landscape, data growth is exponential, driving businesses to seek highly scalable, reliable, and cost-effective storage solutions. While premium cloud providers offer robust Simple Storage Service (S3) architectures, the recurring costs and data egress fees can quickly erode the margins of small-to-medium enterprises (SMEs) and independent developers. This economic challenge has sparked a renaissance in self-hosted infrastructure.

Enter Garage (developed by GarageHQ), an open-source, lightweight, distributed object storage service designed specifically to turn heterogeneous, low-bandwidth, and budget-friendly hardware into a highly resilient cluster. Unlike complex alternatives like Ceph or MinIO, which demand heavyweight resources and identical node configurations, Garage thrives in constrained environments. In this comprehensive guide, we will walk through the architectural philosophy and step-by-step deployment of a distributed object storage system across three budget Virtual Private Servers (VPS).

---

Why Garage S3 Over Traditional Solutions?

When engineering a distributed system, architects often grapple with the CAP Theorem (Consistency, Availability, Partition Tolerance). Traditional distributed file systems lean heavily toward strict consistency, requiring high-throughput networks and low latency. Garage, conversely, is built on the following foundational pillars tailored for budget infrastructure:

• CRDT-Driven Consistency: By utilizing Conflict-Free Replicated Data Types, Garage achieves eventual consistency without a centralized coordinator, making it exceptionally partition-tolerant.
• Low Resource Footprint: Written in Rust, a single node can efficiently operate on as little as 512MB of RAM, leaving precious CPU and memory cycles for your applications.
• Geo-Distributed Optimization: It is natively designed to replicate data across physically distinct data centers, mitigating the risk of single-region outages.

"Garage does not try to emulate a local filesystem. It implements the S3 API directly on top of a specialized distributed key-value layer, maximizing performance over wide-area networks."

---

Prerequisites and Cluster Architecture

To establish a resilient deployment capable of surviving the failure of a single node (achieving a quorum of 2 out of 3), we require three distinct VPS instances. For maximum fault isolation, consider sourcing these instances from different providers or geographical regions.

Minimum Node Requirements

1. CPU: 1 vCPU (x86_64 or ARM64)
2. Memory: 1 GB RAM
3. Storage: 20 GB+ NVMe or SSD (SATA HDDs work but will degrade performance)
4. OS: Debian 12 / Ubuntu 24.04 LTS
5. Network: Static Public IPv4 address per node with an unblocked port structure

Network Layout Matrix

We will allocate specific ports for cluster internal communication (RPC) and external client API access. Ensure your firewalls (UFW or provider panels) permit traffic according to this schema:

Service	Default Port	Traffic Type	Scope
Garage RPC	3901	TCP	Internal Cluster Only
S3 API	3900	TCP	Public / Application Client
Web S3 Virtual Hosts	3902	TCP	Public (Optional)

---

Step-by-Step Deployment Protocol

Step 1: System Provisioning and Binary Installation

First, SSH into each of your three target nodes. Download the latest stable production binary of Garage. We will place it in the standard system binary path and ensure execution permissions are granted.

wget [https://garagehq.opera-digital.com/releases/v1.0.0/x86_64-unknown-linux-musl/garage](https://garagehq.opera-digital.com/releases/v1.0.0/x86_64-unknown-linux-musl/garage)
mv garage /usr/local/bin/
chmod +x /usr/local/bin/garage

Verify the installation on all hosts by executing garage --version.

Step 2: Crafting the Configuration Topology

Create a dedicated configuration directory on each node at /etc/garage/. The configuration file, written in TOML syntax, must mirror structural parameters across all nodes, changing only the local binding details and unique node metadata directories.

Below is a standardized template for /etc/garage/garage.toml on Node 1:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
data_block_size = 1048576

[rpc_bind]
allowed_public_key_hashes = []
secret_key = "GENERATED_CLUSTER_HEX_SECRET"

[rpc_public_addr]
# Replace with the public IP of Node 1
addr = "192.0.2.1:3901"

[s3_api]
api_bind_addr = "0.0.0.0:3900"
api_public_host = "s3.yourdomain.com"
root_domain = ".s3.yourdomain.com"

Critical Security Note: The secret_key variable must be an identical 32-byte hex string across all three nodes to authenticate peer communication. Generate it using openssl rand -hex 32.

Step 3: Orchestrating Systemd Services

To guarantee operational uptime and automatic recovery upon system reboots, wrap the execution within a systemd service descriptor. Create /etc/systemd/system/garage.service:

[Unit]
Description=Garage Distributed Object Storage
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/garage server
Restart=always
RestartSec=5
User=root
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Execute the daemon reload sequence, enable, and start the service on all nodes:

systemctl daemon-reload
systemctl enable --now garage
systemctl status garage

---

Cluster Initialization and Data Replication

With the individual daemons active, they remain isolated instances until clustered explicitly. Connect to the command-line interface (CLI) of Node 1 to orchestrate the joining protocol.

1. Node Connection Phase

Retrieve the unique Node ID from Node 2 and Node 3 by checking their logs or using the status command. Then, issue connection requests from Node 1:

garage node connect [NODE2_ID]@[NODE2_PUBLIC_IP]:3901
garage node connect [NODE3_ID]@[NODE3_PUBLIC_IP]:3901

2. Structuring Layout and Assigning Weight

Garage operates on a virtual ring topology. Assign tokens, zone tags, and physical capacity weights to the cluster layout. For a balanced three-node cluster, we define three unique zones to ensure data is mirrored uniformly:

garage layout assign [NODE1_ID] -z zone1 -c 10G
garage layout assign [NODE2_ID] -z zone2 -c 10G
garage layout assign [NODE3_ID] -z zone3 -c 10G

Review the plan configuration using garage layout plan. If the visual topology aligns with expectations, finalize and execute the replication matrix:

garage layout apply --version 1

---

Production Optimization: Reverse Proxy and Security

Exposing port 3900 directly to the internet is functional but suboptimal for high-volume enterprise production. Interposing a reverse proxy like Nginx or Caddy simplifies Transport Layer Security (TLS/SSL) offloading and enables load balancing across your storage cluster.

Sample Nginx Configuration with SSL Offloading

Configure Nginx to act as a reverse proxy, mapping incoming public HTTPS requests to the internal Garage S3 API port:

server {
    listen 443 ssl http2;
    server_name s3.yourdomain.com *.s3.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/[yourdomain.com/fullchain.pem](https://yourdomain.com/fullchain.pem);
    ssl_certificate_key /etc/letsencrypt/live/[yourdomain.com/privkey.pem](https://yourdomain.com/privkey.pem);

    # Maximize upload thresholds for large binary objects
    client_max_body_size 10G;

    location / {
        proxy_pass [http://127.0.0.1:3900](http://127.0.0.1:3900);
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Essential for S3 streaming chunked uploads
        proxy_buffering off;
        proxy_request_buffering off;
    }
}

---

Conclusion and Performance Analysis

By leveraging GarageHQ on a minimalist three-node VPS deployment, you effectively eliminate single points of failure without incurring astronomical monthly hyperscaler bills. Should one VPS experience hardware degradation or network blackouts, the remaining two nodes seamlessly handle continuous data mutations and read requests via quorum logic. This layout represents an ideal architecture for hosting application assets, managing backup systems, and hosting private Docker registries securely and affordably.