Deploying Garage Object Storage: Highly Available, Fault-Tolerant Distributed Storage for Small VPS Clusters

Introduction: The Storage Dilemma for Small VPS Clusters

In the era of cloud-native architectures, object storage has become the bedrock for modern applications. Whether it is storing user uploads, hosting static assets, or managing database backups, the AWS S3 API is the undisputed standard. However, for engineering teams operating on small, self-hosted Virtual Private Server (VPS) clusters, implementing a resilient, distributed object storage system has historically been a significant challenge.

Enterprise-grade solutions like Ceph offer unparalleled scalability but demand massive computational resources, specialized networking, and dedicated operational teams. On the other end of the spectrum, standalone instances of MinIO are lightweight but introduce a single point of failure (SPOF) unless deployed in complex distributed modes that still require significant overhead. This leaves a critical gap for deployments spanning 3 to 5 low-spec VPS nodes.

Enter Garage Object Storage. Developed by Deuxfleurs, Garage is an open-source, distributed object storage service specifically engineered to be lightweight, highly available, and exceptionally fault-tolerant. In this comprehensive guide, we will explore the architecture of Garage and provide a step-by-step blueprint for deploying it across a small VPS cluster.

Why Garage? Architectural Advantages for Small Deployments

Garage was designed from the ground up to thrive in environments where traditional distributed storage systems fail. Here is why it is uniquely suited for small VPS clusters:

• Minimal Resource Footprint: Written in Rust, Garage consumes negligible memory and CPU compared to Java or Go-based alternatives. It can run comfortably on VPS instances with as little as 1GB of RAM.
• Asymmetric and Heterogeneous Clusters: Unlike many distributed file systems that require identical disk sizes and node performance, Garage handles heterogeneous nodes gracefully. You can mix a 50GB NVMe VPS with a 200GB HDD VPS seamlessly.
• Network Partition Tolerance: Garage is built on a custom Conflict-Free Replicated Data Type (CRDT) model over a Dynamo-like architecture. It tolerates severe network latency and temporary partitions, self-healing automatically when connectivity is restored.
• No Single Point of Failure (SPOF): Every node in a Garage cluster is identical and acts as a coordinator. There are no dedicated metadata servers (like Ceph Mon) or master nodes that can bring down the entire system if they fail.

Key Takeaway: Garage sacrifices strict consistency in favor of eventual consistency and extreme availability, making it the perfect match for geo-distributed or budget-friendly VPS setups.

Prerequisites and Network Architecture

Before initiating the deployment, ensure your environment meets the following baseline criteria:

1. Nodes: A minimum of three (3) VPS instances located in the same or different data centers to achieve quorum.
2. Operating System: Linux (Ubuntu 22.04 LTS or newer recommended).
3. Networking: Public or private static IP addresses for each node. Port 3901 is required for inter-node communication (RPC), and port 3900 is typically used for the S3 API endpoint.
4. Security: Firewall rules (UFW or cloud firewalls) configured to allow traffic on RPC ports strictly between cluster members.

Sample Cluster Blueprint

For the purpose of this guide, we will reference a three-node topology:

• Node 1: 192.168.1.10 (Data directory: /var/lib/garage/data)
• Node 2: 192.168.1.11 (Data directory: /var/lib/garage/data)
• Node 3: 192.168.1.12 (Data directory: /var/lib/garage/data)

Step-by-Step Deployment Blueprint

Step 1: Installing the Garage Binary

Garage is distributed as a single static binary, making installation remarkably simple. Execute the following commands on all nodes to download and install the latest stable release:

curl -s [https://garagehq.opera-digital.com/releases/v1.0.0/x86_64-unknown-linux-musl/garage](https://garagehq.opera-digital.com/releases/v1.0.0/x86_64-unknown-linux-musl/garage) -o /usr/local/bin/garage
chmod +x /usr/local/bin/garage

Verify the installation by checking the version:

garage --version

Step 2: Designing the Configuration File

Create the configuration file at /etc/garage.toml on each node. While the configurations will be nearly identical, the metadata_dir, data_dir, and rpc_bind_addr must accurately reflect the specific node's environment. Below is a production-ready template:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"

rpc_bind_addr = "0.0.0.0:3901"
rpc_public_addr = "192.168.1.10:3901" # Change to the respective node's IP
rpc_secret = "GENERATED_HEX_SECRET_FOR_CLUSTER_AUTH"

[s3_api]
bind_addr = "0.0.0.0:3900"
api_region = "us-east-1"

[s3_web]
bind_addr = "0.0.0.0:3902"

Note: Generate a secure 32-byte hex string for the rpc_secret using openssl rand -hex 32. This token must be identical across all nodes to authenticate intra-cluster communication.

Step 3: Managing the Daemon via Systemd

To ensure high availability, manage the Garage process using systemd. Create a service file at /etc/systemd/system/garage.service:

[Unit]
Description=Garage Object Storage Daemon
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/garage server -c /etc/garage.toml
Restart=on-failure
User=root
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Reload the systemd manager, enable, and start the service on all nodes:

systemctl daemon-reload
systemctl enable --now garage
systemctl status garage

Cluster Initialization and Data Layout Layout

With the daemons operational, the nodes are running in isolation. We must now connect them into a unified cluster and assign their storage weights.

Connecting Nodes

Log into Node 1 and utilize the Garage CLI to check node status and connect to the auxiliary nodes:

garage node status

To connect Node 2 and Node 3 from Node 1, retrieve their Node IDs from their respective logs or status screens, then execute:

garage node connect [NODE2_ID]@192.168.1.11:3901
garage node connect [NODE3_ID]@192.168.1.12:3901

Configuring the Topology

Garage uses a structural layout configuration to determine data distribution based on physical locations and capacities. Configure the layout by assigning zones and capacities to each node:

garage layout assign [NODE1_ID] --capacity 50G --zone dc-1
garage layout assign [NODE2_ID] --capacity 50G --zone dc-1
garage layout assign [NODE3_ID] --capacity 50G --zone dc-1

Review the layout plan to ensure correctness:

garage layout plan

If the plan matches your structural intentions, apply it to finalize the cluster configuration:

garage layout apply --version 1

The cluster will automatically begin balancing data using its internal ring architecture. You can verify health via garage status.

Creating S3 Credentials and Buckets

Now that your fault-tolerant storage pool is active, you can provision access keys and S3 buckets for application integration.

1. Create an API Key

garage key create web-app-key

This command outputs an Access Key ID and a Secret Access Key. Save these securely; they are required by your S3 client applications.

2. Create a Bucket

garage bucket create my-application-assets

3. Grant Permissions

Authorize the newly created key to read and write to the bucket:

garage bucket allow my-application-assets --read --write --key web-app-key

Operational Best Practices for Production

Running a distributed storage layer demands adherence to operational discipline to maintain optimal performance and data integrity:

• Monitoring and Metrics: Garage natively exposes a Prometheus metrics endpoint. Integrate it with Grafana to monitor disk I/O, network replication queues, and API error rates.
• Reverse Proxy Integration: Never expose the raw S3 API directly to the public internet for production apps. Place a reverse proxy like Nginx or Caddy in front of the S3 port (3900) to handle TLS termination, request filtering, and rate limiting.
• Disk Health: While Garage handles disk writes robustly, underlying filesystem corruption can impair performance. Use stable underlying filesystems like ext4 or XFS, and monitor disk health using SMART tools.

Conclusion

Garage Object Storage redefines what is possible for small-scale self-hosted infrastructure. By combining the lightweight nature of a single-binary Go/Rust application with the sophisticated distributed principles of the Dynamo architecture, it provides an unparalleled storage solution for small VPS clusters. By deploying Garage, independent developers and small enterprise teams can achieve resilient, highly available, and S3-compatible object storage without succumbing to the operational complexity or resource demands of legacy enterprise systems.