DPTCloud
Back to articles
Technology Insight

Scaling Low-Cost Infrastructure: Building a Distributed S3-Compatible Object Storage with Garage and Oracle ARM VPS

June 5, 2026

Introduction: The Quest for Resilient, Low-Cost Cloud Storage

In the modern DevOps landscape, S3-compatible object storage has become the de facto standard for handling unstructured data, backups, and static assets. While hyperscalers like AWS, Google Cloud, and Azure offer robust solutions, the costs—particularly egress fees and API call charges—can escalate rapidly as your data footprint grows. For developers and small-to-medium enterprises (SMEs) looking for high availability without the enterprise price tag, the Oracle Cloud Infrastructure (OCI) Free Tier offers a unique opportunity: four ARM-based Ampere A1 Compute instances with a combined 24GB of RAM.

However, running a single storage node introduces a single point of failure. This is where Garage comes in. Garage is an open-source, lightweight, distributed object storage service tailored for self-hosting. Unlike MinIO, which often requires significant resources or complex orchestration for true geo-distribution, Garage is designed to run on low-power hardware and across unstable networks, making it the perfect candidate for a distributed cluster across Oracle’s free ARM nodes.

Understanding the Architecture: Why Garage?

Garage differs from traditional storage systems through its focus on simplicity and resilience. It utilizes a Directed Acyclic Graph (DAG) and a custom consensus mechanism based on CRDTs (Conflict-free Replicated Data Types) rather than the more resource-heavy Raft protocol used by systems like Ceph or MinIO. This allows Garage to maintain consistency and availability even when nodes are spread across different geographic regions or experience network latency.

Key Advantages for ARM-Based Deployments:

  • Minimal Resource Footprint: Written in Rust, Garage consumes very little CPU and RAM, leaving plenty of overhead for other applications on your ARM instances.
  • No Metadata Database: Garage does not require an external database like PostgreSQL or Redis; it manages its own state, simplifying the deployment stack.
  • Elastic Scaling: You can easily add or remove nodes from the cluster. Garage automatically redistributes data blocks across the available participants.
  • Resilience to Network Partitions: Designed for the 'messy' internet, it handles inter-node connectivity drops gracefully.

Step 1: Provisioning Your Oracle ARM Instances

To build a robust cluster, you should aim to distribute your instances across different Availability Domains or even different regions if your account allows. For this setup, we will assume you have provisioned three or four ARM Ampere A1 instances running Ubuntu 22.04 or 24.04.

  1. Network Configuration: Ensure your VCN (Virtual Cloud Network) Security Lists allow traffic on the following ports:
    • 3901: For the S3 API (Client access).
    • 3902: For inter-node communication (RPC).
    • 3903: For the administration API (optional).
  2. Storage Allocation: Attach a Block Volume to each instance. While the Free Tier provides 200GB of total block storage, distributing this evenly (e.g., 50GB per node) ensures data redundancy across the cluster.

Step 2: Installing and Configuring Garage

Since we are working with ARM architecture, we must ensure we use the aarch64 binaries. Garage provides pre-compiled binaries that make installation straightforward.

The Configuration File (garage.toml)

Each node requires a garage.toml file. The beauty of Garage is that the configuration is nearly identical across all nodes, with the exception of the metadata_dir and data_dir if your mount points differ.

Important: Always use private IP addresses for inter-node communication (RPC) to reduce latency and improve security, while using public IPs or a load balancer for the S3 API endpoint.
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"

[rpc_bind]
rpc_bind_addr = "[::]:3902"

[s3_api]
s3_bind_addr = "[::]:3901"
s3_region = "garage"

[s3_web]
bind_addr = "[::]:3903"
root_domain = ".s3.yourdomain.com"

Step 3: Initializing the Cluster

Once the Garage service is running on all nodes, they need to be introduced to one another. This is done via the CLI using the garage node connect command. Unlike other systems, Garage nodes don't automatically form a cluster; this manual 'handshake' ensures you have full control over the topology.

After connecting the nodes, you must define the Layout. Garage uses a 3-tier location system (Zone, Capacity, and Tag). For an Oracle Cloud setup, you might set the "zone" to the specific AD (Availability Domain) the instance resides in. Once the layout is configured, you apply it, and Garage begins the process of partitioning the virtual keyspace across the nodes.

Data Redundancy and Reliability

By default, Garage targets 3-way replication. This means that every piece of data uploaded to your S3 endpoint is stored on three different physical nodes. In a four-node cluster, you can lose one node entirely without losing access to any data. If two nodes go down, the data remains safe on the remaining two, though the cluster may enter a read-only state depending on your quorum settings.

Effectively, you are trading raw storage capacity for high availability. If you have 200GB of total block storage across four nodes (50GB each), your usable S3 capacity will be approximately 66GB (200GB divided by 3 replicas).

Performance Optimization on ARM

The Ampere A1 nodes are surprisingly powerful, but I/O performance on cloud block volumes can be a bottleneck. To optimize your Garage cluster:

  • Filesystem Choice: Use XFS or Ext4 for your data partitions. XFS is generally preferred for large-scale storage due to its handling of parallel I/O.
  • Memory Tuning: While Garage is lightweight, increasing the metadata cache in the config file can significantly speed up LIST operations if you have thousands of small objects.
  • Compression: Garage supports Zstandard (zstd) compression. Enabling this can save disk space and improve I/O throughput at the cost of slight CPU cycles—a great trade-off for the 4-core ARM CPUs.

Conclusion: A Sovereign Cloud Strategy

Building a distributed S3 storage layer using Garage on Oracle ARM VPS is more than just a cost-saving exercise; it is an architectural statement. It proves that with the right open-source tools, high-availability infrastructure is accessible to anyone. Whether you are storing backups for your personal projects, hosting static assets for a high-traffic blog, or building a private container registry, this setup provides a robust, S3-compatible foundation that scales with your needs.

By mastering Garage, you move away from provider lock-in and toward a truly decentralized infrastructure where your data is resilient, accessible, and, most importantly, under your control.