New Generation Distributed Storage Architecture Optimizing Costs with JuiceFS and Ceph OSD on 3 Cheap VPS
A modern distributed storage solution on 3 low-cost VPS using JuiceFS and Ceph OSD enables small and medium enterprises to build scalable, reliable, and cost-effective storage...
Image credit: TheDigitalArtist · Pixabay Content License
A modern distributed storage solution on 3 low-cost VPS using JuiceFS and Ceph OSD enables small and medium enterprises to build scalable, reliable, and cost-effective storage systems. This architecture leverages open-source technologies for distributed file management and decentralized data storage, ideal for Docker, databases, and internal storage. This guide details setup steps, configuration, testing, and optimization for Docker, databases, and internal storage.
Overview of the New Generation Distributed Storage Architecture
In the context of increasing data storage demands, businesses need a distributed storage solution that is scalable, highly reliable, and cost-effective. The new generation distributed storage architecture combining JuiceFS and Ceph OSD on 3 low-cost VPS is an efficient choice. JuiceFS is a distributed file system based on Redis and an object storage system, enabling POSIX file access. Ceph OSD (Object Storage Device) is the data storage component in a Ceph cluster, handling distributed storage, backup, and data recovery. Together, these technologies enable the construction of a powerful distributed storage system that can run on affordable NVMe-based VPS infrastructure, such as that provided at NVMe Cloud VPS for Docker, Nginx & Databases.
The main advantages of this architecture include:
- Cost Optimization: Using 3 low-cost VPS instead of investing in dedicated storage systems or high-end server clusters.
- Scalability: Easily expand the cluster by adding new VPS nodes.
- High Reliability: Ceph provides automatic data replication and recovery in case of failure.
- Easy Integration: Seamlessly integrates with Docker, Nginx, and database applications.
Requirements and Environment Preparation
To deploy this architecture, you need:
- 3 Linux-based VPS servers (Ubuntu 20.04/22.04 or CentOS 8/9).
- At least 20GB of NVMe SSD storage on each VPS.
- Stable network connectivity between the VPS nodes.
- SSH access to each VPS.
Before starting, ensure the following tools are installed on each VPS:
sudo apt update
sudo apt upgrade -y
sudo apt install -y git curl wget
Set the hostname for each VPS for easier management:
sudo hostnamectl set-hostname vps1
sudo hostnamectl set-hostname vps2
sudo hostnamectl set-hostname vps3
Setting Up Ceph OSD and JuiceFS
On each VPS, install Ceph OSD and JuiceFS. Below are the configuration steps for each component.
Configuring Ceph OSD
On each VPS, install Ceph OSD and initialize the storage directory:
sudo apt install -y ceph-osd
sudo mkdir -p /var/lib/ceph/osd/ceph-0
sudo ceph-osd --mkfs --osd-id 0 --mkkey --osd-data /var/lib/ceph/osd/ceph-0
Start the Ceph OSD service:
sudo systemctl start ceph-osd@0
sudo systemctl enable ceph-osd@0
Configuring JuiceFS
On one VPS (choose VPS1 as the management server), install JuiceFS and create the distributed file system:
curl -sSL https://get.juicefs.com | sh
juicefs format --storage=redis --metaaddr=redis://localhost:6379 --data-shards=1 --name=juicefs-storage redis://localhost:6379
juicefs mount /mnt/juicefs redis://localhost:6379
The value redis://localhost:6379 is the Redis address used to store metadata. You can replace it with a separate Redis server if needed.
Testing and Performance Optimization
After configuration, test the system operation:
- On VPS1, check the JuiceFS status:
juicefs status redis://localhost:6379
- On VPS2 and VPS3, check the Ceph OSD status:
sudo ceph osd tree
sudo ceph -s
To optimize performance, you can adjust parameters such as the number of data shards, block size, and Redis configuration to match your usage needs.
Troubleshooting and References
Common issues include:
- Unable to mount JuiceFS: Check Redis connectivity and ensure JuiceFS is correctly installed.
- Ceph OSD not running: Check logs at
/var/log/ceph/ceph-osd.log. - High latency: Ensure stable network connectivity between VPS nodes and use NVMe SSD storage.
Refer to the official documentation for more details:
To learn more about self-hosting applications on VPS, refer to the guide on Building a Self-Hosted Mini Vercel on ARM VPS with Coolify and Nixpacks.
Recommended NVMe Cloud VPS Plans to Deploy This Setup
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-3E
10% VAT included
- CPU: 2 vCPU
- RAM: 4 GB
- SSD: 50 GB NVMe
