Edge Asset Optimization: Building a Distributed Media Processing Pipeline with Fly.io and Benthos on Private VPS

Introduction: The Paradox of Modern Asset Delivery

In the digital-first business landscape, delivering media assets—images, videos, dynamic banners, and heavy payloads—with minimal latency is paramount to user retention and conversion rates. Traditionally, organizations rely on commercial Content Delivery Networks (CDNs) or heavy cloud-native media pipelines. However, as enterprise scale increases, these solutions often introduce two critical challenges: unpredictable egress costs and vendor lock-in.

What if you could build a bespoke, globally distributed asset processing and optimization layer that leverages the low latency of edge computing while keeping your heavy-lifting storage and compute on your cost-effective, private Virtual Private Servers (VPS)? This architectural guide demonstrates how to build an Edge Asset Optimization solution by combining the global orchestration of Fly.io with the lightweight, declarative stream processing power of Benthos (now known as Redpanda Connect).

---

Understanding the Component Stack

Before diving into the implementation, it is vital to understand why this specific stack offers an unparalleled balance of performance, cost, and operational simplicity.

1. Fly.io: Micro-VMs at the Global Edge

Fly.io transforms applications into globally distributed services by running servers on physical machines around the world. It utilizes lightweight Firecracker micro-VMs that boot in milliseconds. For asset optimization, Fly.io acts as our smart routing and edge-compute layer, intercepting user requests closest to their geographic location.

2. Benthos: The Swiss Army Knife of Data Engineering

Benthos is a high-performance, resilient stream processor. It is statistically typed, memory-efficient, and configured entirely via simple declarative YAML files. In our architecture, Benthos functions as an on-the-fly mutation engine, capable of fetching, transforming, compressing, and caching assets seamlessly.

3. The Private VPS: Cost-Effective Core Storage

While the edge handles immediate requests and transformations, your private VPS (hosted on providers like Hetzner, OVH, or DigitalOcean) serves as the source of truth—holding the original, unoptimized high-resolution assets and the primary database. This separation ensures you do not pay premium cloud storage and egress fees.

---

Architectural Overview and Data Flow

The system operates on a hybrid topology designed to minimize time-to-first-byte (TTFB) while protecting the origin server from traffic spikes.

1. The Request: A user requests an asset (e.g., an unoptimized banner image) via a URL pointing to the nearest Fly.io edge node.
2. The Edge Evaluation: Benthos running on the Fly.io micro-VM checks its local in-memory or volume cache for the optimized asset.
3. The Origin Fetch (Cache Miss): If the asset is missing, Benthos securely fetches the raw asset from your centralized private VPS via an optimized HTTP/2 or WireGuard tunnel.
4. On-the-Fly Optimization: Benthos processes the asset (e.g., converting PNG to WebP, resizing, compressing text payloads).
5. Delivery & Caching: The optimized asset is cached at the edge node and simultaneously streamed back to the user.

Key Benefit: Future requests for the same asset in that geographic region hit the edge cache directly, reducing origin VPS load to near zero and eliminating repetitive cross-network data transfer.

---

Step-by-Step Implementation Guide

Step 1: Preparing Your Origin Private VPS

Your private VPS must expose an internal or secure public endpoint where raw assets are archived. Ensure your asset directory is served efficiently using a lightweight web server like Nginx, protected by token authentication or IP whitelisting that only allows requests coming from Fly.io's network ranges.

Step 2: Configuring the Benthos Optimization Pipeline

The core intelligence of this solution lies in the Benthos configuration. We define an input that listens to HTTP traffic, a processing pipeline that manipulates the data, and an output that serves the result. Create a file named benthos.yaml:

input:
  http_server:
    path: /assets/{asset_id}
    method: GET

pipeline:
  processors:
    - try:
      # 1. Check local Fly.io cache
      - cache:
          operator: get
          resource: edge_cache
          key: '${! meta("benthos_http_path") }'
      
      # 2. Catch failures (Cache Miss) and fetch from Origin VPS
      - catch:
        - http:
            url: [https://your-private-vps.internal/raw-assets/$](https://your-private-vps.internal/raw-assets/$){! error_joined() }
            method: GET
        # 3. Perform asset optimization (Example: Triggering an image mutation script or internal processor)
        - mutation: |
            root = this
            meta.optimized = "true"
        # 4. Populate cache for subsequent requests
        - cache:
          operator: set
          resource: edge_cache
          key: '${! meta("benthos_http_path") }'
          value: '${! this }'

output:
  sync: {}

Step 3: Containerizing for Fly.io

To deploy Benthos onto Fly.io, package it inside a minimal Docker container. Create a standard Dockerfile:

FROM jeffail/benthos:latest
COPY benthos.yaml /config.yaml
ENTRYPOINT ["benthos", "-c", "/config.yaml"]

Step 4: Deploying to Fly.io’s Global Regions

Initialize your Fly application using the Fly CLI. Run the following command in your terminal:

fly launch --name edge-asset-optimizer --no-deploy

Modify the newly generated fly.toml file to establish persistent storage volumes for your edge cache. This ensures that even if a micro-VM restarts, your optimized assets remain cached locally:

[[mounts]]
  source = "cache_vol"
  destination = "/cache"

[http_service]
  internal_port = 4195
  force_https = true

Deploy your application globally across multiple targets (e.g., lax for Los Angeles, fra for Frankfurt, sin for Singapore) to match your primary user demographic:

fly deploy
fly scale count 3 --region lax,fra,sin

---

Performance and Cost Analysis

By establishing this architecture, enterprise infrastructure teams realize immediate structural improvements over traditional setups:

Metric	Traditional Centralized VPS	Commercial CDN + S3 Pipeline	Fly.io + Benthos Edge Solution
Time-to-First-Byte (TTFB)	High (Depends on distance)	Very Low	Very Low
Data Egress Costs	Moderate to High	Extremely High (Cloud Provider Fees)	Predictable & Minimal
Operational Control	Absolute Control	Vendor Locked / Limited Rules	Absolute Programmatic Control
Scalability	Manual / Hard Limits	Automated / Expensive	Automated / Elastic Micro-VMs

Because Benthos operates completely in-memory or reads from local SSD blocks attached to the Fly.io micro-VMs, latency for cached assets drops to single-digit milliseconds. More importantly, because your asset optimization logic is written in declarative code, your team can add features like dynamic WebP/AVIF compression, real-time log streaming, and rate-limiting without re-architecting the core infrastructure.

---

Conclusion: Future-Proofing Your Delivery Stack

Building your own Edge Asset Optimization solution is no longer a luxury reserved for massive tech conglomerates. By combining the global, instantaneous deployment capabilities of Fly.io with the ultra-lightweight processing efficiency of Benthos, and backing it with the raw, economical compute power of your private VPS, you create a hybrid infrastructure that delivers elite performance without elite pricing.

As your user base expands, scale your edge nodes with a single command, keeping your core business logic and assets securely anchored on your own infrastructure.