Scaling On-Demand TLS: Optimizing Caddy Server v3 for 50,000+ Customer Domains

Introduction: The Multi-Tenant Domain Challenge

For modern Software-as-a-Service (SaaS) platforms, e-commerce builders, and content management systems, allowing users to point their custom domains to your infrastructure is a standard requirement. However, managing SSL/TLS certificates at scale for tens of thousands of unique hostnames introduces massive operational complexity.

Traditionally, infrastructure teams had to rely on complex cron jobs, message queues, and custom scripts to orchestrate Let's Encrypt or ZeroSSL API calls, often hitting strict rate limits or suffering from slow provisioning times. Caddy Server v3 revolutionizes this workflow through its native On-Demand TLS feature. Instead of pre-generating certificates, Caddy issues them on-the-fly during the TLS handshake. In this guide, we will explore how to architect, optimize, and secure Caddy Server v3 to handle automated SSL issuance for over 50,000 custom customer domains seamlessly.

Understanding Caddy v3 On-Demand TLS Architecture

Before diving into the configuration, it is critical to understand how On-Demand TLS operates during a connection request. When an un-cached custom domain connects to your Caddy instance, the server executes a rapid internal lifecycle:

1. The TLS Handshake Begins: A client initiates a connection via Server Name Indication (SNI).
2. Local Cache Lookup: Caddy checks its internal storage to see if a valid certificate already exists for that specific hostname.
3. The Ask Endpoint Validation: If the certificate is missing, Caddy makes an internal HTTP GET request to a designated backend API (the "Ask" endpoint) to verify if the domain is authorized to use your service.
4. Certificate Issuance: If the backend returns a 200 OK, Caddy instantly requests a certificate from an ACME provider (e.g., Let's Encrypt) and completes the handshake. If the backend returns a 4xx, the handshake is aborted immediately.

Crucial Guardrail: Never enable On-Demand TLS without an ask endpoint. Without verification, malicious actors could point arbitrary domains to your IP, causing your server to exhaust ACME rate limits and fill local storage with useless certificates, triggering a distributed denial-of-service (DDoS) vulnerability.

Step-by-Step Production Configuration

To support a massive multi-tenant scale of 50,000+ domains, your Caddyfile needs to transition from basic syntax into an enterprise-grade configuration utilizing global options, robust storage backends, and strict rate limits.

1. Global Options and Storage Clustering

At scale, running a single Caddy instance introduces a single point of failure. You must deploy Caddy in a cluster behind a layer 4 load balancer. This requires a shared storage provider so all Caddy instances can access the same certificate pool.

{
    # Define global storage via Redis or PostgreSQL for clustering
    storage redis {
        address "redis-cluster.internal:6379"
        username "caddy_node"
        password "env:REDIS_PASSWORD"
        db      0
    }

    # Global TLS parameters
    tls {
        on_demand_tls {
            ask      [https://api.yourcompany.com/v1/validate-domain](https://api.yourcompany.com/v1/validate-domain)
            interval 2m
            burst    10
        }
    }
}

In this snippet, the ask parameter points to your internal service. The interval and burst settings act as a native rate limiter, restricting on-the-fly certificate generation to a maximum of 10 within a 2-minute window to prevent abuse during traffic spikes.

2. The Multi-Tenant Reverse Proxy Block

Next, define the site block that will dynamically capture incoming requests across all 50,000+ customer hostnames.

:443 {
    # Enable on-demand certificate issuance for all incoming traffic matching this block
    tls {
        on_demand
    }

    # Compress response payloads to optimize bandwidth
    encode gzip zstd

    # Route traffic to internal application clusters
    reverse_proxy {
        to http://app-backend-cluster
        
        # Pass authentic header data to your application
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Proto {http.request.scheme}
    }
}

Designing a High-Performance 'Ask' Endpoint

Because Caddy queries your ask endpoint during the initial TLS handshake, its performance directly impacts your connection establishment speed. A slow endpoint creates a bottleneck. Follow these best practices to ensure sub-millisecond response times:

• In-Memory Database Caching: Store the list of allowed custom domains in a fast cache layer like Redis rather than querying a relational database (like PostgreSQL or MySQL) directly for every TLS handshake.
• Keep-Alive Connections: Ensure the network communication between Caddy and your Ask API utilizes HTTP/2 or persistent HTTP/1.1 connections to eliminate TCP handshake overhead.
• Minimal Payload: Caddy only requires an HTTP status code. Your endpoint should return an empty body with a 200 OK status for authorized domains or 404 Not Found for unauthorized ones.

Advanced Optimization for 50,000+ Domains

Operating at this volume requires fine-tuning operating system boundaries and optimizing how Caddy handles memory management and ACME fallback mechanics.

Operating System Level Tuning

Caddy relies heavily on system resources to maintain tens of thousands of open connections. Modify your Linux kernel parameters under /etc/sysctl.conf to prevent resource exhaustion:

Increase the maximum number of open files and file descriptors to handle concurrent TLS handshakes safely:

fs.file-max = 2097152
fs.inotify.max_user_watches = 524288

Additionally, optimize your system network stack to allow rapid local port recycling and larger TCP window sizes.

ACME CA Provider Redundancy

Relying on a single Certificate Authority (CA) presents a severe risk if they experience an outage. Caddy v3 allows configuring secondary issuers. If Let's Encrypt times out or hits a localized rate limit, Caddy automatically switches to the alternative provider to fulfill the on-demand request.

tls {
    on_demand
    issuer acme {
        email [email protected]
        dir   [https://acme-v02.api.letsencrypt.org/directory](https://acme-v02.api.letsencrypt.org/directory)
    }
    issuer acme {
        email [email protected]
        dir   [https://acme.zerossl.com/v2/DV90](https://acme.zerossl.com/v2/DV90)
    }
}

Monitoring, Observability, and Alerting

Maintaining high availability across a massive custom domain portfolio requires deep observability. Caddy v3 exposes comprehensive structural telemetry via Prometheus metrics natively. You should continuously monitor key performance metrics to catch infrastructure stress before it affects end-users:

• caddy_tls_certificates_total: Monitors the total volume of active certificates held in memory and local storage caches.
• caddy_tls_on_demand_issuances_total: Tracks the rate of successful on-the-fly certificate generations over time.
• caddy_tls_on_demand_handshake_errors_total: Measures failed handshakes, signaling either misconfigured client DNS records or issues with your internal validation backend.

By coupling these Prometheus metrics with a dashboard in Grafana, operations teams can quickly set up active alerting systems that trigger notifications if the percentage of failed handshakes spikes, indicating a potential configuration issue or a coordinated domain-targeting attack.

Conclusion

Transitioning from complex certificate orchestration scripts to Caddy v3's On-Demand TLS allows you to treat SSL management as a seamless, automated utility. By backing Caddy with a highly scalable, cached Ask endpoint, distributing your nodes across a Redis cluster, and establishing fallback ACME providers, your platform can confidently scale past 50,000 customer domains. This setup minimizes operational overhead while maintaining top-tier security, performance, and reliability for your multi-tenant ecosystem.