Enterprise Multi-Region Database DR & HA Architecture

In modern mission-critical Kubernetes environments, achieving high availability (HA) within a single region is no longer sufficient. Enterprise resilience demands a robust Disaster Recovery (DR) strategy spanning geographically dispersed cloud regions. This guide details how to construct an automated, declarative, multi-region PostgreSQL architecture using CloudNativePG, AWS S3 Cross-Region Replication (CRR), and ArgoCD.

Architecture Blueprint

The solution connects two isolated EKS clusters located in distinct AWS regions: us-east-1 (Primary Cluster) and us-west-2 (Designated Standby DR Cluster).

• Primary Region (us-east-1): Runs an active 3-node CloudNativePG PostgreSQL cluster. Continuous Write-Ahead Logs (WAL) and full base backups are streamed to a local S3 bucket via barman-cloud-wal-archive.
• AWS S3 CRR: Replicates binary WAL archives asynchronously from the primary S3 bucket to a target S3 bucket in us-west-2 with S3 Object Lock for ransomware immutability.
• DR Region (us-west-2): Runs a CloudNativePG standby cluster configured in replica mode, continuously fetching WAL segments from the replicated S3 bucket.
• ArgoCD GitOps: Manages state declarations across both regions, orchestrating promotion, failover, and traffic redirection through declarative Git commits.

1. AWS Infrastructure Setup (S3 Replication & IRSA)

First, configure AWS S3 Cross-Region Replication with IAM roles via AWS CLI or Terraform to support low-latency WAL synchronization between regional buckets.

# Create Primary and Secondary S3 Buckets with Versioning Enabled
aws s3api create-bucket --bucket cnpg-wal-primary-us-east-1 --region us-east-1
aws s3api put-bucket-versioning --bucket cnpg-wal-primary-us-east-1 \
  --versioning-configuration Status=Enabled

aws s3api create-bucket --bucket cnpg-wal-dr-us-west-2 --region us-west-2 \
  --create-bucket-configuration LocationConstraint=us-west-2
aws s3api put-bucket-versioning --bucket cnpg-wal-dr-us-west-2 \
  --versioning-configuration Status=Enabled

# Configure S3 Cross-Region Replication (CRR) Policy
cat <<EOF > replication-rules.json
{
  "Role": "arn:aws:iam::123456789012:role/s3-crr-replication-role",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::cnpg-wal-dr-us-west-2",
        "ReplicationTime": {
          "Status": "Enabled",
          "Time": { "Minutes": 15 }
        },
        "Metrics": {
          "Status": "Enabled",
          "EventThreshold": { "Minutes": 15 }
        }
      }
    }
  ]
}
EOF

aws s3api put-bucket-replication --bucket cnpg-wal-primary-us-east-1 \
  --replication-configuration file://replication-rules.json

2. Primary CloudNativePG Manifest Configuration

Deploy the primary PostgreSQL cluster in us-east-1 with Barman Object Store integration using AWS IRSA (IAM Roles for Service Accounts).

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-primary
  namespace: database
spec:
  instances: 3
  primaryUpdateStrategy: Unsupervised
  
  storage:
    size: 100Gi
    storageClass: gp3-encrypted
    
  walStorage:
    size: 50Gi
    storageClass: gp3-encrypted

  resources:
    requests:
      cpu: "2"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "16Gi"

  postgresql:
    parameters:
      shared_buffers: "2GB"
      work_mem: "32MB"
      max_connections: "500"
      archive_timeout: "60s"

  backup:
    barmanObjectStore:
      destinationPath: s3://cnpg-wal-primary-us-east-1/
      endpointURL: https://s3.us-east-1.amazonaws.com
      s3Credentials:
        inheritFromIAMRole: true
      wal:
        compression: gzip
        maxParallel: 8
      data:
        compression: gzip
        immediateCheckpoint: true
        jobs: 4
    retentionPolicy: "30d"

3. Designated Standby Cluster Configuration (DR Region)

In us-west-2, deploy a replica CloudNativePG cluster configured as a designated standby instance reading directly from the replicated S3 bucket.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-dr
  namespace: database
spec:
  instances: 3
  
  # Configure standby replica mode
  replica:
    enabled: true
    source: s3-replicated-backup

  externalClusters:
    - name: s3-replicated-backup
      barmanObjectStore:
        destinationPath: s3://cnpg-wal-dr-us-west-2/
        endpointURL: https://s3.us-west-2.amazonaws.com
        s3Credentials:
          inheritFromIAMRole: true
        wal:
          maxParallel: 8

  storage:
    size: 100Gi
    storageClass: gp3-encrypted

  resources:
    requests:
      cpu: "2"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "16Gi"

4. GitOps Orchestration with ArgoCD ApplicationSet

To ensure deterministic multi-cluster lifecycle management, use an ArgoCD ApplicationSet using the Matrix Generator to control region-specific cluster configurations.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cloudnativepg-multiregion
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: primary-cluster-east
            region: us-east-1
            path: environments/us-east-1
          - cluster: dr-cluster-west
            region: us-west-2
            path: environments/us-west-2
  template:
    metadata:
      name: 'pg-{{region}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/enterprise/database-infra.git'
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        name: '{{cluster}}'
        namespace: database
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

5. Automated DR Failover Playbook

In the event of a catastrophic outage in us-east-1, execute the following orchestrated promotion procedure via GitOps and kubectl:

Step 1: Fence Primary Region (Prevent Split-Brain)
Update primary Git repository manifest or apply network policy isolation to prevent stale writes.

# Annotate primary cluster to prevent any incoming connections (Fencing)
kubectl annotate cluster postgres-primary -n database \
  cnpg.io/fencedInstances='["*"]' --overwrite

Step 2: Promote Standby DR Cluster in GitOps
Modify environments/us-west-2/postgres.yaml in Git to disable replica mode, converting it into a primary standalone cluster:

# environments/us-west-2/postgres.yaml update
spec:
  replica:
    enabled: false # Disables standby mode and promotes to primary

Step 3: Trigger ArgoCD Sync & Verify Promotion

# Force Sync via ArgoCD CLI
argocd app sync pg-us-west-2

# Verify Promotion status via CloudNativePG kubectl plugin
kubectl cnpg status postgres-dr -n database

Key Enterprise Metrics Comparison

Conclusion

Combining CloudNativePG declarative database management with AWS S3 Cross-Region Replication and ArgoCD GitOps provides an enterprise-ready, fully auditable multi-region DR framework. This architecture limits RPO to under 60 seconds while removing manual operational complexity during regional outages.