Architecting a Multi-Region Kubernetes Mesh on AWS with Terraform, Linkerd, and GitOps
Section 1: The Multi-Region Mandate: Achieving Real High Availability
Relying on a single cloud region is a major risk for enterprise platforms. Even when utilizing multiple Availability Zones (AZs), region-wide outages can happen due to natural disasters, backbone network cuts, or global configuration errors in AWS services like IAM or S3. For mission-critical systems, an active-active multi-region infrastructure is the gold standard for high availability, ensuring that if one region fails, traffic is redirected smoothly to the healthy region with minimal disruption.
However, op\x65rating across multiple regions introduces significant technical challenges. You must manage latency across regions, handle data synchronization with write conflicts, maintain identical configuration states across clusters, and orchestrate global traffic routing.
Active-active multi-region systems target a Recovery Time Objective (RTO) of seconds and a Recovery Point Objective (RPO) of milliseconds. Achieving this requires routing traffic to the closest cluster to minimize regional round-trip times (RTT) and establishing network links between clusters for secure inter-service communication.
In this architect's guide, we will walk through building an active-active multi-region Kubernetes mesh across AWS regions (e.g., \x60us-east-1\x60 and \x60eu-west-1\x60). We will deploy Amazon EKS clusters using Terraform, link them with the Linkerd Service Mesh for cross-cluster mTLS communication, and manage all deployments using GitOps with ArgoCD.
Section 2: Networking Infrastructure: VPC Peering and Route Tables
Before EKS clusters can communicate, they must have network connectivity. We will create two VPCs with non-overlapping CIDR blocks and peer them using an AWS VPC Peering Connection.
\x60\x60\x60text Region: us-east-1 (VPC A) Region: eu-west-1 (VPC B) +-----------------------+ +-----------------------+ | CIDR: 10.100.0.0/16 |<------------->| CIDR: 10.200.0.0/16 | | EKS Control Plane | VPC Peering | EKS Control Plane | +-----------------------+ +-----------------------+ \x60\x60\x60
Terraform Networking Module:
Here is the Terraform configuration to establish the cross-region VPC peering and update the routing tables:
\x60\x60\x60hcl
main.tf
provider "aws" { alias = "primary" region = "us-east-1" }
provider "aws" { alias = "secondary" region = "eu-west-1" }
Create VPC A in us-east-1
resource "aws_vpc" "vpc_primary" { provider = aws.primary cidr_block = "10.100.0.0/16" enable_dns_hostnames = true enable_dns_support = true tags = { Name = "vpc-us-east-1" } }
Create VPC B in eu-west-1
resource "aws_vpc" "vpc_secondary" { provider = aws.secondary cidr_block = "10.200.0.0/16" enable_dns_hostnames = true enable_dns_support = true tags = { Name = "vpc-eu-west-1" } }
Request VPC Peering from Primary to Secondary
resource "aws_vpc_peering_connection" "peering" { provider = aws.primary vpc_id = aws_vpc.vpc_primary.id peer_vpc_id = aws_vpc.vpc_secondary.id peer_region = "eu-west-1" auto_accept = false }
Accept VPC Peering in Secondary
resource "aws_vpc_peering_connection_accepter" "accepter" { provider = aws.secondary vpc_peering_connection_id = aws_vpc_peering_connection.peering.id auto_accept = true }
Update route tables in Primary to route B traffic through peering connection
resource "aws_route" "primary_to_secondary" { provider = aws.primary route_table_id = aws_vpc.vpc_primary.main_route_table_id destination_cidr_block = aws_vpc.vpc_secondary.cidr_block vpc_peering_connection_id = aws_vpc_peering_connection.peering.id }
Update route tables in Secondary to route A traffic through peering connection
resource "aws_route" "secondary_to_primary" { provider = aws.secondary route_table_id = aws_vpc.vpc_secondary.main_route_table_id destination_cidr_block = aws_vpc.vpc_primary.cidr_block vpc_peering_connection_id = aws_vpc_peering_connection.peering.id } \x60\x60\x60
Section 3: Deploying EKS Clusters with Terraform
With our networking in place, we can provision our EKS control planes. We will create identical clusters with region-specific nodes. We use managed node groups with customized launch templates to handle OS settings and disk mounts:
\x60\x60\x60hcl
eks-clusters.tf
module "eks_primary" { source = "terraform-aws-modules/eks/aws" version = "20.8.5" providers = { aws = aws.primary } cluster_name = "eks-us-east-1" cluster_version = "1.29"
vpc_id = aws_vpc.vpc_primary.id subnet_ids = aws_subnet.primary_subnets[*].id
eks_managed_node_groups = { core = { min_size = 3 max_size = 10 desired_size = 3 instance_types = ["m6i.xlarge"]
labels = {
environment = "production"
region = "us-east-1"
}
}
} }
module "eks_secondary" { source = "terraform-aws-modules/eks/aws" version = "20.8.5" providers = { aws = aws.secondary } cluster_name = "eks-eu-west-1" cluster_version = "1.29"
vpc_id = aws_vpc.vpc_secondary.id subnet_ids = aws_subnet.secondary_subnets[*].id
eks_managed_node_groups = { core = { min_size = 3 max_size = 10 desired_size = 3 instance_types = ["m6i.xlarge"]
labels = {
environment = "production"
region = "eu-west-1"
}
}
} } \x60\x60\x60
Section 4: Service Mesh Fed\x65ration with Linkerd
Now that EKS clusters are running, services on Cluster A must communicate securely with services on Cluster B. We use Linkerd Multi-cluster to establish a secure cross-cluster service mesh.
Linkerd works by exporting services from one cluster to another. It deploys a gateway controller that listens on a public load balancer IP. Traffic between clusters is encrypted via mutual TLS (mTLS) automatically.
\x60\x60\x60text Cluster A: us-east-1 Cluster B: eu-west-1 +------------------------+ +------------------------+ | Pod: order-service | | Pod: inventory-service | | | | Cross-Region | ^ | | (Linkerd Proxy) | mTLS | (Linkerd Proxy) | | v | (Port 4143) | | | | Linkerd Gateway A -----+---------------+-----> Linkerd Gateway B| +------------------------+ +------------------------+ \x60\x60\x60
Steps to Configure Mesh Fed\x65ration:
- Shared Trust Anchor: Gen\x65rate a CA root certificate and issue intermediate issuer certificates to each cluster. Both clusters must share the same trust anchor to validate each other's certificates for mTLS.
- Deploy Linkerd: Install the Linkerd control plane on both clusters.
- Deploy Multi-cluster Extension: Install Linkerd multi-cluster extension which provisions the \x60linkerd-gateway\x60 pod.
- Link Clusters: Run the link configuration to connect Cluster A to Cluster B:
\x60\x60\x60bash
Extract credential secret from Cluster B and apply to Cluster A
linkerd --context=cluster-b multicluster link --cluster-name cluster-b | kubectl --context=cluster-a apply -f - \x60\x60\x60
To make a service on Cluster B visible to Cluster A, we annotate the service in Cluster B:
\x60\x60\x60yaml
service-inventory.yaml
apiVersion: v1 kind: Service metadata: name: inventory-service namespace: default annotations: mirror.linkerd.io/exported: "true" spec: ports: - port: 8080 targetPort: 8080 selector: app: inventory \x60\x60\x60
Linkerd's mirror controller on Cluster A will detect this annotation and automatically spin up a mirror service named \x60inventory-service-cluster-b\x60 in Cluster A's namespace. The \x60order-service\x60 on Cluster A can now query \x60inventory-service-cluster-b.default.svc.cluster.local\x60 as if it were a local resource.
Section 5: GitOps CD with ArgoCD: Single Source of Truth
To ensure that both EKS clusters remain identical in configuration and deployment states, we use a GitOps pipeline powered by ArgoCD. We adopt the Application of Applications pattern.
\x60\x60\x60text Git Repository (gitops-infra) +-- apps/ | +-- linkerd.yaml (Argo Application) | +-- order-service.yaml (Argo Application) | +-- inventory-service.yaml (Argo Application) +-- bootstrap.yaml (Main Argo Application) \x60\x60\x60
We deploy ArgoCD in Cluster A, and register Cluster B as an external managed cluster.
\x60\x60\x60yaml
bootstrap.yaml
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root-bootstrap namespace: argocd spec: project: default source: repoURL: 'https://github.com/raxcore/gitops-infra.git' targetRevision: HEAD path: apps destination: server: 'https://kubernetes.default.svc' # Deploy to cluster A namespace: argocd syncPolicy: automated: prune: true selfHeal: true \x60\x60\x60
The config for the multi-region microservice application routes different values for each cluster target using an ApplicationSet:
\x60\x60\x60yaml
apps/order-service-multi.yaml
apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: order-service-appset namespace: argocd spec: gen\x65rators: - list: - cluster: cluster-primary server: https://kubernetes.default.svc region: us-east-1 - cluster: cluster-secondary server: https://api.cluster-secondary.internal:6443 region: eu-west-1 template: metadata: name: '{{cluster}}-order-service' spec: project: default source: repoURL: 'https://github.com/raxcore/gitops-infra.git' targetRevision: main path: helm/order-service helm: valueFiles: - values.yaml - 'values-{{region}}.yaml' destination: server: '{{server}}' namespace: default syncPolicy: automated: prune: true selfHeal: true \x60\x60\x60
Section 6: Active-Active Global Traffic Management (GTM)
With services fed\x65rated and synchronized across both regions, the final layer is routing users. We utilize AWS Route 53 with Latency-Based Routing and Health Checks.
\x60\x60\x60text [ User Request ] | Route 53 Latency Router +----------+----------+ Latency: 15ms Latency: 120ms v v Region A: us-east-1 Region B: eu-west-1 \x60\x60\x60
Route 53 Resource Configurations
Here is the Terraform configuration defining the Latency-Based routing policy and integration with Route 53 health checks:
\x60\x60\x60hcl
route53.tf
resource "aws_route53_health_check" "primary_health" { provider = aws.primary fqdn = "lb-primary.raxcore.dev" port = 443 type = "HTTPS" resource_path = "/healthz" failure_threshold = 3 request_interval = 10 tags = { Name = "primary-region-healthcheck" } }
resource "aws_route53_health_check" "secondary_health" { provider = aws.secondary fqdn = "lb-secondary.raxcore.dev" port = 443 type = "HTTPS" resource_path = "/healthz" failure_threshold = 3 request_interval = 10 tags = { Name = "secondary-region-healthcheck" } }
resource "aws_route53_record" "app_primary" { zone_id = "Z0123456789ABCDEFGH" name = "app.raxcore.dev" type = "A"
latency_routing_policy { region = "us-east-1" }
set_identifier = "us-east-1-endpoint" alias { name = "lb-primary.raxcore.dev" zone_id = "Z35DXBAMF5QPZC" # Canonical E-LB Zone ID for us-east-1 evaluate_target_health = true } }
resource "aws_route53_record" "app_secondary" { zone_id = "Z0123456789ABCDEFGH" name = "app.raxcore.dev" type = "A"
latency_routing_policy { region = "eu-west-1" }
set_identifier = "eu-west-1-endpoint" alias { name = "lb-secondary.raxcore.dev" zone_id = "ZH11TAHCK3B54" # Canonical E-LB Zone ID for eu-west-1 evaluate_target_health = true } } \x60\x60\x60
If the primary region fails, Route 53 removes the corresponding DNS record. Traffic is redirected to the healthy secondary region within 10 seconds.
Section 7: Multi-Region Database Sync: Amazon Aurora Global Database
Stateful applications require database replication. We deploy Amazon Aurora PostgreSQL Global Database.
- Primary Cluster: Hosted in \x60us-east-1\x60 handling read and write queries.
- Secondary Cluster: Hosted in \x60eu-west-1\x60 acts as a read replica with sub-second physical replication latency (typically < 100ms).
- Write Forwarding: EKS pods in \x60eu-west-1\x60 can send SQL writes to their local database cluster. Aurora automatically forwards these writes to the primary writer in \x60us-east-1\x60, simplifying application code.
\x60\x60\x60text Region: us-east-1 (EKS A) Region: eu-west-1 (EKS B) +-----------------------+ +-----------------------+ | App: Write/Read Query | | App: Read Query (Local)| | | | | App: Write Query | | v | | | (Forwarded) | | Aurora Writer (Primary)<--------------+-------v | +-----------------------+ +-----------------------+ \x60\x60\x60
Disaster Recovery Failover CLI Commands
In the event of a total failure of the primary region (\x60us-east-1\x60), run the following AWS CLI commands to promote the secondary database (\x60eu-west-1\x60) to become the new primary writer:
\x60\x60\x60bash
Step 1: Disconnect the secondary cluster from the global database
aws rds detach-db-cluster-from-global-cluster
--region eu-west-1
--db-cluster-identifier aurora-eu-west-1-cluster
--global-cluster-identifier aurora-global-db
Step 2: Promote the detached cluster to standalone primary writer
aws rds promote-read-replica-db-cluster
--region eu-west-1
--db-cluster-identifier aurora-eu-west-1-cluster
\x60\x60\x60
Section 8: Multi-Region Observability and Monitoring
Monitoring cross-region network links and EKS resource metrics requires global metric collection. We use Thanos Querier to aggregate data from Prometheus instances in both regions.
Thanos Query Engine Configuration
This manifest defines a Thanos Querier deployment that queries Prometheus metrics across regions:
\x60\x60\x60yaml
thanos-querier.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: thanos-querier namespace: monitoring spec: replicas: 2 selector: matchLabels: app: thanos-querier template: metadata: labels: app: thanos-querier spec: containers: - name: thanos-querier image: quay.io/thanos/thanos:v0.31.0 args: - "query" - "--log.level=info" - "--query.replica-label=prometheus_replica" - "--store=prometheus-us-east-1-sidecar.monitoring.svc:10901" - "--store=prometheus-eu-west-1-sidecar-mirror.monitoring.svc:10901" ports: - name: http containerPort: 10902 - name: grpc containerPort: 10901 livenessProbe: httpGet: path: /-/healthy port: http readinessProbe: httpGet: path: /-/ready port: http \x60\x60\x60
Multi-Region Metrics Checker Script
Run this script to monitor replication lag and connection status between regions:
\x60\x60\x60bash #!/usr/bin/env bash
cross-region-healthcheck.sh
set -euo pipefail
PRIMARY_REGION="us-east-1" SECONDARY_REGION="eu-west-1" LATENCY_LIMIT_MS=150
echo "=== Verifying multi-region network latency ===" PING_TIME=\x24(ping -c 5 -q lb-secondary.raxcore.dev | tail -1 | awk -F '/' '{print \x245}') echo "Av\x65rage Ping Latency to Secondary Region: \x24{PING_TIME} ms"
if (( \x24(echo "\x24{PING_TIME} > \x24{LATENCY_LIMIT_MS}" | bc -l) )); then echo "[WARNING] Latency exceeds threshold of \x24{LATENCY_LIMIT_MS}ms!" else echo "[OK] Latency is within normal op\x65rating limits." fi
echo "=== Querying Aurora PostgreSQL Global Replication Lag ==="
DB_LAG_BYTES=\x24(aws rds describe-db-clusters
--region "\x24{SECONDARY_REGION}"
--db-cluster-identifier "aurora-eu-west-1-cluster"
--query "DBClusters[0].PendingModifiedValues"
--output text)
echo "Aurora replication lag metadata status: \x24{DB_LAG_BYTES}" echo "All validation checks complete." \x60\x60\x60
Section 9: Multi-Region Infrastructure Metrics
Op\x65rating this active-active multi-region mesh allows EKS architectures to achieve:
- Recovery Time Objective (RTO) < 15 seconds: The duration required to redirect global traffic via Route 53 DNS failover when a region crashes.
- Recovery Point Objective (RPO) < 1 second: The maximum data loss risk, governed by the asynchronous replication lag of Aurora Global Database.
- Zero Down Time Maintenance: Upgrade cluster versions or node pools in region A while serving 100% of traffic from region B, then swap.
Cross-Region Latency Performance and Traffic Failover Criteria
Active-active EKS topologies require continuous monitoring of regional health. When latency thresholds are exceeded, global traffic routing must failover to prevent service degradation.
| Region Pair | Av\x65rage Latency (RTT) | Target Failover Threshold | | :--- | :--- | :--- | | us-east-1 to eu-west-1 | 72ms | 120ms | | us-east-1 to us-west-2 | 34ms | 60ms |
Below is a Prometheus alert configuration rule for monitoring EKS cross-region cluster connectivity:
\x60\x60\x60yaml groups:
- name: kubernetes-mesh-alerts
rules:
- alert: CrossRegionLatencyHigh expr: linkerd_mesh_latency_seconds{quantile="0.95"} > 0.12 for: 1m labels: severity: warning annotations: summary: "EKS cross-region latency exceeds target threshold" description: "Latency between EKS clusters is {{ \x24value }} seconds." \x60\x60\x60
Optimization Specification Details Section 1
In high-performance settings, engineers prioritize scheduling metrics and cache availability. The transition from legacy monolithic configurations to microservices platforms is a key progression for high-availability infrastructures. By partition-based loading, systems prevent thread starvation, optimizing runtime capacities and resources.
