Deploying mission-critical application updates without dropping a single active TCP/HTTP connection is a non-negotiable benchmark in high-scale enterprise engineering. At AxonFlow AI, our cloud architecture pods enforce strict 99.99% SLA standards across healthcare, fintech, and real-time inference workloads. In this technical postmortem, we break down the exact deployment architecture we implement across AWS EKS and Google Cloud GKE clusters.

1. The Four Golden Rules of Zero-Downtime Deployments

Achieving zero downtime is never achieved through a single Kubernetes flag. It requires an orchestrated handshake between the cloud load balancer, container lifecycle hooks, application connection draining, and the relational database schema layer:

  • Graceful Shutdown (SIGTERM Handling): When Kubernetes terminates a pod, the application container must catch SIGTERM, immediately reject new traffic on its health check endpoint, yet complete in-flight transactions within a guaranteed grace period (typically 30 seconds).
  • Synchronized Ingress Endpoint De-registration: Kubernetes kube-proxy and cloud ingress controllers (AWS ALB Controller / NGINX Ingress) update iptables asynchronously. Without a pre-stop sleep hook, traffic continues arriving at terminating pods, producing random 502 Bad Gateway spikes.
  • Dual Readiness & Liveness Probes: Readiness probes dictate when traffic enters; liveness probes dictate when a frozen pod is restarted. They must never share the same failure threshold or execution frequency.
  • Decoupled Multi-Phase Schema Migrations: The application code and database schema must maintain n-1 backward and forward compatibility at all times.

2. Production Kubernetes Deployment Specification

Below is our hardened production Kubernetes manifest demonstrating bidirectional readiness gating, maxSurge headroom, and preStop execution delays:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: axonflow-core-api
  namespace: production
  labels:
    app.kubernetes.io/name: core-api
    app.kubernetes.io/part-of: axonflow-platform
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Spin up new pods before killing old ones
      maxUnavailable: 0    # Never allow capacity to dip below 100%
  selector:
    matchLabels:
      app: core-api
  template:
    metadata:
      labels:
        app: core-api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: api
        image: axonflow/core-api:v2.4.1
        ports:
        - containerPort: 8080
          name: http
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"] # Critical: Allows Ingress iptables propagation
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 2
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
          timeoutSeconds: 3
          failureThreshold: 5

3. Blue-Green Traffic Shifting with ArgoCD & Istio

For high-risk core releases, standard rolling updates do not allow canary telemetry validation. We employ Istio VirtualService weights paired with Argo Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-canary-rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: { duration: 10m } # Real-time Prometheus error rate analysis
      - setWeight: 20
      - pause: { duration: 15m }
      - setWeight: 50
      - pause: { duration: 10m }
      trafficRouting:
        istio:
          virtualService:
            name: api-vservice
            routes:
            - primary

4. The Expand-Contract Database Schema Migration Pattern

The most common cause of deployment failure is a breaking database column rename. We mandate the 3-step Expand & Contract methodology:

  1. Phase 1 (Expand): Introduce the new column (e.g., phone_e164 alongside phone_number) with a database trigger mirroring writes. Deploy the migration without touching existing application queries.
  2. Phase 2 (Migrate): Backfill historic data using batched background jobs running during off-peak hours. Deploy application code that reads from phone_e164 and writes to both.
  3. Phase 3 (Contract): Once the canary deployment reaches 100% saturation for 48 hours without rollbacks, drop the legacy column and triggers in an isolated maintenance operation.

5. Automated Rollback Safeguards with Prometheus Metrics

Deployments must never rely on manual human observation. Prometheus alerting rules monitor HTTP 5xx ratios and P99 latency thresholds. If P99 response time exceeds 650ms or 5xx error rate spikes above 0.25% over a 2-minute rolling window, the deployment initiates an automated instantaneous rollback via the ArgoCD notification engine.

← Back to Engineering Blog Schedule Architecture Review →