Service Mesh & Infrastructure

Before meshes, each microservice imported its own HTTP client, TLS trust store, retry policy, and tracing headers. One team used Resilience4j with three retries; another used none; a Go service skipped trace propagation entirely. On-call incidents became archaeology: “Is the 503 from the app, the client, or the network?”

A service mesh inserts a proxy beside each pod (the sidecar pattern). All inbound and outbound traffic flows through that proxy, which enforces policies pushed by a control plane— without recompiling Java, Go, or .NET services. The mesh governs east-west traffic inside the cluster; north-south ingress still uses Gateway / load balancers, often also managed by Istio Gateway resources.

Meshes shine when you have many polyglot services on Kubernetes, ephemeral pod IPs, and a platform team that can own control plane upgrades. They complement—but do not replace—application concerns: sagas, domain authorization, and database consistency stay in code—see Data Patterns and Security.

Data plane — Envoy sidecars

Envoy is a high-performance L4/L7 proxy originally built at Lyft, now the de facto mesh data plane. In Istio, each application pod gets an istio-proxy container injected via mutating webhook. iptables or eBPF rules redirect traffic so the app calls inventory-service:8080 but packets traverse Envoy first. Envoy terminates outbound TLS, applies retries, emits access logs, creates trace spans, and load-balances across upstream pod IPs from Kubernetes endpoints.

Control plane — Istiod (successor to Pilot)

Istiod consolidates what was once split across Pilot (xDS config push), Citadel (certificate authority), and Galley (config validation). It watches Kubernetes Services, Endpoints, and Istio CRDs; computes effective config per sidecar; pushes via xDS APIs (CDS, EDS, LDS, RDS). When you apply a VirtualService, every affected Envoy hot-reloads routing without pod restart—usually within seconds.

flowchart TB
  subgraph podA [Pod Order Service]
    APP1[order-app]
    ENV1[Envoy sidecar]
    APP1 --> ENV1
  end
  subgraph podB [Pod Payment Service]
    APP2[payment-app]
    ENV2[Envoy sidecar]
    APP2 --> ENV2
  end
  ENV1 -->|mTLS| ENV2
  ISTIOD[Istiod control plane] -. xDS config .-> ENV1
  ISTIOD -. xDS config .-> ENV2

Retry multiplication is the classic foot-gun: Resilience4j retries 3× and Envoy retries 3× → up to 9 calls to a failing Payment service. Platform runbooks should assign one primary retry layer per hop—often mesh for generic HTTP, library only for payment client with idempotency. See Resilience → Retry and Observability → Mesh vs app.

Envoy sidecar — data plane

Every meshed pod runs Envoy (unless using Istio ambient mode with ztunnel—advanced topic). Envoy listeners bind to redirected ports; clusters map to upstream Kubernetes service endpoints; filters implement HTTP routing, JWT validation (optional), WASM extensions, and telemetry emission. Sidecar resource limits matter: 128–256 MiB memory per proxy × 200 pods adds up—capacity plan explicitly.

Istiod — unified control plane (evolution of Pilot + Citadel + Galley)

Key CRD families

• Networking — VirtualService, DestinationRule, Gateway, ServiceEntry, Sidecar, WorkloadEntry
• Security — PeerAuthentication, RequestAuthentication, AuthorizationPolicy
• Telemetry — Telemetry API (metrics, access logs, tracing exporters to Prometheus/Jaeger)

Sidecar injection: label namespace istio-injection=enabled or pod annotation sidecar.istio.io/inject: "true". Install profiles: demo for learning, default for production baseline. Upgrades require control plane / data plane version skew within supported window—coordinate with platform team and GitOps freeze windows.

Choose Istio when you need header-based routing, extensive AuthorizationPolicy, multi-cluster federation, or WASM extensions. Choose Linkerd when ~20–100 services need encrypted east-west traffic and golden metrics with a two-person platform team. Always benchmark p99 latency and memory on your instance types—proxy cost is measurable on t3.small nodes. Other options: Cilium (eBPF, sidecarless L3/L4), Consul Connect (HashiCorp multi-platform).

A VirtualService binds to one or more hosts (Kubernetes service DNS names like order-service.default.svc.cluster.local). Ordered http (or tcp) routes are evaluated top-down—first match wins. Each route specifies destinations (service + subset), weights, rewrites, redirects, timeouts, retries, and faults.

Routing capabilities

• URI / header match — route internal QA to v2 via x-canary: true
• Traffic shifting — 90/10 weight split between v1 and v2 subsets for canary
• Retries — per-route retry count, timeout, retriable status codes
• Timeouts — max request duration at proxy—complements app timeouts
• Fault injection — delay or abort percentage for chaos testing (see Fault injection)

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
    - order-service
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: order-service
            subset: v2
    - route:
        - destination:
            host: order-service
            subset: v1
          weight: 90
        - destination:
            host: order-service
            subset: v2
          weight: 10
      timeout: 10s
      retries:
        attempts: 2
        perTryTimeout: 3s
        retryOn: connect-failure,refused-stream,503

Subset names (v1, v2) must exist in a matching DestinationRule—VirtualService cannot route to subsets that are not defined. Attach VirtualService to Gateway for north-south hosts via spec.gateways field.

Subsets for version routing

Map pod labels to named subsets used in canary VirtualServices. During deploy, v1 and v2 pods coexist; weights shift traffic without changing Service selector. Label convention: version: v1 on Deployment pod template.

Load balancing

Algorithms: ROUND_ROBIN (default), LEAST_REQUEST, RANDOM, PASSTHROUGH. LEAST_REQUEST helps when pod CPU varies; session affinity via consistent hash on header/cookie when sticky sessions required—prefer stateless JWT over sticky where possible.

Connection pool — mesh-level bulkhead

Cap http1MaxPendingRequests, maxRequestsPerConnection, and TCP connection limits per upstream host. Prevents one slow Inventory service from holding thousands of open connections from every Order sidecar— complements app-level bulkheads in Resilience → Bulkhead.

Outlier detection — circuit breaking at infrastructure

Envoy passively ejects hosts returning consecutive 5xx or high latency—similar to circuit breaker OPEN state. Tune consecutive5xxErrors, interval, baseEjectionTime, maxEjectionPercent to avoid ejecting all pods during rolling restart (start conservative in staging).

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Gateway (Istio networking CRD) is not the same object as Kubernetes Gateway API resource—similar purpose, Istio-native model. Gateway selects which ports and hosts ingress gateway pods listen on; VirtualService with gateways: [my-gateway] binds routing rules to external traffic.

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: public-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: api-tls-cert
      hosts:
        - api.example.com
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api-routes
spec:
  hosts:
    - api.example.com
  gateways:
    - public-gateway
  http:
    - route:
        - destination:
            host: order-service
            port:
              number: 8080

Traffic path: public DNS → cloud load balancer → Istio ingress gateway pods (Envoy) → VirtualService → sidecar-protected service. cert-manager commonly provisions credentialName secrets for TLS. Internal admin APIs skip public Gateway—cluster-internal Service + AuthorizationPolicy only. Complements Communication → API Gateway patterns—some teams use Spring Cloud Gateway inside mesh, others terminate at Istio ingress only.

Traditional canary needed separate load balancers or DNS tricks. With Istio:

1. Deploy v2 Deployment with version: v2 label—pods register to same Service
2. DestinationRule defines v1 and v2 subsets
3. VirtualService starts at 95/5 or 90/10 weight split
4. Prometheus compares istio_requests_total error rate and latency v1 vs v2
5. Promote weights or revert—no DNS TTL wait

flowchart TB
  DEP[Deploy v2 pods ready] --> W5[Weight 5 percent v2]
  W5 --> OBS[Observe 15 min SLO]
  OBS -->|pass| W50[Weight 50 percent]
  OBS -->|fail| RB[Rollback 100 percent v1]
  W50 --> OBS2[Observe again]
  OBS2 --> FULL[Weight 100 percent v2]

Automate with Flagger or Argo Rollouts: analysis templates query Prometheus, auto-increment weights, abort on SLO burn. Combine with pod readiness probes and PDB from Deployment → Canary— do not shift traffic until v2 passes health checks. Gate on business metrics (checkout conversion), not only HTTP 5xx—see Observability → SLOs.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: inventory-fault-test
spec:
  hosts:
    - inventory-service
  http:
    - match:
        - headers:
            x-test-fault:
              exact: "enabled"
      fault:
        delay:
          percentage:
            value: 100
          fixedDelay: 5s
        abort:
          percentage:
            value: 10
          httpStatus: 503
      route:
        - destination:
            host: inventory-service

Delay — simulates slow dependency; verify Order service timeout and bulkhead release threads. Abort — returns 503 without hitting app; verify retry policy does not amplify load (retry storm). Scope with header match or dedicated namespace—never blanket fault injection on production without executive approval. Pair with game days validating Resilience4j and mesh outlier detection together.

Mutual TLS (mTLS) between all services

In plaintext cluster networking, any compromised pod can call any Service IP. With mesh mTLS, each sidecar presents a client certificate; receiver verifies against Istio CA. Workload identity follows SPIFFE format: spiffe://cluster.local/ns/default/sa/order-service — tied to Kubernetes service account, not pod IP (which changes on restart).

Istiod acts as CA: issues short-lived certs, rotates automatically, distributes to sidecars. Apps often still speak HTTP locally—the sidecar encrypts on the wire between proxies. User-facing OAuth/JWT at north-south gateway remains required—mTLS is service-to-service, not end-user auth. Deep dive on zero trust and JWT: Security.

sequenceDiagram
  participant A as Order sidecar
  participant B as Payment sidecar

  A->>B: ClientHello + order SA cert
  B->>A: ServerHello + payment SA cert
  A->>B: Encrypted HTTP request
  B->>A: Encrypted HTTP response

Migration path: PERMISSIVE → STRICT

PERMISSIVE accepts both plaintext and mTLS—use during rollout while legacy pods lack sidecars. STRICT rejects non-mTLS—target state for production namespaces. Monitor mix percentage in Kiali before flipping STRICT cluster-wide.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

More specific policies override broader: workload-level PeerAuthentication beats namespace default beats mesh-wide default. Pair STRICT with sidecar injection on every pod in namespace—otherwise legitimate traffic fails. VMs and external services use WorkloadEntry + ServiceEntry with appropriate mTLS config.

After mTLS proves identity, AuthorizationPolicy decides permission: “checkout-service SA may POST /api/v1/orders; inventory-service SA may GET /api/v1/orders/{id}; everyone else DENY.” Reduces lateral movement—attacker in compromised marketing pod cannot reach admin APIs.

This is not a replacement for application authorization: mesh cannot know if user A owns order 123—BOLA checks stay in Java with JWT subject validation. Mesh blocks wrong services; app blocks wrong users.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: order-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: order-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/production/sa/checkout-bff
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/v1/orders", "/api/v1/orders/*"]
    - from:
        - source:
            principals:
              - cluster.local/ns/production/sa/inventory-service
      to:
        - operation:
            methods: ["GET"]
            paths: ["/api/v1/orders/*"]

Deny policies explicitly block (e.g. block POST from legacy SA during deprecation). Default deny requires enabling authorization on mesh—document break-glass for debugging. Audit policies in GitOps PRs like application code—one overly broad ALLOW undoes zero trust.

Automatic distributed tracing — no code changes

Envoy generates spans for each request hop and propagates W3C traceparent headers automatically. Configure export via Istio Telemetry API to Jaeger, Tempo, Zipkin, or Datadog. Trace shows sidecar-to-sidecar latency breakdown—identify whether slowness is network, upstream app, or retry amplification.

Limitations: mesh spans are generic HTTP—no orderId unless app adds OpenTelemetry annotations or baggage. Double instrumentation risk: app and sidecar both create SERVER spans—configure telemetry to avoid duplicate trace trees. Full stack guidance: Observability → OpenTelemetry.

Traffic metrics — RED without Micrometer

For every meshed service, Prometheus scrapes Istio metrics:

• Rate — istio_requests_total (requests per second by source, destination, response_code)
• Errors — ratio of 5xx (and optionally 4xx) over total requests
• Duration — istio_request_duration_milliseconds histogram buckets for p50/p95/p99

Compare v1 vs v2 during canary by filtering on destination subset labels or workload version. Grafana dashboards per namespace are standard platform deliverables—see Observability → RED. App-level business metrics (orders placed, revenue) still require Micrometer in code—mesh does not know your domain.

sum(rate(istio_requests_total{
  destination_service_name="order-service",
  response_code=~"5.."
}[5m]))
/
sum(rate(istio_requests_total{
  destination_service_name="order-service"
}[5m]))

Service graph — nodes are workloads/services; edges show request rate, error percentage, and latency animation. During canary, visually confirm traffic split matches VirtualService weights— “is v2 actually getting 10%?”

mTLS indicators — lock icons on edges show encrypted traffic; broken lock highlights PERMISSIVE/plaintext paths still in use before STRICT cutover.

Config validation — flags broken VirtualService/DestinationRule references (subset not found, host typo)—catches GitOps mistakes before on-call pages.

Integrated views — jump to Grafana dashboards or Jaeger traces from a service node—reduces tab switching during incidents. Install Kiali as part of Istio addon profile or standalone; restrict access via OAuth2 proxy—graph exposes internal architecture.

• Fewer than ~10 services — Spring Cloud Gateway + Resilience4j + OTel agent may suffice
• No Kubernetes — VM estates: library stack or Consul Connect; full sidecars awkward
• No platform team — misconfigured VirtualService causes URX 503 mysteries; hire ops first
• Strict latency budget — measure sidecar hop cost; HFT paths may exclude mesh
• Serverless-heavy — short-lived functions poor fit for classic sidecar (ambient mesh evolving)