Microservices Security

Attack surface grows with every new service: more JWTs, more service accounts, more ConfigMaps someone mistyped, more admin Actuator endpoints forgotten on port 8080. Perimeter firewalls that treat the VPC as trusted fail when an attacker lands in any pod via supply-chain CVE or stolen CI credential—they pivot to internal APIs with no user context.

Security splits across platform (mesh mTLS, NetworkPolicy, secret stores, WAF) and application (resource-level authorization, input validation, secure domain logic). Failures happen at handoffs: “gateway validated JWT, so service skipped checks” or “mesh encrypts traffic, so we skipped BOLA tests.” Both layers are mandatory; neither replaces the other.

Never trust, always verify — including internal services

Legacy model: firewall around the datacenter; anything inside may call anything. Zero trust assumes the network is hostile: compromised marketing pod, malicious insider, misconfigured security group. Every request—browser, batch job, or service-to-service—must authenticate and be authorized against policy.

For humans: OIDC login (Okta, Auth0, Keycloak) → access token with subject and scopes. For workloads: mTLS certificates (SPIFFE IDs from Istio), OAuth2 client credentials, or signed service tokens— not “it came from 10.0.4.12 so it must be Inventory.”

Identity for services — not just users

Microservices multiply machine identities: Order Service SA, Payment SA, nightly batch CronJob SA. Each should have least privilege: Payment may call Ledger; Catalog may not call Admin API. Kubernetes service accounts + Istio AuthorizationPolicy or NetworkPolicy enforce which identity may reach which port/path. Compromised checkout pod must not reach payroll—even if both are “internal.”

• Verify explicitly — no unauthenticated /internal/* shortcuts
• Least privilege — narrow scopes, narrow network policy, narrow DB grants per service
• Assume breach — encrypt in transit and at rest; segment namespaces; ship audit logs to SIEM
• Continuous validation — short token TTL, cert rotation, revoke on incident

Mesh implements zero trust at L4/L7: Service Mesh → mTLS, AuthorizationPolicy. Application JWT validation remains mandatory for user context on every service that serves user data.

Perimeter-only security fails when: CI pipeline leaks prod credentials, developer laptop has VPN access, partner API key grants too much, or SSRF from webhook handler hits internal metadata. Defense in depth adds independent barriers—breaching one layer should not unlock everything.

flowchart TB
  U[User] --> WAF[WAF and CDN]
  WAF --> GW[API Gateway JWT]
  GW --> M[Mesh mTLS and AuthZPolicy]
  M --> S[Service validation and BOLA]
  S --> DB[(Encrypted DB least privilege)]

Access tokens are JWTs (signed, verify via JWKS) or opaque (introspection endpoint). Spring Security OAuth2 Resource Server validates signature, iss, aud, exp, clock skew—then maps claims to authorities.

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: https://auth.example.com/realms/production
          audiences:
            - order-api
            - order-api-internal

Gateway validation responsibilities

• Verify signature against IdP JWKS (cache keys with TTL, handle key rotation)
• Validate iss, aud, exp, nbf
• Reject alg: none and unexpected algorithms
• Enforce gateway-level scopes for route (e.g. orders:read for GET /orders)
• Strip dangerous headers from client; optionally inject trusted internal headers

Downstream options

Forward Bearer token — simplest; each service validates JWT independently (defense in depth). Signed internal headers — gateway injects X-Internal-Identity HMAC-signed with key only gateway holds; services trust only if request arrives via mTLS from gateway SA. Never trust client-supplied X-User-Id without cryptographic binding.

sequenceDiagram
  participant User
  participant GW as API Gateway
  participant Svc as Order Service

  User->>GW: Bearer JWT
  GW->>GW: Validate sig iss aud exp
  GW->>Svc: Forward JWT or signed claims
  Svc->>Svc: Re-validate or trust bound identity
  Svc->>Svc: AuthZ on resource ID

Without relay, Payment sees only Order’s client-credentials token—all users look the same; BOLA checks impossible. With relay, Payment validates same JWT, reads sub, verifies user owns the order being refunded. Spring Cloud Gateway and OpenFeign OAuth2 token relay attach incoming bearer to outbound WebClient/Feign calls automatically when configured.

sequenceDiagram
  participant User
  participant GW as API Gateway
  participant Ord as Order Service
  participant Pay as Payment Service

  User->>GW: Bearer user JWT
  GW->>Ord: forward JWT
  Ord->>Pay: relay same JWT
  Pay->>Pay: validate JWT + user owns order

Trade-offs and alternatives

• Token expiry — long sagas may outlive 5-minute access token; use refresh at gateway or token exchange
• Token exchange (RFC 8693) — Order exchanges user token for Payment-scoped short-lived token
• Signed delegation claim — Order includes HMAC-signed onBehalfOf claim trusted by Payment
• Dual auth — client credentials for service + separate user claim header bound to mTLS

Payment must validate both: caller service identity (mesh mTLS / client creds) and user identity (JWT subject) for sensitive operations.

OAuth2 client credentials flow

Each microservice registers as confidential OAuth client with authorization server (Keycloak, Auth0, custom AS). At startup or per-request, service POSTs to token endpoint with grant_type=client_credentials, client_id, client_secret (or JWT client assertion—preferred over long-lived shared secret). Returned access token carries scopes like inventory.read—no sub for a human user.

Store client secrets in Vault; rotate via dynamic secret or automated rotation job. Cache token in memory until exp; refresh proactively before expiry. Never embed client_secret in Docker image or git—scan with gitleaks in CI.

Mutual TLS (mTLS)

Client and server present X.509 certificates; identity encoded in SPIFFE URI SAN. Istio automates cert issuance and rotation via Istiod CA—see PeerAuthentication. Without mesh: cert-manager issues short-lived certs mounted into pod; Java SSLContext configured for outbound calls. mTLS encrypts and authenticates transport—it does not replace application authZ: any compromised service with valid cert still connects unless AuthorizationPolicy denies.

@Bean
OAuth2AuthorizedClientManager clientCredentialsManager(
    ClientRegistrationRepository registrations,
    OAuth2AuthorizedClientService clientService) {
  var provider = OAuth2AuthorizedClientProviderBuilder.builder()
      .clientCredentials()
      .build();
  var manager = new AuthorizedClientServiceOAuth2AuthorizedClientManager(
      registrations, clientService);
  manager.setAuthorizedClientProvider(provider);
  return manager;
}

@Bean
WebClient inventoryWebClient(OAuth2AuthorizedClientManager manager) {
  var oauth = new ServletOAuth2AuthorizedClientExchangeFilterFunction(manager);
  oauth.setDefaultClientRegistrationId("inventory-client");
  return WebClient.builder().apply(oauth.oauth2Configuration()).build();
}

Gateway enforces coarse scopes: POST /orders requires orders:write. Service enforces resource rules: even with orders:write, user cannot POST body with someone else’s customerId (BOLA). BFF may use roles to show/hide UI; never rely on hidden buttons alone—API must enforce.

@PreAuthorize("hasAuthority('SCOPE_orders:write')")
@PostMapping("/orders")
public OrderDto create(@Valid @RequestBody CreateOrderRequest req,
                       @AuthenticationPrincipal Jwt jwt) {
  if (!req.customerId().equals(jwt.getSubject())) {
    throw new AccessDeniedException("Cannot create order for another customer");
  }
  return orderService.create(req);
}

Rules that survive audits

• No secrets in git — ever; use gitleaks/trufflehog in pre-commit and CI; block merge on findings
• No secrets in images — build args and ENV persist in layer history; inject at runtime from vault
• No shared superuser DB password — one DB role per service with minimal GRANTs
• Rotation — automated where possible; runbooks for manual emergency rotation
• Audit — who read which secret when; alert on anomalous access

Secret categories in microservices

Dynamic secrets

Instead of static password in config, app requests DB creds from Vault database secrets engine. Vault creates PostgreSQL user v-token-order-a1b2c3 valid 1 hour with GRANT on order_schema only. App connects; on lease expiry Vault revokes user—stolen credential window bounded even if log file leaked.

Secret leasing and renewal

Every Vault response includes lease TTL. Apps renew leases before expiry for long-running processes; if renewal fails, reconnect with fresh creds. Short TTL limits blast radius; ops monitors lease expiration metrics. Static KV secrets also versioned—rotation creates v2; apps reload via Spring Cloud refresh or sidecar.

AppRole authentication for machines

Services authenticate to Vault without human login:

1. Platform team creates AppRole with policy allowing read secret/data/order-service/*
2. Deploy delivers role_id (less sensitive) and secret_id (via K8s init container or CSI) to pod
3. App exchanges for Vault token; fetches secrets; renews token lease

path "database/creds/order-service-role" {
  capabilities = ["read"]
}

path "secret/data/order-service/production/*" {
  capabilities = ["read"]
}

Production: HA Raft storage, auto-unseal with cloud KMS, namespaces per environment, break-glass procedures documented. Never run vault server -dev in prod. Pair with NetworkPolicy—Vault token useless from wrong namespace without network path to DB anyway.

Encryption at rest

Enable Kubernetes EncryptionConfiguration with KMS provider (AWS KMS, GCP KMS, Azure Key Vault key) so etcd stores encrypted Secret objects. Without this, anyone with etcd snapshot or backup access reads plaintext equivalents. Cloud-managed K8s (EKS, GKE, AKS) offers one-click or documented enablement—verify it is on for prod clusters.

RBAC on Secrets

Restrict who can get/list/watch Secrets in namespace: only CI deploy SA and namespace admin—not every developer’s user account. Audit RBAC bindings quarterly; remove overly broad cluster-admin grants. Prefer External Secrets Operator syncing from Vault—humans never kubectl apply raw Secret YAML with passwords.

Mounting best practices

• Mount as volumes (tmpfs) with defaultMode: 0400—not env vars visible in /proc and crash dumps
• Separate Secret per concern—do not one mega-secret with all keys
• Rotate triggers rolling restart via Reloader or staged deploy when static secret version changes

# bootstrap.yml — loads before main context
spring:
  application:
    name: order-service
  cloud:
    vault:
      uri: https://vault.example.com
      authentication: KUBERNETES
      kubernetes:
        role: order-service
        service-account-token-file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kv:
        enabled: true
        backend: secret
        default-context: order-service/production

---
# application.yml — references injected properties
spring:
  datasource:
    url: ${db.url}
    username: ${db.username}
    password: ${db.password}

Add spring-cloud-starter-vault-config. Kubernetes auth: pod SA token proves identity to Vault; Vault role maps to policy. @RefreshScope beans reload when secrets rotate and actuator /refresh or Spring Cloud Bus event fires— prefer connection pool recycle on DB password change.

EKS pods authenticate via IRSA (IAM Roles for Service Accounts)—no long-lived AWS keys in cluster. Pod SA annotated with role ARN reads only arn:aws:secretsmanager:...:order-service/*. Azure uses workload identity federation with federated credential on Key Vault access policy or RBAC role Key Vault Secrets User.

spring:
  config:
    import: aws-secretsmanager:order-service/production/db
  cloud:
    aws:
      region:
        static: eu-west-1

External Secrets Operator can sync AWS/Azure secrets into K8s Secret objects for non-Spring workloads—single source of truth in cloud store, uniform injection pattern across polyglot services.

In microservices, BOLA appears on internal APIs too—attacker with mesh access hits Order service directly. Every service validates auth regardless of “only called from gateway” assumptions—enforce with AuthorizationPolicy + app checks.

Input validation at gateway AND service level

Gateway layer (Spring Cloud Gateway filters, Kong plugins, Envoy WASM):

• Max request body size (reject 50 MB JSON bombs at edge)
• JSON Schema validation for known public routes
• Block obvious SQLi/XSS patterns in query strings for legacy endpoints
• Required headers: Authorization, Content-Type: application/json
• Reject requests missing correlation ID or with invalid format

Service layer (non-negotiable even if gateway validates):

• Jakarta Bean Validation on DTOs: @Valid, @NotNull, @Size, custom constraints
• Whitelist enums—reject unknown status values
• @JsonIgnoreProperties(ignoreUnknown = false) on sensitive write DTOs—prevent mass assignment
• Sanitize file uploads; virus scan; store outside web root
• Validate path UUIDs format before DB lookup—reduce error oracle leakage

public record UpdateOrderRequest(
    @NotBlank @Size(max = 500) String deliveryNote
) {}

@PatchMapping("/orders/{id}")
public OrderDto update(@PathVariable UUID id,
                       @Valid @RequestBody UpdateOrderRequest req,
                       @AuthenticationPrincipal Jwt jwt) {
  Order order = orders.findOwnedBy(id, jwt.getSubject())
      .orElseThrow(() -> new NotFoundException(id));
  return orders.updateNote(order, req.deliveryNote());
}

Rate limiting as security control — not just traffic management

Rate limits protect against abuse, not only overload:

• Credential stuffing — limit POST /auth/login per IP and per username
• User enumeration — limit GET /users?email= attempts
• Partner API key abuse — per-key quota with 429 and alert
• Expensive endpoints — stricter limits on search/report APIs (API4)
• Internal APIs — limit even “trusted” callers to detect compromised SA

Pair rate limits with account lockout and SIEM alerts on threshold breach—not silent drop only. Algorithms and gateway config: Resilience → Rate limiting. Return 429 Too Many Requests with Retry-After header; log security event with client IP and user id hash.

Additional hardening checklist

• CORS: explicit origins—never * with credentials
• Security headers: HSTS, CSP on BFF HTML responses
• Actuator on management port 8081, network-restricted, separate auth
• Dependency scanning in CI (Snyk, OWASP Dependency-Check)
• Disable XML external entities (XXE) if legacy SOAP

• Zero trust: mTLS STRICT or AuthorizationPolicy on east-west paths
• OAuth2/OIDC: correct issuer, audience, short access token TTL
• JWT validated at gateway and re-validated or cryptographically bound at service
• Service-to-service: client credentials or mTLS; no IP-based trust
• Secrets from Vault / cloud SM / ESO—none in git or image layers
• K8s Secrets encrypted at rest; RBAC least privilege
• BOLA test on every /{id} route; mass assignment tests on POST/PATCH
• Input validation at gateway and service; max body size enforced
• Rate limits on auth and sensitive flows; anomaly alerts
• Audit logs for auth failures and admin actions → SIEM
• Dependency and container image scanning in CI
• Incident runbook: token revocation, key rotation, compromised SA