Redis is unavailable — fail open or fail closed?

After reading, you should be able to:

• Classify Redis usage: optional cache vs required coordination.
• Choose fail-open vs fail-closed per use case with a decision table.
• Implement timeouts, circuit breakers, and DB fallback without stampeding the database.
• Align health checks and runbooks with business risk.

Why — Redis is not always “just cache”

Fail open (degraded mode): on Redis error, skip cache and continue—usually hit the database or a default. Service stays available; load and latency may spike. Fail closed: on Redis error, reject or error the operation—used when correctness or security requires Redis (session store, rate limit, distributed lock, idempotency token).

Use cases and typical stance

Redis role	Fail open?	Risk if wrong
Read-through cache	Often yes (fallback DB)	DB overload — stampede, pool exhausted
Session store	Usually no	Users logged out or cross-session bleed if misdesigned
Rate limiting	Policy: no (strict) or yes (availability)	Abuse vs outage
Distributed lock	No	Double payment, duplicate jobs
Idempotency keys	No	Duplicate charges on retry
Pub/sub invalidation	Yes with stale TTL backstop	Temporary staleness
Feature flags / config cache	Often yes (defaults or DB)	Wrong flag rare if versioned in DB

What — assess impact and symptoms

1. Confirm Redis is the failure point — app errors (RedisConnectionException, timeouts); Redis PING fails from pod; cloud console shows failover/maintenance.
2. Inventory call sites — grep codebase for Redis template, Redisson, Spring Data Redis; tag each: cache / session / lock / limiter.
3. Metrics — command error rate, latency p99, connection count; correlate with API errors and DB QPS spike.
4. User-visible impact — 500 on all routes vs only slow reads vs auth failures.
5. Readiness probe behavior — are pods removed from LB because Redis is in readiness? intentional or not?

How — implement resilient behavior

1. Short timeouts and circuit breaker

# Lettuce / Spring Boot — fail fast, don't block threads
spring.data.redis.timeout=200ms
# Resilience4j: open circuit after Redis error rate threshold

Threads blocked on Redis tie up HTTP workers—same as slow DB.

2. Cache path: fail open with guardrails

Optional<User> cached = redisGet(key);
if (cached.isPresent()) return cached.get();
try {
  return redisGetOrLoad(key, () -> userRepo.findById(id));
} catch (RedisException e) {
  meter.increment("redis.fallback.db");
  return userRepo.findById(id);  // direct DB, one request
}

• Do not retry Redis indefinitely per request.
• When circuit open, skip Redis entirely for cooldown window.
• Cap DB QPS: rate limit, cache locally (short Caffeine L1), or shed load — backpressure.

3. Session / auth: usually fail closed

Return 503 “try again” or maintain sticky sessions to another region—do not invent anonymous sessions. Optional: read-only mode for public catalog only.

4. Rate limit: explicit product decision

Policy	When
Fail closed	Fraud-sensitive APIs, login, payments
Fail open	Internal tools, read-mostly during known Redis maintenance (with approval)

5. Locks and idempotency: fail closed

If lock cannot be acquired reliably, do not process payment—return retryable error. Never run two writers because Redis lock disappeared.

6. Health checks

• Liveness — JVM up; do not include Redis.
• Readiness — include Redis only if pod cannot serve traffic without it (sessions). For cache-only Redis, stay ready and degrade.

7. Infrastructure

• Redis Sentinel or cluster with automatic failover; multi-AZ.
• Connection pooling (Lettuce), reasonable max connections per pod.
• Chaos test: block Redis port in staging, observe DB and error budget.

Runbook (incident)

1. Confirm scope: all apps or one namespace.
2. Enable degraded mode flag if you have one (force DB-only with rate limit).
3. Scale DB/read replicas if fail-open is active.
4. Restore Redis; watch stampede on recovery as cache cold—prewarm hot keys.

Interview one-liner

Related scenarios

• Stale cache
• Cache stampede
• Circuit breaker
• 502 but pods healthy