Resilience
Failure is always an option.
Possible tools: Tools
- Caching, Fallback, Retry, Circuit Breaker.
Timeout
Every remote call needs a deadline. Propagate cancellation and keep all attempts within the caller's total time budget.
Retry
Retry only transient failures and only when the operation is idempotent or otherwise repeatable.
- Limit attempts.
- Use exponential backoff with jitter.
- Respect
Retry-After. - Retry at one layer only. Three layers with three attempts can create up to 27 downstream calls.
Circuit Breaker
If calls to a dependency fail repeatedly, stop calling it for a short time and fail fast instead. This avoids waiting for timeouts and gives the dependency time to recover.
- Closed => Calls go through normally.
- Open => Calls are blocked and fail immediately.
- Half-open => Allow a test call. Close the circuit after success; open it again after failure.
- Complements retries; it is not a replacement for timeouts.
- Returning stale or degraded data must be a deliberate business decision.
Bulkhead
Split limited resources into isolated pools. If one dependency or operation exhausts its pool, the rest of the application can still work.
Example: Limit calls to a slow service to 10 concurrent requests instead of allowing it to consume every worker or connection.