Foresight AI > Reliability Intelligence: Introduce circuit breakers around dependencies

Reliability Intelligence: Introduce circuit breakers around dependencies

At its core, a circuit breaker provides a way to skip calling a protected set of code, like a remote dependency call, after a condition has been met. Common conditions include exceeding a number of failures within a current interval.

For a simple example (Java), see Putting it all together. For in-depth details of circuit breakers at larger scales, see Fault Tolerance in a High Volume, Distributed System.

‍

Step-by-step with Failsafe

The following example uses the Java library Failsafe, which contains primitives like the CircuitBreaker interface for easy implementation and lots of flexibility. Similar libraries exist for other languages (Failsafe-go, circuitbreaker).

‍

Step 1: Identify a dependency call to protect

Java


 /// Simulate a call to an external service
  public String fetchData() throws InterruptedException {
    Thread.sleep(10_000); // simulate a slow dependency
    return "fetched-value";
  }

‍

Step 2: Define the CircuitBreaker

Java


static final CircuitBreaker<String> CIRCUIT_BREAKER =
  CircuitBreaker.<String>builder()
  .withFailureRateThreshold(
      CIRCUIT_BREAKER_FAILURE_RATE,
      CIRCUIT_BREAKER_FAILIRE_MIN,
      CIRCUIT_BREAKER_FAILURE_RATE_PERIOD)
  .build();

‍

Step 3: Wrap calls to protected dependency in a Failsafe

Java


static void tryFetchData(FailsafeExecutor<String> failsafe, CircuitBreakerClient client)
      throws InterruptedException {
    long startTime = System.nanoTime();
    try {
      String result = failsafe.get(() -> client.fetchData());
      long endTime = System.nanoTime();
      long durationMs = (endTime - startTime) / 1_000_000;
      System.out.printf("success: %s (%dms)\n", result, durationMs);
    } catch (RuntimeException e) {
      long endTime = System.nanoTime();
      long durationMs = (endTime - startTime) / 1_000_000;
      System.err.printf("failed: %s (%dms)\n", e, durationMs);
    }
  }
}

‍

Step 4: Putting it all together

Java


import dev.failsafe.CircuitBreaker;
import dev.failsafe.Failsafe;
import dev.failsafe.FailsafeExecutor;
import dev.failsafe.Timeout;
import java.time.Duration;

public class CircuitBreakerClient {
  /// Often, your dependency client can be configured with a timeout instead
  static final Timeout<String> TIMEOUT =
      Timeout.<String>builder(Duration.ofSeconds(1)).withInterrupt().build();
  /// The rate of failure that must be reached before the circuit breaker closes
  static final int CIRCUIT_BREAKER_FAILURE_RATE = 50;
  /// The minimum number of failures that need to be seen before the circuit breaker closes
  static final int CIRCUIT_BREAKER_FAILIRE_MIN = 2;
  /// The period of time by which `CIRCUIT_BREAKER_FAILURE_RATE` is evaluated
  static final Duration CIRCUIT_BREAKER_FAILURE_RATE_PERIOD = Duration.ofSeconds(30);
  static final CircuitBreaker<String> CIRCUIT_BREAKER =
      CircuitBreaker.<String>builder()
          .withFailureRateThreshold(
              CIRCUIT_BREAKER_FAILURE_RATE,
              CIRCUIT_BREAKER_FAILIRE_MIN,
              CIRCUIT_BREAKER_FAILURE_RATE_PERIOD)
          .build();

  /// Simulate a call to an external service
  public String fetchData() throws InterruptedException {
    Thread.sleep(10_000); // simulate a slow dependency
    return "fetched-value";
  }

  public static void main(String[] args) throws InterruptedException {
    CircuitBreakerClient client = new CircuitBreakerClient();
    FailsafeExecutor<String> failsafe = Failsafe.with(CIRCUIT_BREAKER, TIMEOUT);
    tryFetchData(failsafe, client); // failed: dev.failsafe.TimeoutExceededException (1011ms)
    tryFetchData(failsafe, client); // failed: dev.failsafe.TimeoutExceededException (1012ms)
    tryFetchData(failsafe, client); // failed: dev.failsafe.CircuitBreakerOpenException (1ms)
    tryFetchData(failsafe, client); // failed: dev.failsafe.CircuitBreakerOpenException (0ms)
  }

  static void tryFetchData(FailsafeExecutor<String> failsafe, CircuitBreakerClient client)
      throws InterruptedException {
    long startTime = System.nanoTime();
    try {
      String result = failsafe.get(() -> client.fetchData());
      long endTime = System.nanoTime();
      long durationMs = (endTime - startTime) / 1_000_000;
      System.out.printf("success: %s (%dms)\n", result, durationMs);
    } catch (RuntimeException e) {
      long endTime = System.nanoTime();
      long durationMs = (endTime - startTime) / 1_000_000;
      System.err.printf("failed: %s (%dms)\n", e, durationMs);
    }
  }
}

‍

(Optional) Step 5: Introduce a Fallback

A Failsafe can also be configured with a fallback, which provides an alternate way to serve the requested data. This helps avoid errors when calls fail or the CircuitBreaker opens. See Implement a fallback around network dependency to avoid errors when the dependency is unreachable.