liveBook · Manning

Circuit breaker pattern

As explained in gRPC Microservices in Go

A circuit breaker is a fault-tolerance pattern that temporarily stops calls to a failing dependency to prevent cascading failures.

A circuit breaker protects calls between microservices when a dependency is failing. Once failures reach a configured threshold, it opens the circuit and rejects further calls immediately. This avoids repeatedly waiting on an unhealthy service and helps prevent a failure from cascading through dependent services. section 6.1.3 section 6.1

A breaker moves through three states:

  • Closed: calls pass through, and their results contribute to the failure count. An interval can clear that count while the circuit remains closed. section 6.1.3
  • Open: calls fail immediately rather than reaching the dependency. A configured timeout determines when to test recovery. section 6.1.3
  • Half-open: a limited number of calls pass through. Successful calls close the circuit; continued failures reopen it. section 6.1.3

Protect an order-service call

In Go, configure a sony/gobreaker CircuitBreaker with a ReadyToTrip function that checks the failure ratio, then place the gRPC order client's Create call inside cb.Execute. Handle the returned error, including an open-circuit failure, rather than assuming an order was created. The example configuration trips at a failure ratio of 0.6; MaxRequests limits half-open probes, Timeout controls when probing begins, and OnStateChange can log transitions. Different dependencies can use differently configured breakers. section 6.1.3

Listing 6.6 Circuit breaker interceptor
package middleware
 
import (
    "context"
    "github.com/sony/gobreaker"
    "google.golang.org/grpc"
)
 
func CircuitBreakerClientInterceptor(cb *gobreaker.CircuitBreaker) 
 grpc.UnaryClientInterceptor {                                    #1
    return func(
    ctx context.Context, 
    method string, 
    req, reply interface{}, 
    cc *grpc.ClientConn, 
    invoker grpc.UnaryInvoker, 
    opts ...grpc.CallOption
    ) error {
        _, cbErr := cb.Execute(func() (interface{}, error) {        #2
            err := invoker(ctx, method, req, reply, cc, opts...)    #3
            if err != nil {
                return nil, err
            }
 
            return nil, nil
 
        })
        return cbErr
    }
}
...
// Interceptor usage on the client side
...
var opts []grpc.DialOption
    opts = append(opts, grpc.WithInsecure())
    opts = append(opts, grpc.WithUnaryInterceptor(middleware
     .CircuitBreakerClientInterceptor(cb)))                       #4
    conn, err := grpc.Dial("localhost:8080", opts...)
    if err != nil {
        log.Fatalf("Failed to connect order service. Err: %v", err)
    }
...

The client-side unary interceptor places the gRPC invoker inside cb.Execute and returns its error. This applies breaker behavior at the call boundary instead of wrapping each call in business logic. section 6.1.3

Choose which failures should influence the breaker using structured gRPC errors. Status codes can guide whether to open or close a circuit or retry; a validation error calls for correcting the request rather than blindly retrying it. Breakers complement retries and deadlines: retries may add load to an unavailable dependency, while a deadline limits how long a call blocks. section 6.2.2 section 6.4

How gRPC Microservices in Go covers it

How deep it goes
Explains state transitions and configuration, then implements a Go breaker around a gRPC call and in a client interceptor.
Examples
Uses an order-creation gRPC call, a failure-ratio setting, state-change logging, and a unary client interceptor.
What it assumes
Assumes familiarity with Go service calls and gRPC clients while defining breaker states and their settings.
What you can do after
Apply it via sony/gobreaker to protect Go gRPC client calls and configure when failures open or reset the circuit.
How it connects
Connects circuit breakers to retries, timeouts, deadlines, cascading failures, gRPC status codes, and client interceptors.

Also explained in

Compare how 14 books explain circuit breaker pattern →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of gRPC Microservices in Go. The section links point to the parts of the book it draws on.