Retry repeats a failed operation when the failure may be temporary. In a microservice system, a consumer can retry a gRPC request when a dependent service is unavailable, giving communication another chance to succeed. The decision must depend on the response and the operation, not merely on the fact that a call failed. section 6.4 section 5.0
First check whether repeating the operation is safe. An idempotent operation has no additional effect when called more than once. For example, if a response to a user deletion is lost in a network failure, repeating an idempotent deletion does not change the resource again. Where idempotency is unsuitable, return validation errors that tell the client when to stop retrying. section 1.1.3
Set the retry policy around the failure and the time the caller can afford to wait:
- Retry selectively. gRPC status codes such as Unavailable or ResourceExhausted can justify another attempt; a validation error caused by an invalid payload calls for correction instead. section 6.4 section 6.2.2
- Limit waiting with a context timeout or deadline. Retry blocks the gRPC call while the client waits for its dependency, and recovery time may be hard to predict. section 6.4
- Consider a circuit breaker when repeated attempts burden an unready service. After a failure limit, it can stop calls temporarily and try again later. section 6.4
Order and Shipping service failure
If the Shipping service cannot reach its database, shipping calls from the Order service can fail and potentially contribute to a cascading failure. Time out the shipping operation or fail fast and retry later, rather than leaving dependent calls waiting indefinitely and impairing unrelated requests. section 6.1
ctx, cancel := context.WithTimeout(context.Background(),
10 * time.Second)
defer cancel() #1
payment := pb.NewPaymentClient(conn)
result, err := payment.Create(ctx, &CreatePaymentRequest{}) #2
if err != nil {
// handle Payment create an error
}
A Go gRPC Payment client passes a context with a ten-second timeout to Create. The listing shows a time limit for a call, not a complete retry loop or retry policy. section 2.5.3
Structured errors make the choice actionable: a client can inspect gRPC status and error details to distinguish a condition worth retrying from a request that needs new input. Retry works alongside timeouts and circuit breakers; it does not replace them. section 6.2 section 6.4
