A retry repeats an operation after a failure that may be temporary. Network timeouts, rate limits, and temporary service unavailability can justify another attempt; an invalid request generally cannot. A retry policy limits attempts and delay so recovery does not turn into repeated pressure on a struggling service. section 3.4.1
A practical retry policy specifies:
- Which failures qualify: for example, HTTP 429, timeouts, and HTTP 503, rather than HTTP 400. section 3.4.1
- A retry limit and an initial delay, with optional exponential backoff and a maximum delay. Backoff can wait one, two, and then four seconds between successive retries. section 3.4.1
- What happens afterward: fail, let the caller handle the error, or try an ordered fallback provider. Some errors can be configured to fail without fallback. section 9.3.2 section 3.4.2
Listing 3.11 RetryConfig dataclass for transient failure handling
@dataclass
class RetryConfig:
max_retries: int = 3 #A
initial_delay: float = 1.0
exponential_backoff: bool = True #B
max_delay: float = 60.0
retry_on: List[FailureType] = None #C
RetryConfig groups the attempt limit, delay controls, and eligible failure types into a reusable model-service policy. section 3.4.1
Retry at the boundary where the failure occurs. A workflow SDK interceptor retries transient gRPC failures between a workflow container and platform services; the Model Service independently retries failures between itself and external providers. The application configures the policies while the platform executes them. section 9.3.2 section 3.4
Listing 9.13 gRPC retry interceptor for SDK service calls
import time
import grpc
RETRYABLE_CODES = {
grpc.StatusCode.UNAVAILABLE, #A
grpc.StatusCode.DEADLINE_EXCEEDED, #A
grpc.StatusCode.RESOURCE_EXHAUSTED, #A
}
class RetryInterceptor(
grpc.UnaryUnaryClientInterceptor
):
def __init__(self, max_retries=3,
initial_delay=0.1):
self._max_retries = max_retries #B
self._initial_delay = initial_delay
def intercept_unary_unary(
self, continuation, call_details,
request
):
last_error = None
for attempt in range( #C
self._max_retries + 1 #C
): #C
try:
return continuation( #D
call_details, request #D
) #D
except grpc.RpcError as e:
last_error = e
if (e.code() not in #E
RETRYABLE_CODES): #E
raise #E
if attempt == self._max_retries:
raise #F
delay = (self._initial_delay #G
* (2 ** attempt)) #G
time.sleep(delay) #G
raise last_errorThe gRPC interceptor retries UNAVAILABLE, DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED; other errors propagate immediately. section 9.3.2
For workflow-to-workflow HTTP calls, the retry loop treats server errors and connection failures as transient, but returns client errors without retrying. Its loop permits an initial request plus max_retries additional attempts, then raises the last error. section 9.4.3
Choose a policy for the workload
A patient-facing assistant can use short delays and a fallback chain to restore service quickly. A batch analytics pipeline can use longer backoff without fallback, leaving failed items for the pipeline to skip, queue, or alert on. A synchronous workflow with a tight timeout needs fewer, shorter retries than a long-running asynchronous workflow. section 3.4.3 section 9.1.1
Keep retries observable even when they hide failures from callers. Frequent fallback use can indicate a struggling primary provider; workflow metrics can reveal which steps fail and how long they take. Shallow readiness checks can leave transient dependency failures to per-request retries rather than removing healthy containers from service. section 3.4.2 section 2.1.7 section 9.3.3
























