Operations · Guide
Give retries a budget, not an endless loop
Bound retry time, request count, and business impact when providers slow down.
Repeated effort can amplify failure
In an illustrative document workflow, the browser retries, the API retries, and the worker retries the same model call. Each layer seems reasonable in isolation, but together they multiply requests during an outage. Map where retries already occur, including SDK defaults. Choose a responsible layer for each remote operation and document the maximum total attempts rather than adding another loop whenever a timeout appears.
Budget three dimensions
Our proposed budget contains elapsed time, attempt count, and side-effect exposure. A low-risk read can tolerate a different policy from an uncertain external write. AWS Builders Library explains backoff and jitter; apply those mechanisms within the business deadline. Do not keep retrying a report after the meeting it was meant to support. A final state should tell the owner whether the job is stopped, awaiting reconciliation, or safe to restart.
Workflow at a glance
Classify errors before retrying
Separate malformed input, denied access, quota responses, connection failures, and an unknown write result. Return validation problems to the person who can correct them; do not consume provider calls repeatedly. For an unknown write, reconcile the destination before repeating. Build a short error-to-action table and verify its assumptions against each provider’s current API behavior. A generic catch-all retry is easy to write and difficult to operate.
Prove that the system stops
Test a prolonged outage and assert the request eventually reaches a bounded terminal or review state. Measure total provider attempts per business request, not merely worker failures. Add a recovery test to ensure delayed jobs do not flood a provider at once. The right budget depends on urgency and destination semantics; it is a business decision informed by engineering evidence, not a universal number copied from a tutorial.
Account for total retry cost
Include provider calls, queue waiting, and manual reconciliation in the cost report. Eventual success may still be uneconomical. Compare repeated automation with a manual fallback for exceptions, and prioritize work that remains useful before allowing expired jobs to occupy all available capacity.
Sources and further reading
- Timeouts, retries, and backoff with jitterTimeouts, retries, and backoff with jitterAWS · Documentation ·
- Handling errors in Step Functions workflowsHandling errors in Step Functions workflowsAWS · Documentation ·
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.