Implementation
Retries Are Not Recovery: Designing Reconciliation for AI Workflow Side Effects
A timed-out request may already have changed an external system. Reliable recovery starts by verifying the business outcome, not merely repeating the call.
A renewal message timed out, but it may already have been sent
Imagine an AI-enabled workflow that prepares a customer renewal message, checks the account details and submits the finished email to a delivery provider. The provider does not respond before the request times out.
The workflow now has two dangerous shortcuts available. It can send the message again and risk contacting the customer twice. Or it can mark the run as failed and stop, potentially leaving the renewal untreated even though no one has established whether the first message was accepted.
The useful answer is neither “retry everything” nor “give up.” Preserve the customer case, mark the business outcome as unknown, suppress a blind second send and inspect whatever documented provider or business records are available. If those records cannot establish the outcome, assign the case for reconciliation rather than allowing it to disappear inside a failed workflow run.
This hypothetical renewal case illustrates a general rule for payments, record updates, bookings, notifications and other external effects: a technical response is not the same thing as a verified business outcome.
Start with the actions and records you can inspect
Before selecting recovery controls, list every workflow action that can change something outside the workflow itself. Typical categories include:
sending a customer communication;
creating or changing a payment-related object;
updating a customer, order or inventory record;
submitting a booking or cancellation;
triggering another system that performs one of these actions.
For each action, identify the component that actually applies the change. Then consult that target's current documentation, operation records or support channel to answer three practical questions:
Can repeated requests for the same intent be made idempotent?
Can the result be found later by an operation ID, correlation value or business identifier?
Which record is authoritative enough to decide whether another attempt is safe?
Named-provider behavior should come from current authoritative documentation rather than assumptions carried over from another integration. Someone must also own cases that remain unresolved after the automated checks finish. Without an accountable owner, a reconciliation queue is only another place for work to stop.
A retry records another attempt, not a recovered outcome
A timeout tells the caller that it stopped waiting. It does not establish whether the remote service performed the requested side effect. AWS guidance explicitly warns that a timeout or failure does not necessarily mean a side effect did not happen, and that retrying side-effecting calls requires particular care (AWS Builder Center).
Represent the workflow with four separate kinds of information:
Business intent: the single outcome the business wants, such as sending one renewal notice for account A and renewal cycle B.
Attempt: each submission made in pursuit of that intent, with its own timestamp, destination and attempt status.
Provider response: the response returned for an attempt, or an explicit record that no conclusive response arrived.
Verified outcome: the best supported determination of what happened in the business system, recorded separately as confirmed, not completed or unknown.
In the renewal example, the intent remains “send one renewal notice.” Attempt one timed out. Its provider response is absent or inconclusive. The verified outcome remains unknown until another source establishes it.
A second request would create attempt two. Even if it receives a success response, that response does not prove attempt one had no effect. Recovery is complete only when the business outcome has been established or the unresolved case has reached an accountable decision-maker.
Classify the effect before choosing a recovery path
Choose controls per external action, not once for the whole workflow. Classify the action along four dimensions:
Outcome certainty: Can the target conclusively report whether it applied the effect?
Repeatability: Can the same business intent be submitted again without applying the effect twice?
Reversibility: Is there a defined corrective action, and does it create an acceptable business state?
Consequence: What happens if the action is duplicated, omitted, delayed or corrected incorrectly?
Use the classification to choose the path:
Immediate retry fits a known-safe, transient failure where no harmful duplicate effect can occur.
Retry with the same idempotency key fits an operation where the receiver enforces idempotency for that request and the key still falls within its documented rules.
Verify before retry fits a timeout or other ambiguous response after the target may have committed the action.
Compensate when an effect is confirmed and a permitted corrective action can move the process into an acceptable state.
Reconcile manually when available evidence cannot establish the outcome or the appropriate response requires judgment.
Require human approval when the final action is consequential, poorly observable or difficult to correct.
These controls have operating costs. They require identifiers, retained records, access to target-side evidence, exception ownership and sometimes slower completion. That cost should be compared with the less visible cost of duplicated customer contact, conflicting records or abandoned cases. Related failure boundaries are explored in Where AI implementation usually breaks.
Put idempotency where the external effect is committed
The relevant idempotency boundary is the component that commits the externally visible effect. Preventing the workflow runner from processing one internal job twice is useful, but it does not by itself stop a downstream target from accepting two separate requests.
An idempotency key should represent one stable business intent. Every retry of that same intent uses the same key. A genuinely different intended action uses a different key. The key should not change merely because the workflow restarted, an AI step regenerated equivalent content or another worker picked up the job.
Stripe provides a bounded example of receiver-enforced idempotency. Its documentation explains that clients can attach an idempotency key to supported requests and reuse that key after a connection error. It also documents API-specific behavior, parameter checks and key-retention limits (Stripe documentation). Those details demonstrate why “the provider supports idempotency” is not enough. Operators must check which operation accepts the key, how repeated parameters are treated and how long the protection remains available.
Suppose the renewal workflow deduplicates executions using an internal run ID, but each run sends an ordinary request to the email provider. A restarted run may have a new run ID and issue another send. The upstream deduplication has protected the workflow's own record, not the customer-facing effect. Duplicate prevention must reach the service that decides whether to apply that effect.
When the target cannot guarantee an idempotent action
Some targets do not expose a duplicate-safe operation for the action the workflow needs. In that case, attach stable correlation data where the documented interface permits it, then use that data to connect the business intent with target-side operation logs, message records or business records.
Verification must rely on a documented status facility or an appropriate system of record. A local timeout, a missing workflow response or the absence of an immediate callback is not enough to conclude that the target did nothing.
If the available records remain inconclusive, keep the case in reconciliation. Do not convert “unknown” into “failed” merely to let the automated path continue.
The simpler alternative is to avoid automating the commitment. The workflow can prepare a renewal draft and present the relevant account information while an authorized person decides whether to send. This can be the better design when the effect is consequential, the target is poorly observable or duplicate detection depends on subjective review.
Automate with verification and reconciliation
This route fits lower-consequence actions when dependable target-side records can establish outcomes and an owned operating queue can resolve the remaining ambiguity.
Keep the final action human-committed
This route fits high-consequence or poorly observable actions. Automation can prepare a draft, but an authorized person commits the external effect.
Handle the uncertain renewal message in this order
After the hypothetical provider timeout, follow a sequence that preserves both the customer case and the evidence:
Preserve the renewal case and its stable intent identifier. Do not create a replacement intent simply because the first workflow execution stopped.
Record the first attempt as timed out with an unknown outcome. Keep the response state distinct from the business result.
Suppress an automatic second send. Hold further delivery attempts while the outcome may already exist.
Check documented provider and business records. Use the available operation ID, correlation value, account identifier or renewal reference according to the target's supported lookup process.
Record one determination: confirmed sent, confirmed not sent or still unknown. Include which evidence supported the determination.
Apply the chosen retry rule only when safe. If the message is confirmed not sent, retry according to the target's documented behavior. If it is confirmed sent, close the sending action without another delivery attempt.
Assign an inconclusive case to a named owner. Include an escalation route and a decision deadline so the renewal cannot remain unknown indefinitely.
Record the final disposition. Capture the verified result or authorized business decision rather than treating technical workflow completion as proof of delivery.
This sequence is deliberately conservative at the point where a customer could observe the effect. Monitoring should make the held case visible rather than allowing the suppression of a duplicate to become silent abandonment. See Monitoring the AI operating layer for the broader monitoring context.
Compensation changes the situation; it does not erase history
Compensation is a new corrective action, not a guarantee that history has been reversed. Microsoft’s compensating transaction guidance notes that corrective work may need business-specific rules and may not return data to its exact starting state, particularly when concurrent activity has changed the situation (Microsoft Learn).
A workflow can automate compensation only when the corrective action is:
well defined for the confirmed state;
permitted for the service or person performing it;
safe to repeat if its own response becomes uncertain;
observable through a reliable completion record.
Ambiguous, high-impact or externally observed effects should receive human review. A sent renewal message cannot be made unsent. Depending on the business context, the suitable response might be to leave the original communication in place, send an authorized clarification or contact the customer through an established process. That decision should not be improvised by a generic retry handler.
Compensating actions can also time out or fail. Record their progress separately so they can resume safely or escalate without losing the history of the original effect. Microsoft’s guidance likewise recommends tracking compensation progress and allowing manual intervention when automated recovery cannot complete.
Make reconciliation an owned operating queue
Reconciliation is the operating path for effects that automation cannot determine safely. Each queued case needs enough information for someone to reach and record a decision:
the preserved case and correlation identifiers;
the current certainty state;
the evidence already checked;
the accountable owner;
the permitted next actions;
the approval threshold;
the due time and escalation route;
the final disposition when resolved.
Monitor the queue for aging unknown cases and failed compensations. Restrict corrective actions to people or services authorized to perform them. An investigator may be allowed to inspect records without being allowed to resend a message, issue a refund or alter a customer account.
Conservative automation creates visible review work. That is a real maintenance cost, but it can be planned, assigned and measured. Blind retries and silent abandonment create business risk that is harder to see because the workflow may appear technically complete.
Apply these controls to one existing workflow
Choose one externally visible action in an existing workflow, such as sending a message or changing a customer record. Complete the worksheet for that action, then trace what happens if its request times out after the target may have accepted it.
The trace should end in one of two places: a verified business outcome or a named reconciliation owner with authority to decide the next action. If it ends only with “workflow failed,” “retry later” or “check logs,” the recovery path is incomplete.
For the renewal-message situation, this exercise produces a documented control path before the next ambiguous timeout occurs. Start with one action rather than redesigning the entire workflow. The observable result should be a clear retry rule, verification source and accountable destination for uncertainty.
Per-action control worksheet
Control Renewal-message example (hypothetical) Intended external effect Send exactly one renewal message for account A and renewal cycle B. Stable business-intent identifier Use an internal identifier such as renewal-account-A-cycle-B, and keep it unchanged across every attempt for that same renewal intent.Component that commits the effect The email delivery provider’s send operation commits the customer-facing effect, not the AI step or workflow runner that prepared the request. Receiver-enforced idempotency Treat this as unknown until current provider documentation confirms that the specific send operation enforces idempotency, explains how the same business intent must be represented and states any applicable limits. Do not infer support from workflow-level deduplication. Documented verification source Check the provider’s currently documented operation or message records using only identifiers or correlation data its interface supports, then compare them with the renewal system’s business records. If those sources cannot establish the outcome, keep it marked unknown and send the case to owned reconciliation. Retry rule and limit Suppress blind resending after an ambiguous timeout. Permit another attempt only when the message is verified as not sent, or when current documentation confirms receiver-enforced idempotency for the same business intent. Allow one controlled retry under that rule; if its outcome is also uncertain, stop and reconcile. Compensation or follow-up action A sent message cannot be made unsent. Record the incident and, if business context requires a clarification, route that consequential follow-up for authorization before sending it. Manual reconciliation owner Assign the unresolved case to the renewal operations owner, who reviews the available provider and business records and records the final determination. Escalation route and due time Escalate an unresolved case to the operations manager within one business day, or before the renewal’s next scheduled customer action if that comes sooner. Keep the outcome marked unknown until an authorized decision is recorded. Approval threshold Require operations-manager approval for any action taken while the outcome remains unknown and for any customer-facing clarification. A routine retry may proceed without further approval only after the message is verified as not sent or under documented receiver-enforced idempotency rules. Apply the same assessment to one externally visible action in your own workflow and confirm that an ambiguous outcome ends with either verified evidence or a named reconciliation owner.
Questions and answers
Should a workflow retry every timed-out request?
No. A timeout establishes that the caller stopped waiting, not that the target rejected the operation. Retry automatically only when the action is known to be safe, or when the component committing the effect enforces idempotency for the same business intent.
What if the target system has no idempotency support?
Use correlation data and documented target-side records to verify the outcome where possible. If the outcome remains uncertain, preserve the case for manual reconciliation or keep the final external action under human control.
Can compensation undo a message that was already sent?
It cannot make an observed message unsent. Compensation may instead create an appropriate new state, such as recording the incident or sending an authorized clarification. Ambiguous or consequential responses should receive human review.