Implementation
Durable execution or a job queue: what an AI workflow actually needs
Choose the least complex execution foundation that can survive your workflow’s real waits, retries and failures without confusing recovery with correctness.
A worker failed, but the workflow was still waiting
Imagine a supplier-onboarding process that collects company documents and asks an AI model to draft a risk summary. A compliance specialist reviews the summary, checks the supporting evidence and either approves the supplier or requests more information. That approval may take several days.
While the process is waiting, the worker that started it is replaced or crashes. The approval eventually arrives, but there is no useful process memory left on that worker. Later, a call to the procurement platform times out. The team cannot tell whether the supplier was created or whether it is safe to try again.
This is not one failure. It is a combination of lost execution state, an uncertain external effect and a potentially inconsistent business record.
The concise architecture answer is to use the least complex foundation that can reconstruct the workflow’s required state after failure. That may be replay-based durable execution, or it may be a queue, scheduler and explicit application state. Neither choice alone proves what happened in the procurement platform, repairs the internal supplier record or establishes that the AI summary was correct.
Durability is three different obligations
Teams often use “durability” to describe three separate responsibilities. Keeping them separate prevents an orchestration decision from being mistaken for a complete consistency design.
1. Reconstruct workflow progress
The system must know which steps completed, what it is waiting for and which transition is allowed next. In the supplier process, it should know that document collection and summary generation finished and that approval remains pending, even if the original worker no longer exists.
This is the obligation most directly addressed by an execution platform or an application-managed state machine.
2. Determine the external outcome
After the procurement request times out, the workflow needs to determine whether the external platform created the supplier. A timeout reports that the caller did not receive a timely answer. It does not establish that the remote operation failed.
Recovering the orchestration to the “create supplier” step therefore does not answer whether that step should be retried.
3. Reconcile the business record
The company’s authoritative supplier record must eventually agree with the intended and observed outcome. It might still say “approved, creation pending” even though the external supplier exists, or it might say “created” while storing no valid external identifier.
Reconciliation compares those states and routes disagreements to automated correction or a named human owner. A reliable execution history helps an investigation, but it does not make the business record correct by itself.
Map the workflow before choosing its foundation
Start with a specific workflow rather than a product shortlist. Its shape determines whether application-controlled recovery will remain understandable or turn into a substantial orchestration system of its own.
Pause duration matters because a process that waits for days cannot depend on a worker, connection or in-memory callback remaining available. Long waits require persistent state and a durable way to resume after an event or scheduled time.
Step count, ordering and branching indicate how much transition logic must be represented. A short linear process is easier to manage in a database row than a process with parallel checks, repeated requests for information and several legal next states.
Retry requirements and side effects reveal where automatic recovery can be dangerous. Repeating a read is different from repeating a supplier-creation request. Classify permanent failures separately from transient unavailability so the system does not keep retrying invalid input.
Worker-loss expectations define how much progress may be repeated. If a task can restart from the beginning safely, a coarse job record may be enough. If recovery must resume from a precise boundary, the design needs finer state or recorded execution history.
Audit needs determine which decisions, approvals, request identifiers and transitions must remain inspectable. Throughput affects queueing, worker concurrency and state contention, but volume alone does not decide whether replay is required. Operational ownership is equally important: somebody must maintain retries, permissions, monitoring and exception handling whichever foundation is chosen.
Capture one candidate workflow
Record its start condition and definition of completion.
List the ordered steps and permitted branches.
Mark human and system waits, including expected durations.
Identify every external call and the business records it can change.
State retry limits and the expected recovery after worker loss.
Estimate normal and peak work volume.
List the evidence needed for audit or investigation.
Record the data, system and operator permissions involved.
Name the likely operational owner.
What replay-based durable execution changes
A platform such as Temporal records an ordered event history for a workflow. When execution resumes on another worker, workflow code can replay that history to reconstruct its state. Recorded events guide the code back to the previous point without repeating work whose result is already in the history. Temporal documents this replay model for Workflows and Workflow Executions.
For the supplier process, the history could show that documents were collected, an AI summary Activity returned a result and the workflow began waiting for an approval. If a worker disappears during that wait, another worker can reconstruct the same logical state from the recorded events. Timers and supported waiting operations are part of the workflow history rather than ordinary in-process sleeps.
That mechanism imposes a determinism constraint. Given the same history, workflow code must make decisions that match the decisions recorded previously. Direct, unrecorded network calls, changing clock values and uncontrolled randomness can produce different paths during replay. Temporal places interactions such as API calls, database access and LLM invocations in Activities so their outcomes can be recorded separately from deterministic workflow logic.
Activities are also retry boundaries. Temporal’s documented default is to retry failed Activities under a Retry Policy, while Workflow Executions do not receive a retry policy by default. Retry timing, maximum duration and non-retryable failures must still match the operation rather than being accepted without review. The Temporal retry documentation explains that distinction.
Crucially, an Activity can execute more than once. A worker might complete the external operation and then fail before reporting completion. Temporal’s Python guidance describes Activities as following an at-least-once execution model and recommends idempotent Activity behavior for that reason. Replay itself does not rerun a result already recorded in workflow history, but an Activity attempt can still be repeated before successful completion is recorded. See the official Activity error-handling guidance.
Timeouts and heartbeats help detect failed or stalled Activity work. Temporal relies on Activity timeouts to detect cases such as a worker crashing after it starts a task; heartbeats can report progress for suitable long-running Activities. These controls improve execution recovery, but they do not determine whether an unacknowledged external effect occurred. The distinctions are documented in Detecting Activity failures.
Replay-based execution becomes compelling when the workflow contains long waits, many dependent transitions, parallel or compensating paths, and recovery requirements that would otherwise demand extensive custom state-machine code. Its value is not that queues disappear. It is that the platform owns more of the machinery needed to rebuild orchestration state consistently.
When a queue and explicit application state are enough
A queue-backed design can be the simpler adequate choice when the process has a small number of well-defined states and its recovery rules fit naturally in the application’s data model.
A supplier workflow might store states such as summary_pending, approval_pending, creation_pending, reconciliation_required and completed. Queue messages prompt workers to attempt eligible transitions. A scheduler emits work when a review deadline or follow-up time arrives. The database, not the worker or queue message, remains the source of truth for workflow progress.
The surrounding application must then provide several controls:
Persist each accepted state transition and the identifiers needed by the next step.
Prevent a stale or duplicate worker from applying an invalid transition.
Set retry limits and backoff according to the failure type.
Schedule delayed work without keeping a worker occupied.
Route repeatedly failing work to an exception path for investigation.
Recover partial progress without relying on one worker’s memory.
Amazon SQS illustrates representative queue behavior, not a universal queue contract. When a consumer receives an SQS message, the message remains in the queue but becomes temporarily invisible. If it is not deleted before the visibility timeout expires, it can become visible for another processing attempt. AWS also states that standard queues use at-least-once delivery and may deliver a message more than once, including during the visibility period. See the official SQS visibility timeout documentation.
That redelivery can recover abandoned work, but it does not remember the whole business process. If an approval arrives three days later, the application must know which workflow it belongs to and which transition it permits. If supplier creation completed before a worker crashed, the next attempt must identify that possibility. Enqueueing every step without persistent application state merely moves the uncertainty between workers.
A queue, scheduler and application state machine are often enough for a bounded process with clear transitions and modest recovery complexity. If those controls already exist in a well-owned application, introducing another execution platform can add more infrastructure and operational concepts than the workflow needs.
For a concrete queue-mode implementation rather than this architecture decision, see the guide to a recoverable n8n AI workflow.
Compare both designs against the same failures
Do not compare architectures only on their happy paths. Apply the same set of disruptions: terminate a worker during a wait, deliver the same work twice, delay approval, time out a task, stop after partial completion, make a dependency unavailable and lose the response to an external mutation.
The important distinction is where orchestration recovery lives. Replay-based execution derives workflow progress from recorded history and platform-managed execution semantics. A queue-backed design requires the application database, scheduler and transition handlers to carry that information explicitly.
Both designs still cross the same distributed-system boundary when they call the procurement platform. Neither can infer the external outcome from a missing response alone.
Two valid recovery paths
Replay-based durable execution
Recorded history reconstructs progress after worker loss, durable waits can hold delayed approval, and Activities can retry timeouts or unavailable dependencies. Duplicate execution and ambiguous external outcomes still require idempotency and reconciliation.
Queue plus explicit application state
Redelivery can recover abandoned work, while the application database and scheduler must preserve ordering, waits, partial completion and retry state. Duplicate delivery and ambiguous external outcomes still require safe handlers and reconciliation.
Recovery does not prove what happened outside the workflow
Suppose the procurement platform receives the supplier-creation request, creates the supplier and sends a response. The response is lost before the workflow receives it. From the workflow’s perspective, the call timed out. Retrying without another control could create a duplicate.
Several mechanisms can reduce or resolve this uncertainty, but each has a bounded role.
Idempotency keys let the caller identify retries as the same intended operation when the external API supports such a contract. The same key must represent the same intent across attempts. The service must also retain enough information to recognise the key and respond consistently. Retention and mismatch behavior are properties of that API’s contract, not of the workflow engine. The Amazon Builders’ Library explains why caller-provided request identifiers are preferable to guessing intent from identical parameters in Making retries safe with idempotent APIs.
Stored request and result identifiers connect workflow intent to external evidence. For example, the internal record can store the request key before attempting creation and add the returned supplier identifier when known. That does not resolve a lost response by itself, but it gives recovery logic and operators stable references.
External status checks can determine whether a resource associated with the request exists, provided the external system exposes a suitable lookup. A generic search by supplier name may be inconclusive if another process could have created a similar record. A request identifier or authoritative external ID is stronger evidence.
Transactional outbox patterns address a different boundary: updating an application database and publishing an event. Writing the business change and an outbox record in the same local transaction avoids one succeeding while the other is silently omitted. The event publisher and consumer must still tolerate duplicates. AWS documents these responsibilities in its transactional outbox pattern.
Compensating actions attempt to counter an earlier effect when a multi-step operation cannot complete. They are business operations, not magical rollbacks, and they can fail or require approval themselves.
Human exception handling is appropriate when automated evidence remains insufficient or when reversal has significant consequences. The exception should present intentional identifiers and a fixed summary of the discrepancy, not raw prompts, documents or unfiltered error content.
Reconciliation ties these controls together by comparing three things: what the workflow intended, what the external system reports and what the authoritative business record says. The output is either a confirmed match, a safe correction or a clearly owned exception.
Reliable orchestration is not reliable AI output
The execution foundation can make sure the AI summary step is scheduled, retried and recorded. It cannot establish that the summary is accurate, useful or compliant with company policy.
The supplier summary still needs evaluation against its purpose. Relevant controls may include source references, structured validation, confidence or completeness checks, restricted data access and human approval before consequential actions. For some workflows, a deterministic rules engine or a manual checklist is a better foundation than an AI-generated assessment. That is especially true when criteria are stable, fully expressible and too consequential to infer from free-form output.
The operating tradeoff between the two execution paths is also broader than recovery logic:
Infrastructure and maintenance: Durable execution introduces a service, worker model, histories and platform-specific development constraints. A queue-backed design may use infrastructure the team already operates, but its custom state machine and recovery code also require maintenance.
Observability: A workflow platform can provide execution-oriented history and status. A queue-backed application must connect queue metrics, scheduler activity, database transitions and exception records into a usable operational view.
Permissions: Workers should receive only the data and system permissions required for their activities. Human operators need enough access to inspect and resolve exceptions without gaining unnecessary access to prompts, source documents or unrelated records.
Auditability: Either design can retain useful evidence if identifiers and transitions are designed deliberately. A platform history is not automatically the complete business audit record.
Throughput: Both approaches must account for worker capacity, dependency limits and contention around shared records. Throughput requirements do not remove the need for safe retry and state transitions.
Ownership: A named team must own workflow changes, retry policy, incident response, permission review and reconciliation. Moving orchestration into a platform changes that work but does not eliminate it.
The fair case for replay-based durable execution is that it can replace a substantial amount of custom recovery machinery in a complex, long-running process. The fair case against it is that a short, stable workflow may be easier to understand and operate as explicit application state with queued transitions.
For broader operational monitoring choices, see Monitoring the AI operating layer. If the decision requires implementation support, Workflow & Agent Implementation is the relevant service area.
Choose the least complex design that survives the recovery drills
Use a bounded decision procedure before committing to a wider platform rollout:
Map one existing workflow. Use the earlier capture worksheet to identify its waits, transitions, side effects, records, permissions and likely owner.
Classify each failure obligation. For every important failure, state whether the system must reconstruct workflow progress, establish an external outcome, reconcile a business record or do more than one.
Choose the simpler adequate state model. Prefer a queue, scheduler and explicit application state when the transitions and waits remain clear and the team can own the recovery code. Consider replay-based durable execution when reconstructing many long-running or interdependent transitions would otherwise require substantial custom orchestration machinery.
Assign the unresolved boundaries. Name who owns external status checks, idempotency contracts, compensating actions and human exceptions. Do not leave these responsibilities implicit in the execution-platform choice.
Exercise recovery before expanding. Run controlled scenarios in a safe environment using non-production effects or reversible test records. Terminate a worker during a wait, repeat a delivery and simulate an unavailable or ambiguously responding dependency.
Inspect observable evidence. Confirm that you can locate the persisted workflow state, retry record, external request or resource identifier, current business-record status and any routed human exception.
Adopt only after the evidence is legible. The selected design is adequate when an operator can explain what should happen next after each drill without relying on the memory of the failed worker or making an unsafe guess about an external effect.
Next step: exercise one workflow before committing
Choose one candidate workflow and safely simulate two moments from the supplier-onboarding situation: worker loss while the process is waiting, and a missing response from one external call.
Then try to identify the current workflow state, the observed external outcome and the business record that may need reconciliation. The result should be a visible list of recovery gaps. That evidence is enough to decide whether explicit application state remains manageable or replay-based durable execution is justified.
Questions and answers
Does a long-running AI workflow always need durable execution?
No. A queue, scheduler and explicit application state can be sufficient when the workflow has a small, clear set of states and the application can preserve waits, retries, ordering and partial completion. Replay-based durable execution becomes more compelling as those transitions and recovery paths grow difficult to reconstruct safely.
Does Temporal guarantee that an external API operation happens exactly once?
No. Temporal Activities may be retried, including when a worker completed an external operation but failed before reporting completion. External effects still need controls such as a supported idempotency contract, recorded identifiers, status checks and reconciliation.
Is putting every step on a queue enough to make the workflow durable?
No. A queue can preserve and redeliver work, but the application must still record workflow state, schedule delayed steps, control retries, handle duplicate delivery and recover partial progress. Exact behavior depends on the selected queue.
Does reliable orchestration make an AI result correct?
No. Orchestration can make execution and recovery dependable, but model output still needs suitable inputs, evaluation, policy checks and human review where the consequences require it.