Define the workflow and its completion condition
Map the request, processing steps, external operations and final result. Specify what counts as accepted, running, waiting, completed or failed. Receiving a job does not mean the requested work completed, and returning an acknowledgement should not imply a downstream action succeeded.
Give the workflow an identity that survives worker restarts and repeated delivery. Record the input version and the result the caller needs. In a hypothetical document-processing workflow, acceptance, extraction, reviewer approval and publication are separate states. This example is a design illustration, not an account of a client deployment.
Store progress where recovery can find it
Keep durable state for completed steps and outstanding work. Decide which system owns each transition and how it is recorded. A worker’s local memory is insufficient when a process can restart. Test a failure between producing a result and recording progress; recovery needs to determine whether the work should resume or be reconciled.
Separate workflow coordination from individual operations. A queue can deliver work without defining the whole business process. A workflow engine can help coordinate progress, but the application still needs correct inputs, permissions and external-operation semantics. Choose the mechanism according to the actual duration, dependencies and recovery needs.
Design retries around side effects
Classify failures before retrying. A temporary service failure differs from an invalid document or denied authorization. Define bounded attempts, delays and an exhausted-retry destination with an owner. Retries should give a dependency time to recover rather than repeatedly creating more work during an outage.
The official Temporal Activity documentation explains that an Activity can execute again when its result was not recorded, including after partial execution. It recommends idempotent operations and describes destination-enforced idempotency keys. Durable coordination does not establish that an external side effect happened only once.
For a repeatable external operation, use an operation identity that stays stable across retries and is enforced by the destination or an appropriate transaction boundary. Check retention and conflict behavior for that identity. If the destination cannot deduplicate safely, reconcile its actual state before repeating an uncertain operation.
Handle timeouts, cancellation and partial completion
A timeout means the caller did not receive a timely result; it does not prove the remote operation failed. Record the attempt and inspect the destination when the outcome is uncertain. Avoid converting a missing acknowledgement into a duplicate publication or another irreversible action.
Specify what cancellation can stop and what has already completed. A cancellation request may arrive after an external action. Define reconciliation or compensating steps where the business process permits them, and track their outcomes separately. A compensating action is another operation that can fail, not a guaranteed rewind.
Expose unresolved states to the operator and caller. Preserve evidence for manual review and define who can decide the next step. Keep an audit record of attempted, confirmed and corrected operations so a support team can explain the actual outcome.
Bound concurrency and make backlog visible
Set concurrency according to downstream capacity and the workload. Measure queue age as well as queue length: a short queue can contain jobs that have waited too long. Monitor worker availability, processing duration, repeated failures and requests that remain in intermediate states.
Plan how overload is handled, including admission limits, backpressure or an explicit delayed-work response. Avoid retry amplification across several services by documenting which layer retries which operation. Keep sensitive payloads out of broad diagnostic logs while retaining the identifiers needed to investigate.
Test recovery and release with ownership
Exercise duplicate delivery, worker restart, unavailable dependencies, malformed inputs, delayed acknowledgements and exhausted attempts in an approved test environment. Check completed business outcomes and unwanted duplicate operations, not only whether every job eventually exits a queue.
Version the workflow and operation contracts. Plan how an update affects work already running and how an operator resumes a paused or failed job. Assign owners for monitoring, reconciliation and rollback. Record the limits of any recovery test and repeat relevant checks when integrations change.
For AI workflows, connect these controls to the agent development guide and MLOps release guide, or discuss a scoped workflow integration.