An automation receives an order and asks the sales system to create a record. The connection drops while it waits for a response. Immediately trying again could create a duplicate because the destination may already have succeeded. Doing nothing could leave real work unfinished. The next step is to establish the destination outcome.
Automation used for everyday operations needs to distinguish successful work, known failures and unknown outcomes, with a route for someone to take over. It does not have to repair every failure automatically, but incomplete work should not disappear silently.
Detect missing work as well as explicit errors
Record a work reference, start time, latest step and the outcome returned by the destination. Connect the reference to an order, document or activity the business recognises, so the person receiving an alert can identify what is affected.
Monitor work that has remained incomplete beyond an appropriate period and scheduled work that never started. A missing daily report may produce no error if the triggering data never arrived. The absence of error messages is not evidence that the expected activity happened.
Distinguish a genuinely quiet day from missing input using source evidence or the latest successful check. Treating every day without an order as a system failure creates alerts that people must repeatedly dismiss.
Choose the response according to the failure
| Symptom | Response to consider | Action to avoid by default |
|---|---|---|
| Destination temporarily unavailable | Bounded retries with waiting intervals | Unlimited rapid repeated requests |
| Required information missing or invalid | Return the item for a specific correction | Retrying the same invalid data forever |
| Access or connection has expired | Notify the responsible owner and hold affected work | Expanding permissions to force success |
| No response after a command was sent | Reconcile whether the command succeeded | Creating another record without checking |
| Earlier step succeeded but the next failed | Resume from an established checkpoint | Restarting every action from the beginning |
AWS Step Functions' error-handling documentation distinguishes retry behaviour from fallback handling and allows limits and intervals to be configured. Those mechanisms help control repeated attempts, but do not replace checking whether a business action has already happened.
Set retry limits and delays in line with the connected system's constraints. If every waiting item retries immediately when a service recovers, the recovery itself can create another surge. Spread attempts appropriately and keep the remaining queue visible to its owner.
Return to the order with an unknown outcome
Give the creation request a reference that can be reconciled. If its response is missing, look up the result through that reference or a supported destination-status mechanism. When the existing sales order is found, record the successful result and continue instead of creating another.
If the outcome cannot yet be established, hold the item for review with the information needed to investigate. Designing repeated requests to avoid repeated effects is useful, but the actual destination must support that behaviour. Do not assume that clicking again will automatically be deduplicated.
Keep evidence for completed steps, such as the destination document reference or confirmation time. “Step two completed” without a useful reference may not be sufficient for later investigation. For actions that are difficult to reverse, identify who decides whether and how to compensate for an incomplete sequence.
This is a hypothetical scenario. If your workflow begins in chat, the guide to turning messages into sales orders explains how to establish references and confirmation points earlier in the process.
An alert should help someone continue the work
Describe the affected work, symptom, incomplete step, attempts already made and next action. Include a link to details available to authorised users. Route invalid-data problems to the data owner and connection problems to the relevant system owner instead of sending every issue to the same general channel.
Define a primary owner, cover when that person is unavailable, and escalation when nobody accepts the work. Group alerts caused by the same underlying incident while retaining visibility of affected items. Fixing the shared cause does not necessarily complete every item that failed during it.
Do not include credentials or complete customer documents in notifications. Keep sensitive details in the controlled record and send the reference and necessary context. One reliable investigation location is more useful than fragments copied across several channels.
A recovery guide should explain who can pause which steps, how to inspect work already completed, how to resume, and what evidence establishes completion. If staff temporarily perform tasks manually, record those actions so the returning automation does not repeat them.
Allow the owner to distinguish a paused queue from one that is accepting and processing new work. A recovery action should have a clear scope: one item, one class of failures or a wider workflow. This reduces the chance of a local correction unexpectedly repeating unrelated activity.
Rehearse recovery before depending on the workflow
In a test environment, try a slow destination, invalid data, duplicate events and interruption after some steps succeed. Check that alerts reach the right owner, work remains traceable and restarting does not duplicate the result. Keep test records separate from real business transactions.
After service resumes, reconcile the backlog against source and destination records. A healthy system indicator does not establish that the missed work was completed. Record the cause and response so the team can decide whether to improve data, retry rules or visibility.
For older applications, read integration options when a system has no API. The available evidence and recovery methods may differ from those of a system with direct, supported interfaces, and that difference belongs in the operating plan.
Does somebody still have to check whether every automation ran?
Talvyn Labs can help add work tracking, accountable alerts and recovery paths for incomplete items, so your team can identify the next action without creating duplicate records.
Review automation monitoring and recovery