Handle failures

Use this when a step can fail and the whole run should not have to. The policy is a reserved engine object in the node’s config; the plugin never sees it. Why the engine owns failure is in the failure model.

Steps

Route what fails to a dead-letter node

onError says what happens to a message that still fails: FAIL (default) fails the node, CONTINUE drops the message, ROUTE sends it, with its failure attached, to the node’s error port. Save as failures.json. A ticket whose kind is not bug matches no case, so the strict BRANCH fails it:

failures.json

{
  "sessionId": "failures", "name": "Failures",
  "description": "Sort tickets, keep the bad ones", "initiator": "docs", "projectPath": "/tmp",
  "workflows": {
    "sort": {
      "workflowId": "sort", "description": "Bugs to a queue, the rest to a dead letter",
      "nodes": [
        { "nodeId": "ticket", "type": "API_TRIGGER", "config": {} },
        { "nodeId": "by-kind", "type": "BRANCH",
          "config": {
            "selector": "payload.kind", "cases": { "bug": "bug-queue" },
            "engine": { "retry": { "maxAttempts": 2 }, "onError": "ROUTE" } } },
        { "nodeId": "bugs", "type": "CONSOLE_TERMINAL", "config": {} },
        { "nodeId": "dead-letter", "type": "CONSOLE_TERMINAL", "config": {} }
      ],
      "edges": [
        { "source": "ticket", "target": "by-kind" },
        { "source": "by-kind", "target": "bugs", "sourcePort": "bug-queue" },
        { "source": "by-kind", "target": "dead-letter", "sourcePort": "error" }
      ]
    }
  }
}

An edge with no sourcePort never carries a dead letter: the edge must say "sourcePort": "error", or the failure is dropped.

Retry the transient

retry runs the plugin again for the failing message with exponential backoff and jitter: maxAttempts (total, 1 to 10, default 3), backoffMs (default 200, doubling) and maxBackoffMs (default 10000). Narrow it with retryOn or noRetryOn (exception class names). Bad input (IllegalArgumentException) is never retried unless listed. A message produces all of its output or none: output emitted before a failed attempt is dropped.

Cap the damage with a failure budget

ROUTE and CONTINUE can hide an outage: if a service is down, every message goes quietly to the dead letter. A budget makes the node fail instead once more than maxFailures messages fail within windowMs:

"engine": { "onError": "ROUTE", "failureBudget": { "maxFailures": 10, "windowMs": 60000 } }

It needs onError set to ROUTE or CONTINUE; with FAIL the node is refused at validation.

Decide what a failed branch means at a join

A node with several parents starts by engine.trigger: ALL_SUCCESS (default) fails when one parent fails, ALL_DONE carries on with the parents that worked, ONE_SUCCESS carries on but fails when every parent failed. This is how to send a report even if one optional check failed.

Bound time too

A step that hangs is a failure. elementTimeoutMs limits one message (and counts as a failure, so retry and onError apply), deadlineMs limits the node’s whole run. See Time limits.

Verify it worked

curl -X POST localhost:9858/api/sessions -H 'Content-Type: application/json' -d @failures.json
curl -s -X POST localhost:9858/api/workflow/start -H 'Content-Type: application/json' \
  -d '{"sessionId":"failures","workflowId":"sort","payload":{"kind":"idea"}}'
curl -s -H 'Accept: application/json' localhost:9858/api/sessions/failures/executions/$EXEC/results

The RESULT event comes from dead-letter, and the message carries failure.attempts (2), failure.exception and failure.message; the run still ends SUCCESS. With "kind":"bug" it comes from bugs. Change onError to FAIL and the same ticket fails the node (FAILURE) and ends the run: that run is what the supervisor guide replays (Replay a supervisor).

Variations

A run can end SUCCESS with a failed branch inside it. Look at node statuses (Status values), not only the run status.
Make side effects idempotent. A retried or resumed step can see a message twice (Delivery and back-pressure).
Do not retry AI calls blindly. Prefer a gate after the model, or a fallback model (AI and deterministic work).

Node options reference for every engine key.
Messages reference for the failure.* labels.
Loop and retry for repeating work that has not failed.

Build workflows

Work with AI

Supervise runs

Run and operate

Extend

Handle failures

Steps

Verify it worked

Variations

Steps

Verify it worked

Variations

Related