Skip to main content
A checkpoint is the recorded output of one node in one execution. Yukta records checkpoints so that a restart or a resume can replay what a node already produced instead of running it again.

​
When to use it

Use restart and retry when a run failed and you want to re-run part of it without repeating earlier work. Use resume after the server stopped in the middle of runs: they are marked ABORTED at the next start, and you can continue them from their checkpoints.

​
What a checkpoint is

Every message a node emits is recorded, in order, as it leaves the node. When the node completes normally, a small durable end marker is written and forced to disk before downstream nodes see the node’s completion. A node counts as finished only when its marker exists. A resume trusts only that marker. It does not trust the execution log to say that a node finished, because the log record of a node end may still have been waiting to be written when the server stopped. If a checkpoint cannot be saved, or the marker cannot be written, the run carries on. The node is left unmarked, so a resume runs it again instead of trusting a partial output. The counter yukta.checkpoint.errors counts failed markers.

​
Where checkpoints are kept

​
In-memory caps

A node that goes over a cap is marked incomplete. A restart that needs its output is refused instead of replaying part of it, unless a spill directory is set. Do not point the spill directory at a folder shared with another process. Leftover spill files from an earlier process are removed at start. Messages come back from files and from PostgreSQL as plain JSON values. A resume reads a node’s whole output into memory. See Persistence for the file layout and PostgreSQL tables.

​
Restart, restart from a node, retry a node

These three operations start a new execution, with a new execution id, from a finished or running one. Rules:
  • The execution must still be in the in-memory archive: the newest yukta.restart.retained-executions executions, running or finished. Otherwise the server answers 404 with “no longer restartable”.
  • After a server restart the archive is empty, so restart and restart from a node are not available for earlier runs. Use resume for runs that a restart aborted.
  • On a run that is still going, only parents that have finished (they have their end marker) are replayed. If a parent is still producing output, the request is refused with 400.
  • A parent whose output went over the in-memory caps makes the request fail with 400, unless the output is on disk or in PostgreSQL.
  • Retry on a node that did not fail is refused with 400.
  • A restart takes a slot like any start. It can wait with status QUEUED or be refused with 429.
  • A second restart of the same execution while one is in progress is refused.
The response carries the id of the new execution.

​
Recovery on boot

When the server starts with a durable execution log, it reads the log and:
  1. Marks every run that has a start and no end as ABORTED.
  2. Restores the newest yukta.execution-log.retained-runs runs into history.
  3. Drops older runs from the log and deletes their checkpoints.
  4. Re-applies the newest scheduler limit change recorded in the log.
With the default in-memory log there is nothing to recover and a restart gives a clean server. With high availability the same recovery runs when a server becomes leader. See High availability.

​
Resume

Resume continues a run that a restart aborted, as a new run. Nodes that finished are not run again. Their recorded output is replayed to the nodes that did not finish, and those run.
The response carries the id of the new execution.

​
Conditions

  • A durable execution log (yukta.execution-log.dir or PostgreSQL), because the log holds the start payload.
  • A file or PostgreSQL checkpoint store, because only a durable end marker counts as finished.
  • The run is ABORTED.
  • The workflow definition is unchanged since the run started.
  • A run is resumed once.

​
Refusals

​
Auto-resume

With yukta.restart.auto-resume=true, after each recovery the server resumes every interrupted run, one after another. A run that cannot be resumed is logged and skipped. Auto-resume is off by default.

​
Approvals

An answered approval (a signal sent to a WAIT_FOR_SIGNAL node) is recorded with the checkpoints. A resumed run reuses the answer for the same message instead of waiting again. An approval that timed out is not recorded. A wait that was still open when the server stopped is lost, and the resumed run waits for a new signal.

​
Make plugin side effects safe to repeat

Delivery is at-least-once. There is a short moment between a plugin acting on the outside world and its node’s end marker reaching disk. If the server stops in that moment, a resume runs the node again. Retries repeat per message, and a resume can repeat a node. Make side effects idempotent:
  • Use the stable message id (and the attempt header) as the idempotency key when you call an external service.
  • Prefer “set to this value” operations over “add one”.
  • Check for an existing result before you create one.

​
Limits and defaults