Skip to main content
This page lists symptoms you can observe on a running server, what causes each and how to fix it. Read the symptom table first, then the sections for the ones that need more explanation.

​
Symptom table

​
Execution stays QUEUED

An execution is QUEUED while it waits for a slot. Check:
  1. yukta.executions.running against yukta.execution.max-concurrent, max-concurrent-per-session and max-concurrent-per-workflow.
  2. Whether running executions are stuck. A run holds its slot until it ends, and a paused run keeps its slot with its clocks frozen. Stop or resume stuck runs.
  3. The queue order. Higher yukta.priority goes first, then the session with the fewest running executions, then arrival. Change a waiting execution’s priority with POST /api/executions/{id}/priority?priority=n.
To raise limits without a restart, use POST /api/scheduler/limits?maxConcurrent=..&maxPerWorkflow=.. (needs the ADMIN role). Set yukta.execution.queue-timeout so a start that waits too long ends TIMED_OUT instead of waiting forever. See Admission and limits.

​
A node or run times out

TIMED_OUT has several causes, one per clock: Pausing freezes every clock, so a pause never causes a timeout. See Time limits.

​
UPSTREAM_FAILED is not the failure

Only the node that failed ends FAILURE. Every node behind it ends UPSTREAM_FAILED, and the first terminal that fails ends the run and cancels the other branches. Look for the node with FAILURE or TIMED_OUT, read its logs and fix that one. A node with several parents fails when one parent fails by default. Set engine.trigger to ALL_DONE or ONE_SUCCESS if it should carry on. See Failure handling.

​
A command is rejected with 403

Several checks can answer 403:
  • Role. A VIEWER key cannot change anything, and an OPERATOR key reaches only paths under /api/executions/ and /api/workflow/. Defining workflows and changing scheduler limits need ADMIN.
  • Session limit. A key limited to sessions reaches only its own.
  • Host or Origin. With no key set, an unknown Host or a foreign Origin on a changing request is refused with no body.
  • A reviewer. A supervisor that reviews commands can refuse any command. The refusal is recorded.
For a refused control command, read the reason from the audit: GET /api/control/audit?executionId=<id>. A refused entry shows the command and the reason. See Security.

​
Resume is refused

POST /api/executions/{id}/recover answers 409 when:
  • The run was not aborted. Only a run that a restart aborted can be resumed.
  • The run is being resumed or was already resumed. A run resumes once.
  • The workflow changed since the run started.
  • The start payload was not recorded.
  • Every node had already finished.
It answers 404 when the run is unknown (for example, after a restart with the in-memory log) or its workflow no longer exists. It needs yukta.execution-log.dir or PostgreSQL, and a file or PostgreSQL checkpoint store. See Checkpoints and resume.

​
Approvals invisible after a restart

The list of pending approvals and the waits behind it live in memory. After a restart, or a high-availability takeover, an open approval is gone, and it reappears only when its node waits again, for example in a resumed run. An approval that was already answered is recorded with the checkpoints, and a resumed run reuses the answer.

​
Process execution is refused

PROCESS_EXECUTOR refuses with one of these messages: On a placed node, set the allow-list on the worker. See Security.

​
The server refuses to start on a non-loopback address

The server refuses to listen on any address but loopback unless a key exists. The message reads “Refusing to listen on ’…’ without an API key”. Either set yukta.api-key (YUKTA_API_KEY) or yukta.api-keys, or set YUKTA_SERVER_ADDRESS=127.0.0.1. The container image listens on 0.0.0.0, so it always needs a key. See Security.

​
Log lines are missing

  • Retention. Node log lines are kept in memory for 30 minutes by default, between 1 minute and 24 hours (yukta.logs.store.retention.default-period-minutes). They are lost on restart.
  • Overflow. When more than 10,000 lines wait to be recorded, the extra lines are dropped. yukta.tracker.dropped{type=log} counts them.
  • Failed writes. yukta.tracker.log.append.failed counts failures to write to the execution log. They never block a run.
  • Placed nodes. Log lines a plugin writes on a worker do not reach the execution’s log stream.
  • Cut lines. A node that did not write live logs has its output logged cut to 2,048 characters.

​
Event streams disconnect

  • A server restart or takeover drops every open stream. Reconnect with the Last-Event-ID header to continue.
  • Idle streams send a keep-alive comment every 15 seconds. A proxy with a shorter idle timeout closes them.
  • A client that connects late catches up from the progress history, which is kept for 5 minutes (yukta.streaming.status-history-ttl-minutes), 200 snapshots per execution.
  • Live parts of a stream are best effort. Read final state from the status or result endpoints.
See Streams.

​
PostgreSQL does not start

The server applies the schema migrations first and does not start if one fails. Check:
  • Both URLs are set: spring.flyway.url for migrations (jdbc:postgresql://...) and spring.r2dbc.url for run-time access (r2dbc:postgresql://...). They are different URL forms for the same database.
  • The migration account can change the schema.
  • The postgres profile is active and the build used -Pyukta.persistence=postgres. See Persistence.