# Workflows (experimental)

Source: https://docs.littlebigbrain.com/guides/workflows/

Workflows are opt-in. A server enables them with `--workflows`.

To start from one of LBB's templates, see [Build a workflow](https://docs.littlebigbrain.com/start/first-workflow/).
It tests a workflow on two records, then creates it.

LBB provides message workflows with a stable identity, explicit saved JSON state,
and a durable inbox. A normal TypeScript or Python handler runs one bounded turn per message;
named step outputs are saved as checkpoints. State, reply, inbox advancement and an
optional continuation commit atomically. After a crash, only the unfinished turn
replays. Every completed message automatically starts a fresh history boundary.

The original declarative activity DAG remains available under **Activity graphs**.
This is an experimental engine with its own programming model, not a Temporal SDK.

**Durability:** an acknowledged request is saved in object storage. Workflow
state has the same truth contract as the graph. Every serving node reads and
writes the same objects, and no node keeps workflow state of its own. A lost
node loses no acknowledged work.

## Enable and try it

Start the server with `--workflows`. Workflows need no directory and no
coordinator.

```sh
cargo run -p lbb-server-bin -- \
  --root /private/tmp/lbb-workflow-objects --tenant acme --graph main \
  --bind 127.0.0.1:7400 --api-token lbb_sk_test_local \
  --workflows

# In another terminal (create the graph if it does not already exist):
curl -H 'Authorization: Bearer lbb_sk_test_local' -X POST \
  'http://127.0.0.1:7400/v1/graph/create?graph=main'
curl -H 'Authorization: Bearer lbb_sk_test_local' -H 'Content-Type: application/json' \
  --data-binary @database/examples/workflows/approval.json \
  'http://127.0.0.1:7400/v1/workflows?graph=main'
LBB_API_KEY=lbb_sk_test_local python3 database/examples/workflows/worker.py
```

Environment variables: `LBB_WORKFLOWS_ENABLED=true` on every serving node;
`LBB_WORKFLOW_POLL_MS` (default 1000) and `LBB_WORKFLOW_RECONCILE_MS` (default
30000) tune how fast a node sees work written by another node. Workflow requests
are served by the node that receives them and skip graph affinity, because every
node reads the same objects. Node clocks must be synchronized: leases compare
against the clock of the node that handles the request.

### The Workflows page

Open **Operate → Workflows** in the console. When you run the console
locally, start it with `VITE_LBB_FEATURE_FLAGS=workflow-starters,nvme-workflows`.
The page lists every workflow of the graph, grouped by job. The list includes
the workflows of the templates your worker runs on files (`rows`,
`sections`). **New workflow** creates a workflow from a template, a message
workflow, or an activity graph from its JSON definition. The words on these
pages are defined in the [glossary](https://docs.littlebigbrain.com/start/glossary/).

A workflow's page has these tabs. `documents.link` and `documents.map` also
have **Documents**. `search.embed` and `ontology.fit` have no **Runs** tab.

| Tab | Shows |
| --- | --- |
| Overview | The status, the lag, the waiting work and this month's spend. A workflow your worker runs can show a **No worker** warning. It shows when a message has been ready for two minutes and no turn runs. It gives the command that starts a worker. |
| Results | What the workflow did: per file, per document, or the values it fills. |
| Review | Its review items per status, with a link to the review inbox kept to the workflow. Today the inbox is the **Suggestions** tab of the knowledge graph. |
| Quality | Its gate, the gate's recent calibration sessions, the model checks of its uses, and its open review items. It links to **Evals**. A workflow your worker runs has no Quality tab. |
| Spend | Its model calls this month per use and model, its monthly budget, and links to **Models → Spend & limits**. For a workflow your worker runs, the calls of every worker on the graph, because they are not yet counted per workflow. |
| Runs | Each run with its steps and model calls. |
| Settings | Edit the watched fields, the parameters, the gate and, for a workflow your worker runs, its target, owned outputs and batch. `documents.link`, `documents.map` and `values.extract` also take a monthly budget. A template that fixes what it watches keeps its class and fields. The page shows what changes before it saves. |

The engine details (the instance, the shards and the inbox) are under a
**Details** disclosure.

A row whose running turn reports progress in the
[progress convention](#progress-convention) shows a Running line: the phase,
"40 of 97 pages", how long it runs, and a bar. An engine instance's page
sends a message, pauses, resumes or deletes the workflow. It shows the
Running line of the current turn, the **Queue** (the turn that runs and the messages that wait, oldest or newest first, each with
its [origin](#message-origins), age and size; a message's payload is read when
you open it), and **Turns**, which lists only completed and cancelled turns
newest first, a page at a time. Queued, running, waiting and failed turns stay
in Queue; they do not appear again in Turns. History pages count finished
turns, so pending messages do not displace them. A turn opens its steps,
its message, its result, the state it left, and the operation in
progress. The current turn takes a signal, a retry or a cancel. LBB's own model
workflows show the same way. An activity graph's page shows each step with its state, sends a
signal and cancels the run. Run the DAG example above and send `approved` with
JSON value `true` to finish it. Run the worker in a separate terminal; the
console does not execute activity code.

## Message workflows and automatic rollover

```typescript
import { LbbClient, workflow, WorkflowWorker } from "@littlebigbrain/client";
const client = new LbbClient({ baseUrl: "http://127.0.0.1:7400", apiKey: "lbb_sk_test_local" });
const counter = workflow({
  name: "counter", version: "v1", initialState: { count: 0 },
  async onMessage(ctx, state, message: { add: number; remaining?: number }) {
    const amount = await ctx.step("add", async () => message.add);
    const next = { count: state.count + amount };
    if ((message.remaining ?? 0) > 1)
      return ctx.continue(next, { ...message, remaining: message.remaining! - 1 });
    return { state: next, result: next.count };
  },
});
const handle = await counter.start(client, "agent-1");
await handle.send({ add: 1, remaining: 3 }, { id: "request-1" });
const stop = new AbortController();
await new WorkflowWorker(client, [counter], { worker: "worker-1" }).run({ signal: stop.signal });
```

A runnable worker is `database/examples/workflows/counter.mjs`. Build the SDK first
with `npm run build -w @littlebigbrain/client`; set `LBB_WORKFLOW_START=1` to create
and seed its example, or create it in the console and send `{"add":1}` there.

`ctx.step(key, async ({effectId, signal}) => ...)` saves its JSON result before
proceeding. Put all I/O, clock reads and randomness inside steps. Pass `effectId`
to the downstream service's idempotency facility: the effect can succeed before
its checkpoint, so delivery remains at least once. The key is stable across
attempts and recovery, scoped to this namespace/identity/message/step.

`ctx.sleep(key, milliseconds)` saves a durable timer and releases the worker.
When due, the handler restarts with saved step results and skips the elapsed
sleep. Await steps and sleeps sequentially; parallel or unawaited steps are
rejected. Keys must be unique within the turn. Return JSON values (including
`null`); do not return undefined from a step. Cancellation is cooperative through
`signal`; remote fencing cannot undo external effects.

Unlike a long replay history, the next message starts with saved state and zero
steps. There is no user-managed run chain or mandatory Continue-As-New ceremony.
For one large batch, return `ctx.continue(state, nextMessage)` before the turn's
64-step/256-KiB budget; its successor is appended behind messages already queued.
State, reply and successor are committed together. A failed turn blocks the inbox
until retry or cancellation; it cannot partially publish new state. A paused
workflow accepts messages and allows its already leased turn to finish, but takes
no new claims. Cancellation targets an exact turn and leaves saved state unchanged.

Code versions are pinned for an identity. Keep old worker versions available;
a differently registered version cannot claim its work. Within a version, changing
step order/keys or omitting saved steps is a permanent SDK replay error. This does
not snapshot arbitrary JavaScript stacks or infer state migrations. Timers block
that identity's inbox until due; run independently interruptible work as separate
identities. The SDK renews leases and abandons fenced attempts; claim transport
errors can stop the worker loop, so supervise and restart workers. Ordinary handler errors retry; `WorkflowError` fails permanently.

## LLM runs, signals and progress

On a workflow's page in the console, **Send message** queues another turn on the
workflow identity. It does not send a chat message to a model and cannot
interrupt an existing turn. Use **Respond** on a turn that waits for a signal,
or **Send a signal** on the current turn, for approvals, tool callbacks and other
events an existing turn needs. The SDK keeps `handle.send(input, {id})` for inbox admission and
adds `handle.signal(name, value, {turn, id})` for this distinct operation.

`database/examples/workflows/agent-review.mjs` is a runnable draft → human review
→ delivery example. Build the SDK, then run it with `LBB_API_KEY` and
`LBB_WORKFLOW_START=1`. Without adapter URLs it uses labelled local demo results.
Set `MODEL_URL` and `TOOL_URL` to your own JSON HTTP adapters to run real operations;
provider credentials and provider-specific request mapping belong in those adapters.
The console's **New workflow** can create `agent-review@v1` with initial state
`{}`; then send the message `{"task":"Draft a customer response"}`. Read the
saved draft in the turn's steps, then respond to `approval` with
`{"approved":true}` or `{"approved":false}`.

```typescript
const review = await ctx.waitForSignal<{ approved: boolean }>("review", {
  name: "approval", timeoutMs: 86_400_000,
});
if (!review.received) return { state, result: "approval timed out" };
if (!review.value.approved) return { state, result: "rejected" };

// From a UI, webhook handler or another authenticated client:
await handle.signal("approval", { approved: true }, { turn: 7, id: "review-event-123" });
```

Signals target a specific turn; delayed callbacks never apply to a successor.
Distinct event IDs permit repeated signals of the same name. Exact retries return
the original receipt, including after the turn moved to history; a reused ID with
a different name/value conflicts. A turn buffers up to 64 signals, each at most
32 KiB, within the existing 256 KiB turn budget. New signals to completed/cancelled
turns conflict. Signals to queued, paused or failed runs buffer until they can run.

A signal wait consumes the oldest unconsumed matching signal in the same
conditional write as the checkpoint. If none exists, it releases the worker until a
matching signal or timeout. It resumes the same attempt, replaying saved results.
Signal receipt strictly before the deadline wins; a signal at/after the deadline
stays buffered for a subsequent wait. A timed-out wait returns
`{received:false,value:null}`; JSON null is still a valid received value when
`received:true`. Timeouts are 1 ms–365 days. Signals do not execute concurrent
handlers or interrupt a sleeping/running step: the handler explicitly awaits them.
Use cancellation for interruption. Targeting a turn is intentional; signals are
not automatically transferred across continuations. Unconsumed events remain in
that turn's history, so consume relevant signals before returning.

For long LLM or tool calls, configure a step independently of the worker lease:

```typescript
const pages = await ctx.step("fetch-documents", async ({ checkpoint, heartbeat, signal, effectId }) => {
  // Your tool can resume from this saved cursor after a retry.
  let cursor = checkpoint;
  for await (const page of fetchPages({ cursor, signal, effectId })) {
    cursor = page.nextCursor;
    await heartbeat({ pages: page.count }, { checkpoint: cursor });
  }
  return { cursor }; // Completed result is checkpointed once.
}, { kind: "tool", timeoutMs: 300_000, heartbeatTimeoutMs: 30_000 });
```

`kind` (`model`, `tool`, `step`) is operation metadata, not a provider adapter.
Every SDK step has a per-attempt timeout (default 5 minutes; 100 ms–24 hours).
Optional `heartbeatTimeoutMs` requires **explicit operation progress** within that
interval (at least 100 ms, no longer than the execution timeout). Automatic
lease renewal only reports worker liveness and extends neither deadline. A hung
model request therefore times out even while its worker is alive. Do not enable a
short progress timeout for a non-streaming provider that cannot report progress;
use its execution timeout instead. Operation failures use the workflow's attempt
limit/backoff and replay prior completed steps.

Progress heartbeats replace the current details (8 KiB) and resume checkpoint
(32 KiB). They use increasing sequence numbers per attempt; exact retries do not
extend deadlines. The next attempt receives the last checkpoint through the step
context. A failed/retrying turn exposes its last operation; completing its step
clears active progress. The API/console show operation kind/key, attempt, deadline,
last progress heartbeat, details and cursor separately from worker lease status.

The SDK propagates `AbortSignal` and stops waiting when a lease, operation or
progress timeout is lost, even if a provider ignores cancellation. The server
fences stale checkpoint/completion writes and schedules retries. Cancellation does
not roll back an external tool effect or guarantee that a provider stops billing;
pass the effect ID to an adapter that actually enforces idempotency. Cancelling a
run is observed on the worker's next lease renewal (normally every lease/3).

Keep token streaming on a separate ephemeral channel. Coalesce progress updates;
do not persist every token or full conversation in heartbeat payloads. Progress
replaces the current value, and each acknowledged change is one object write.
Large transcripts/results belong in referenced objects.
This release does not add provider integrations, token/cost enforcement, streaming
transport, child workflows or parallel steps.

### Progress convention

Operation progress details follow one shape when a workflow wants clients to
draw them. The console draws them as the Running line and a bar, for any
workflow.

```json
{"phase": "reading pages", "done": 40, "total": 97, "unit": "pages"}
```

| Field | Rule |
| --- | --- |
| `phase` | Plain words, 1 to 80 characters. Required. |
| `done` | A whole number, at least 0. Required. |
| `total` | A whole number, or absent when the step does not know it. |
| `unit` | `documents`, `pages`, `mentions`, `links`, `proposals`, `instances`, `questions`, `messages`, `items`, or another short word (1 to 32 letters, digits, spaces, `-` or `_`). Optional. |

Details with a `phase` key follow the convention. Other keys may sit beside
the four fields. The server refuses details with a `phase` key that break a
rule (`400`). Details without a `phase` key stay free-form.

The turn and instance views carry a normalized `progress`: `phase`, `done`,
`total`, `unit`, `started_at` (when the operation started) and `heartbeat_at`
(the last progress write), both RFC 3339. A turn's `progress` reads its
operation; a failed or retrying turn keeps its last progress. An instance's
`progress` is set only while its current turn runs. The views fill it on read;
the stored turn does not hold it. Clients read `progress` and never parse
`details`.

The TypeScript SDK reports it with `progress` in the step context:

```typescript
await ctx.step("read", async ({ progress }) => {
  for (let page = 1; page <= pages.length; page++) {
    progress("reading pages", { done: page, total: pages.length, unit: "pages" });
    await readPage(pages[page - 1]);
  }
});
```

`progress` is coalesced. A phase change is sent at once. Other values are sent
at most once per 3 s, or once per half the step's `heartbeatTimeoutMs` when
that is shorter (worker option `progressEveryMs`). The last value wins and is
sent when the step returns. Each call keeps the resume checkpoint of the last
`heartbeat`. A value outside the convention fails the step for good. The Python
worker reports progress the same way
([Run a worker in Python](https://docs.littlebigbrain.com/guides/python-worker/)).

### Message origins

A message can say where it came from. `POST /v1/workflows/instances/message`
takes an optional `origin`, at most 1 KiB:

```json
{"workflow_id": "agent", "id": "m-412", "message": {"document": "report.pdf"},
 "origin": {"kind": "commit", "from_seq": 409, "to_seq": 412,
            "fields": ["https://littlebigbrain.com/p/file_sha256"],
            "requests": ["file-upload:3fa2"]}}
```

| Field | Meaning |
| --- | --- |
| `kind` | Required. `app` (your code), `console` (a person on the console), `manual` (**Run now**), `watch`, `backfill`, `commit`, `schema_changed`, `requeue`, `migration` (a watch), `schedule`, `event` (an event of the data plane), `signal` (a callback) |
| `trigger`, `reason`, `event` | Identifiers, 1 to 128 ASCII letters, digits, `_`, `-` or `.` |
| `from_seq`, `to_seq`, `target_seq` | Commit sequences: the range that caused it, or the commit a pass reads up to |
| `fields`, `requests` | At most 32 entries each: the changed fields, and the request ids of the commits |
| `actor` | Who asked, at most 128 characters |
| `attempt` | How many times the item was sent before |
| `note` | A short note, at most 280 characters |

One origin covers one message. A message of several instances holds the union
of their fields. The engine stores the origin with the turn. The turn, the
inbox and the worker's claimed task show it; the SDK hands it to the handler as
`ctx.origin`. The origin is no part of the message's identity: a repeated
message id with the same message answers the first turn, with the first
origin.

A continuation gets `{"kind": "continuation", "turn": <n>, "message_id": <id>}`,
the turn that continued. A sender cannot set the kind `continuation`. A
developer workflow's trigger messages name their origin from their reason:
`backfill` with `target_seq`, `commit` with `to_seq`, `manual` for a refresh,
`schema_changed`. A CDC wake is `{"kind": "event", "event": "cdc_capture"}`.
The console's **Send message** sends `console`.

### Inbox

`GET /v1/workflows/instances/inbox?id=<id>` lists the unfinished turns of one
instance, oldest first: the turn that runs or runs next, then the queued
messages. Each entry has `turn`, `message_id`, `status` (`queued`, `waiting`,
`running`, `failed`), `received_at_ms`, `ready_at_ms`, `attempt`,
`message_bytes`, `origin`, `error` and the running turn's `progress`. The page
also has the instance's type, version, status, `pending_messages`, and the
`oldest_received_at_ms` and `newest_received_at_ms` of every unfinished turn.
`after` is a turn-number cursor, `limit` is 1 to 100 (default 50), and
`payloads=true` adds each `message`. It is one read of the instance object.

`GET /v1/workflows/inbox?workflow_type=<type>&version=<v>` lists the instances
of a type (of every version without `version`) that have unfinished turns, by
id: each with its `pending_messages`, `oldest_received_at_ms` and inbox `head`
(no payload). `identities` counts the instances of the type in the namespace
index, `with_messages` those the index lists with a turn. The index lags a
write of another node by at most its poll interval, so a listed instance whose
inbox emptied since is left out. `after` is an instance-id cursor, `limit` is 1
to 100 (default 50), and each listed instance is one read.

Every caller can read both, the reserved `lbb.*` types included.

### Message API

All requests inherit graph/stack authentication. Message IDs are required.
While a turn is in the kept history, an identical retry returns that turn. A
changed payload conflicts. The `continue-` prefix is reserved.

| Method and path | Behavior |
| --- | --- |
| `POST /v1/workflows/instances` | `{id,workflow_type,version,state,lease_ms?,max_attempts?}`; identical creation retries are safe |
| `GET /v1/workflows/instances` | `after` ID cursor and `limit`; returns `instances,next_after` |
| `GET /v1/workflows/instances/get?id=…` | Saved state, status, pending count, completed and history counters, `history_pruned_through`, current turn, and the running turn's `progress` |
| `POST /v1/workflows/instances/message` | `{workflow_id,id,message,origin?}`; durable inbox admission; `origin` per [Message origins](#message-origins) |
| `GET /v1/workflows/instances/inbox?id=…` | `after` turn-number cursor, `limit` 1–100, `payloads`; the unfinished turns with their origins ([Inbox](#inbox)) |
| `GET /v1/workflows/inbox?workflow_type=…` | `version`, `after` id cursor, `limit` 1–100; the instances of a type with unfinished turns ([Inbox](#inbox)) |
| `POST /v1/workflows/instances/control` | `{workflow_id,action,turn?}`; `action` is `pause`, `resume`, `retry` (a failed turn), `skip` (a failed turn), `cancel_turn`, or `wake`. `retry` and `skip` need `turn`. `wake` needs no `turn`: it moves a turn that waits without an attempt (a `retry_at_ms` failure or a capacity deferral) to now, and changes nothing when no turn waits so. A step's sleep or signal wait stays |
| `POST /v1/workflows/instances/delete` | `{workflow_id}`; deletes the instance, its turns and its history. `{deleted}` is false when no instance had the id. The id can be created again after successful deletion. In a selected environment, `503 workflow_source_closure_pending` means cleanup is continuing: retry the same delete after `Retry-After` |
| `GET /v1/workflows/instances/history?id=…` | `after` turn-number cursor, `limit` 1–50; `turns,next_after` from the oldest kept turn |
| `GET /v1/workflows/instances/turn?id=…&turn=…` | One turn and its saved steps, reply and state after completion; 404 for a removed turn |
| `POST /v1/workflows/turns/claim` | `{worker,workflows:[{workflow_type,version}],wait_ms?}`; only the inbox head may run |
| `POST /v1/workflows/turns/checkpoint` | `{workflow_id,turn,token,position,key,kind,output?,delay_ms?}`; ordered step or sleep |
| `POST /v1/workflows/turns/complete` | `{workflow_id,turn,token,state,result?,continuation?:{message},usage?}`; exact retries work while the turn is kept. `usage` is what the turn's model calls used; the TypeScript SDK sends it |
| `POST /v1/workflows/turns/fail` | `{workflow_id,turn,token,error,non_retryable?,usage?}` |
| `POST /v1/workflows/turns/signal` | `{workflow_id,turn,id,name,value}`; buffered, idempotent event targeting one turn |
| `POST /v1/workflows/turns/wait-signal` | `{workflow_id,turn,token,position,key,name,timeout_ms}`; consume or suspend |
| `POST /v1/workflows/turns/operation` | `{workflow_id,turn,token,position,key,kind,timeout_ms,heartbeat_timeout_ms?}`; start/resume step |
| `POST /v1/workflows/turns/progress` | `{workflow_id,turn,token,key,sequence,details,checkpoint}`; operation progress, independent of lease; details with a `phase` key follow the [progress convention](#progress-convention) |
| `POST /v1/workflows/turns/heartbeat` | `{workflow_id,turn,token}`; renew a live lease |

Defaults: three attempts and a 60-second lease. Failure backoff doubles from one
second to 60 seconds; durable sleep does not consume an additional attempt.
Limits: 64 sequential steps, 64 KiB saved state/output/reply, 32 KiB message,
256 KiB stored turn, 128 pending messages per identity. IDs/type/version/step keys
use the ASCII identifier rules below. Large payloads belong in referenced objects.
The combined graph budget is 1,000 DAG runs/message identities and 64 MiB of
current state. Writes stop before a budget is exceeded; there is no silent
truncation.

A completed turn leaves its identity object and becomes an immutable
checksummed history object with a message-ID index object. The history objects
are written first; only then does the identity object drop the turn. Completed
payloads stop counting against the current-state budget.

After the first turn moved to history, **new message admission checks the
message-ID index object**. A repeated message ID is a duplicate while its turn
is kept.

**LBB's own workflow types.** A type that starts with `lbb.` belongs to Little
Big Brain: `lbb.model-checks`, `lbb.search-tuning`, `lbb.model-trials`,
`lbb.calibration`, and every first-party type to come. Only Little Big
Brain's own worker runs them. The workflow routes refuse every other caller
with 403 `reserved_workflow_type`, your stack API key and your console
session included:

- Creating an instance of a reserved type.
- A claim that names a reserved type. The claim is refused whole.
- A worker's write to a turn of a reserved instance (checkpoint, complete,
  fail, operation, progress, heartbeat, wait-signal).

A turn of these types that fails for good never holds the next one. Little
Big Brain marks what it worked on as failed, with the turn's error: the
trial (or one new call of it), the session, the test, or the calls of a
check. Then it skips the turn. To retry the turn yourself, pause the
instance first.

A person can send a message, send a signal, pause, resume, retry, skip, cancel
or delete a reserved instance from the console. A stack API key gets 403
`reserved_workflow_type` for these requests. Every caller can read a reserved
instance, its turns and its history. The ids of the server's own instances
(`model-checks`, `search-tuning`, `model-trials`, `calibration`) are reserved
too: only Little Big Brain's own worker creates an instance with one of them.
A stack API key, a console session and a self-hosted server's owner token get
403 `reserved_workflow_id`, whatever the type.

**Integrations and CDC: `sdk.integration` and `sdk.cdc`.** The workflows of
`@littlebigbrain/integrations` run in your own workers too, so their types
are outside Little Big Brain's names. They were `lbb.integration` and
`lbb.cdc` before 2026-10-13. New connections get the new names. A connection
made by an older release keeps its old name until Little Big Brain moves it;
its state, inbox and history stay, and a sync in progress finishes. The
runners of the package claim both names, so update the package in your own
workers before your connections move: a worker of an older release claims
only the old name and stops finding work.

**History retention.** Each instance keeps its newest 256 turns. It also keeps
every turn that finished in the last 24 hours. Older turns are removed.

- Reading a removed turn answers 404.
- History pages start at the oldest kept turn.
- A message ID whose turn was removed counts as a new message.

Self-hosted servers set the bounds with `LBB_WORKFLOW_HISTORY_KEEP_TURNS` and
`LBB_WORKFLOW_HISTORY_KEEP_MS`. A value of 0 turns keeps everything.

This model addresses history growth through explicit state and bounded replay.
Temporal's [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new)
creates a fresh run/history; its [workflow definition rules](https://docs.temporal.io/workflow-definition)
explain deterministic replay. Here the boundary is every committed message.

### Runs

Each attempt of a turn is a run. The engine keeps a small summary of each
run for 30 days, beside the turns it keeps for 24 hours. A summary has the
run's origin, outcome, error, the inbox before and after, what it finished
(the `done` and `unit` of its last progress), a sample, its steps and
phases, the turn result's `counters`, and the usage of its model calls.

- Name what a turn processed in its result: `sample` (up to 20 names) and
  `counters` (whole numbers by name) go into the run's summary.
- The TypeScript and Python workers name the turn's run and the step on
  each activity call, and add up the calls' usage for the completion. Your
  code does nothing for it.
- A run's page lists the model calls it made.

```sh
curl "$LBB/v1/workflows/named/runs?name=rooms&limit=20" \
  -H "Authorization: Bearer $LBB_API_KEY"
curl "$LBB/v1/workflows/instances/runs?id=agent" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

Each answer is one page in the [list envelope](https://docs.littlebigbrain.com/sdks/http-api/#lists),
newest first: pass `next_cursor` as `cursor` for the next page.
`/runs/get?run=<run id>` returns one run with its model calls, and
`/runs/stats?hours=<n>` the runs by hour. The runs of a named workflow are
under `/v1/workflows/named/runs`; the runs of an instance of no named
workflow under `/v1/workflows/instances/runs`. In the SDKs,
`client.workflows.named.allRuns("rooms")` (TypeScript) and
`client.workflows.named.all_runs("rooms")` (Python) read every page.

### Run results

A run also records what it did with each input. Each result names the
input, its outcome and its values, in the shape a test gives them.

| Outcome | Means |
| --- | --- |
| `written` | The run wrote at least one value. |
| `review` | A value waits for a person. The value names its `review_item`. |
| `no_value` | The run read the input and wrote no value. |
| `failed` | The run could not do the input. `error` says why. |
| `duplicate` | An earlier attempt of the turn wrote the output. |

A value has `property`, `value`, the `quote` that states it, its `support`,
and its `outcome`. A number the text states in another unit has `stated`,
as `600 mm`. A value from a file can name its `source` and page `region`.

LBB's `values.extract`, `documents.parse` and `documents.link` record their
results. A worker records them for you:

- `writeOwned` reports each subject it writes, with the fields and the edges.
- `review` of an item with a `subject` reports that input as `review`.
- `requeue` reports each instance as `failed`.

The worker sends the results with the turn's completion, at most 512 KiB.
A step keeps what it reported, so a turn that runs again reports it too.

:::caution[Roll workers forward only]
A step that reported inputs saves them beside its value. An SDK from before
run results does not take them out again. On a replay, it gives your code
`{lbb_step_results, run, results, value}`, not the step's value. This can
happen after a rollback, or in a fleet of mixed versions. So do not go back
to an earlier SDK while newer turns can still run.
:::

```sh
curl "$LBB/v1/workflows/named/runs/results?name=rooms&outcome=review,failed" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

`run=<run id>` reads one run, with `run_counts` for every input by outcome.
`iri=<input>` reads one input's results. The list pages with `cursor` and
`limit`, newest run first. A run keeps its first 200 inputs whole. Results
are kept 30 days, as the runs are. In the SDKs,
`client.workflows.named.runResults("rooms")` (TypeScript) and
`client.workflows.named.run_results("rooms")` (Python) read one page.

## Safe writes, waits and failing instances

These features let a worker write once per turn, wait without a failure,
and keep a shard going when one instance fails.

- **Fenced commits.** `writeOwned` sends the step's turn and token as
  `workflow_fence` with the commit. A worker whose lease ended gets
  `409 workflow_fence_lost`, and the commit writes nothing. A repeat of the
  step's effect id returns the earlier commit. The effect id is new for each
  message, so a value that changes back (A, then B, then A) is written
  again.
- **Owned properties.** A commit under a turn stays inside the properties
  its workflow lists in `owns.properties`. A property outside them answers
  `403 write_outside_owns`. The IRIs are in `details`, and the commit writes
  nothing. A commit that sets `property_merge: "replace"` is refused too.
  The server may first count such commits and write them as before. Make
  your worker write only the properties in `owns`. With the next SDK
  release, a turn that meets this refusal fails at once. The worker does not
  retry it.
- **Waits.** A step that meets `429 workflow_budget_exceeded` fails the turn
  with `retry_at_ms`. The turn waits until the budget is whole again and
  counts no attempt. Throw `WorkflowWait(retryAtMs)` to wait for another
  reason. Run the workflow now, resume it, or call `handle.wake()` to end
  the wait. LBB's own workers wait the same way when the database is busy;
  such a wait shows `reason: "capacity"`.
- **Failing instances.** Send the instances a turn could not do back to the
  watch with `requeueFailed(client, ctx, "requeue", failed, { graph })`, and
  complete the turn. The watch sends them again after 1, 4 and 16 minutes,
  at most 3 times. With `on_exhausted: "skip"`, a turn that fails for good
  stays in the history, and the next message runs.
- **Budgets.** `budget_usd_per_month` on a workflow limits the cost of its
  model calls. The workflow's status shows its spend for the month.
- **Fan-in.** `fan_in: "catalogue"` adds the identity
  `starter.<name>.catalogue`. The shards send it messages with fixed ids,
  for example `generated-<sha256>-<version>`, so a repeat is one message.
  `triggeredWorkflow({ onFanIn })` handles them.
- **Blobs.** A step output holds at most 64 KiB. Put a larger output in a
  blob (`client.workflows.blobs.putJson(value, { workflowId })`, at most
  8 MiB) and keep its key in the step output. Blobs go with their workflow.

```ts
const failed = [{ instance: iri, error: "the model refused the text" }];
await requeueFailed(client, ctx, "requeue", failed, { graph: message.graph });
return { state, result: { counters: { failed: failed.length } } };
```

## Activity graph HTTP API

All routes use the existing bearer-token/stack/graph scope and limits. Add
`?graph=<graph>` as usual. IDs/names for steps, activities, signals and workers use
1–128 ASCII letters, digits, `_`, `-`, `.`. Run display names may be up to 200
bytes. Times are Unix milliseconds.

| Method and path | Request / behavior |
| --- | --- |
| `POST /v1/workflows` | `{id,name,input?,steps}`; identical ID+definition is idempotent, different definition conflicts |
| `GET /v1/workflows` | `after` exclusive run-ID cursor, `limit` 1–100 (default 50); returns `runs,next_after` ordered by ID |
| `GET /v1/workflows/run?id=…` | Definition, run state, step states/results, attempt and worker |
| `POST /v1/workflows/claim` | `{worker,activities,wait_ms?}`; `task` or null; long poll up to 10 seconds |
| `POST /v1/workflows/complete` | `{run_id,step_id,token,output?,error?,claim_next?}`; returns current run and optional next task |
| `POST /v1/workflows/heartbeat` | `{run_id,step_id,token}`; extends a still-live attempt by its configured timeout |
| `POST /v1/workflows/signal` | `{run_id,name,value}`; buffers a declared signal before its step is reached |
| `POST /v1/workflows/cancel?id=…` | Idempotently cancels an active run and fences its outstanding tokens |
| `GET /v1/workflows/status` | Enabled state, durability (`object_storage`), active runs and the index generation. The backlog fields are always zero |
| `POST /v1/workflows/archive` | Check the scheduling index against the identity objects now and return the status |

A step has `id`, `kind` (`activity`, `timer`, `signal`), `depends_on` (default `[]`),
`name` (activity or signal), optional JSON `input` and `delay_ms` (default 0).
Dependencies must form a DAG. Delay begins when dependencies finish. Activities
receive their input, workflow input, and direct dependency outputs. Timer steps
complete when due; a named signal is a one-shot value shared by steps with that
name. Repeating the same signal/value is idempotent; changing its value conflicts.

Activities default to 3 attempts, 1,000 ms retry delay and 60,000 ms timeout.
`max_attempts` is 1–100, `retry_delay_ms` at most one day, `timeout_ms` 100 ms to one
hour, and `delay_ms` at most 365 days. A final activity failure fails the run and
cancels its remaining steps. Timer polling is 25 ms; due work is processed in
batches of 64 runs. Host clocks must stay synchronized for deadline behavior.

A claimed task has an opaque per-attempt `token` and stable `effect_id`. An expired
or cancelled attempt cannot complete or heartbeat. Delivery is **at least once**;
pass `effect_id` to downstream idempotency facilities. Cancellation cannot undo or
kill code already executing in a worker. Completing with `output: null` succeeds;
provide `error` to fail an attempt. Retry the exact completion body after a lost
response. Its receipt prevents duplicate advancement and returns the same next
task only while that attempt remains valid. `claim_next` only claims this run;
use the normal claim endpoint afterward for other runs.

## Storage, recovery and erasure

Namespace: `tenants/<tenant>/graphs/<graph>/branches/main/workflows/v2/epoch=<e>`
(the internal graph object prefix still contains `branches/main`; `<e>` is the
graph head epoch, which changes when a graph is deleted and created again).
Format v2 is isolated from RDF formats.

| Object | Key under the namespace | Written |
| --- | --- | --- |
| Instance and its current turns | `state/instances/<id>.json` | conditional write per mutation |
| Activity-graph run | `state/runs/<id>.json` | conditional write per mutation |
| Scheduling index | `state/index.json` | conditional write when a schedule changes |
| Turn history | `history/i-<id>/turns/<number>.json` | once, immutable |
| Message-ID index | `history/i-<id>/messages/<hash>.json` | once, immutable |

The identity object is the truth. Every mutation reads it, applies the change
and writes it back on the condition that nobody else wrote in between; a
conflict repeats the step. A claim is such a write, so two nodes cannot assign
one turn. Later calls from the worker are checked against the token and
deadline in that object.

The index lists every identity with its type, status and next ready time.
Workers find work through it. It is a hint: a claim always decides on the
identity object and corrects an index entry that is wrong. A node that serves
waiting claims checks the index once per `LBB_WORKFLOW_POLL_MS`. A periodic
listing (`LBB_WORKFLOW_RECONCILE_MS`) repairs an index that missed an update,
so the worst case after a lost index write is a delay of that interval.

Core state operations per step: a checkpoint or heartbeat is one read and one
write. A claim adds an index write. A completion adds two history writes, one
more identity write and an index write. These are not total storage-request
counts: authority checks, scheduling records and selected-profile admission
metadata add operations.

Limits per graph are 64 steps/run, 64 KiB definition, 256 KiB stored run or
turn, 1,000 combined identities/runs and 64 MiB of current state. The identity
limit and the state budget are checked against the index, so concurrent
creations can pass them by a few. Terminal runs are retained until graph
deletion; there is no per-run retention API yet.

There is nothing to recover after a node is lost or replaced: the next request
reads the objects from any node. Timers and expired leases are applied by the
next claim or read, so they do not depend on a process that stays up.

Graph deletion immediately retires the graph lifetime. Ordinary workflow state
and history are removed in that request. In an administratively selected
environment, maintenance workers complete workflow cleanup; the response's
`workflow_cleanup.pending` and `workflow_cleanup.job_ids` report that work.
Environment and account erasure also wait for the required workflow cleanup.
Selected cleanup retains scrubbed barriers that prevent delayed writes from
restoring deleted payload. A graph created again starts with a new epoch and no
workflow state. Erasing workflows does not undo external effects.

## Optional database reads from a worker

With workflow execution reads and batch admission enabled on the server, a
TypeScript `WorkflowWorker` can opt in with `executionReads: true`. Its optional
`ctx.sparql` method runs bounded `SELECT` or `ASK` reads on the claimed graph;
check that the method exists when accepting older context implementations.
Python exposes `client.workflow_turn_reader(task)` for sync and async clients.
Continue heartbeating, checkpointing and completing through the control client.
Put a read inside a named checkpoint to reuse its acknowledged result on replay.

The reader refreshes before first use and when its remaining lifetime cannot
cover the requested read budget plus a margin. It uses server-supplied lifetime
and monotonic elapsed time, including refresh transit, rather than comparing
worker and server wall clocks. A locally exhausted lifetime is retryable; it
never silently replays a query or falls back to the environment API key.

An invalid or already-expired bearer returns `401`; a lost workflow lease or a
credential that expires during a read returns `403`. Batch saturation returns
`429 batch_database_busy`; billing refusal is `402 batch_billing_quota_exceeded`.
Waiting work is bounded per owner and environment as well as globally. These
controls use the existing workflows and workers and do not autoscale the worker
fleet. Both server gates remain off until the deployment is qualified.

## Validation and further work

`lbb-workflows/tests` covers lease fencing across two nodes, idempotent
handoff, timers and signals, message deduplication across history, an index
that missed an update, and an interrupted move to history.
`lbb-sim/tests/integration/workflow_object_store.rs` injects failed operations, lost PUT
responses and applied-then-conflict responses into two interleaved nodes and
checks turn numbering, single holders, state, history and the index. HTTP and
console tests cover the product flow.

The design note and the performance ledger in the repository record the
protocol and the handoff cost. Future work includes a per-node cache of identity objects that removes
the read of most steps, ranged or batched history reads, configurable
retention and state migrations. Child workflows and arbitrary-code stack
snapshots are not implemented.

## Operator recovery modes

Release 3 adds an experimental `LBB_MODEL_WORKFLOW_SCHEDULING` setting for the
existing model-workflow maintenance job: `off` (default), `observe`, or
`recover`. Configured producers for `lbb.model-checks`, `lbb.search-tuning`,
`lbb.model-trials` and `lbb.calibration`, at application version `v1`, publish
durable scheduling records before activating work. Current production writers
use their canonical IDs: `model-checks`, `search-tuning`, `model-trials` and
`calibration`. Creation is idle; the first message publishes the recovery record.
Recovery checks the current source and claim before execution.
Existing environments retain ordinary recovery. Selected new environments use
the profile described below. Other workflow types and versions keep their
existing worker behavior and are outside this advertised recovery mode.

Waiting work and held leases remain recoverable after a delivery disappears,
without another incoming message. Canonical next-wake probes remain available;
Observe/Recover adds a bounded, read-only lookup through the workflow index.
These controls do not provide a fleet recovery-latency or scaling guarantee.

Noncanonical model IDs require the separate
`LBB_MODEL_WORKFLOW_ARBITRARY_PUBLICATION` gate, which defaults to false and
also requires Observe/Recover. The gate alone does not certify complete recovery
coverage. After qualifying compatible writers, readers and erasers, a service
administrator can create a new environment with the selected model profile.
This profile covers the four `v1` types above, including noncanonical IDs, and
requires Recover mode. Its construction proof permits skipping the model
recovery scan of graphs and sources while advertised recovery continues. It does
not adopt existing environments or change other worker families. Global
environment and active-job inventories remain, so this is not zero idle I/O.
These controls are implemented locally; production activation remains an
operator rollout decision. Once scheduling or selection records exist, turning
a gate off does not remove them or make an older writer or purge worker safe to
restore.

For these configured producers, a source proved retired or replaced returns
`400 workflow_scheduling_not_live`; unavailable authority, including a missing
environment reverse locator, returns a retryable `503`. Empty advertisement directories need one bounded presence list, while
ordinary recovery and local orphan checks still run in unselected environments.
This does not make idle graph populations free of storage requests. After scheduling records are written,
keep compatible producers, readers and purge workers during upgrade or rollback;
switching the setting to `off` does not make an older binary safe. Tenant deletion
uses bounded cleanup pages and checkpoints. Inconclusive shared records are
retained with a repair diagnostic and increasing worker retry delay; large or
blocked inventories can still need more than one cleanup pass.

Service administrators can explicitly submit bounded historical repair through
`POST /api/admin/workflows/historical-discovery` with `campaign_id` and
`request_id`, and inspect that campaign with GET on the same path. Each POST
requests one maintenance quantum; it does not automatically continue the whole
inventory. This campaign still selects the four canonical IDs; arbitrary-ID
historical sources are not automatically seeded. Completed traversal still
reports `coverage: false`. A historical campaign does not provide the
construction proof used by selected new environments, and does not retire the
legacy recovery scans for an existing environment.