Workflows (experimental)

Workflows are opt-in. A server enables them with --workflows.

To start from one of LBB’s templates, see Build a workflow. It tests a workflow on two records, then creates it.

LBB provides message workflows with a stable identity, explicit saved JSON state, and a durable inbox. A normal TypeScript or Python handler runs one bounded turn per message; named step outputs are saved as checkpoints. State, reply, inbox advancement and an optional continuation commit atomically. After a crash, only the unfinished turn replays. Every completed message automatically starts a fresh history boundary.

The original declarative activity DAG remains available under Activity graphs. This is an experimental engine with its own programming model, not a Temporal SDK.

Durability: an acknowledged request is saved in object storage. Workflow state has the same truth contract as the graph. Every serving node reads and writes the same objects, and no node keeps workflow state of its own. A lost node loses no acknowledged work.

Enable and try it

Start the server with --workflows. Workflows need no directory and no coordinator.

cargo run -p lbb-server-bin -- \
  --root /private/tmp/lbb-workflow-objects --tenant acme --graph main \
  --bind 127.0.0.1:7400 --api-token lbb_sk_test_local \
  --workflows

# In another terminal (create the graph if it does not already exist):
curl -H 'Authorization: Bearer lbb_sk_test_local' -X POST \
  'http://127.0.0.1:7400/v1/graph/create?graph=main'
curl -H 'Authorization: Bearer lbb_sk_test_local' -H 'Content-Type: application/json' \
  --data-binary @database/examples/workflows/approval.json \
  'http://127.0.0.1:7400/v1/workflows?graph=main'
LBB_API_KEY=lbb_sk_test_local python3 database/examples/workflows/worker.py

Environment variables: LBB_WORKFLOWS_ENABLED=true on every serving node; LBB_WORKFLOW_POLL_MS (default 1000) and LBB_WORKFLOW_RECONCILE_MS (default 30000) tune how fast a node sees work written by another node. Workflow requests are served by the node that receives them and skip graph affinity, because every node reads the same objects. Node clocks must be synchronized: leases compare against the clock of the node that handles the request.

The Workflows page

Open Operate → Workflows in the console. When you run the console locally, start it with VITE_LBB_FEATURE_FLAGS=workflow-starters,nvme-workflows. The page lists every workflow of the graph, grouped by job. The list includes the workflows of the templates your worker runs on files (rows, sections). New workflow creates a workflow from a template, a message workflow, or an activity graph from its JSON definition. The words on these pages are defined in the glossary.

A workflow’s page has these tabs. documents.link and documents.map also have Documents. search.embed and ontology.fit have no Runs tab.

Tab	Shows
Overview	The status, the lag, the waiting work and this month’s spend. A workflow your worker runs can show a No worker warning. It shows when a message has been ready for two minutes and no turn runs. It gives the command that starts a worker.
Results	What the workflow did: per file, per document, or the values it fills.
Review	Its review items per status, with a link to the review inbox kept to the workflow. Today the inbox is the Suggestions tab of the knowledge graph.
Quality	Its gate, the gate’s recent calibration sessions, the model checks of its uses, and its open review items. It links to Evals. A workflow your worker runs has no Quality tab.
Spend	Its model calls this month per use and model, its monthly budget, and links to Models → Spend & limits. For a workflow your worker runs, the calls of every worker on the graph, because they are not yet counted per workflow.
Runs	Each run with its steps and model calls.
Settings	Edit the watched fields, the parameters, the gate and, for a workflow your worker runs, its target, owned outputs and batch. documents.link, documents.map and values.extract also take a monthly budget. A template that fixes what it watches keeps its class and fields. The page shows what changes before it saves.

The engine details (the instance, the shards and the inbox) are under a Details disclosure.

A row whose running turn reports progress in the progress convention shows a Running line: the phase, “40 of 97 pages”, how long it runs, and a bar. An engine instance’s page sends a message, pauses, resumes or deletes the workflow. It shows the Running line of the current turn, the Queue (the turn that runs and the messages that wait, oldest or newest first, each with its origin, age and size; a message’s payload is read when you open it), and Turns, which lists only completed and cancelled turns newest first, a page at a time. Queued, running, waiting and failed turns stay in Queue; they do not appear again in Turns. History pages count finished turns, so pending messages do not displace them. A turn opens its steps, its message, its result, the state it left, and the operation in progress. The current turn takes a signal, a retry or a cancel. LBB’s own model workflows show the same way. An activity graph’s page shows each step with its state, sends a signal and cancels the run. Run the DAG example above and send approved with JSON value true to finish it. Run the worker in a separate terminal; the console does not execute activity code.

Message workflows and automatic rollover

import { LbbClient, workflow, WorkflowWorker } from "@littlebigbrain/client";
const client = new LbbClient({ baseUrl: "http://127.0.0.1:7400", apiKey: "lbb_sk_test_local" });
const counter = workflow({
  name: "counter", version: "v1", initialState: { count: 0 },
  async onMessage(ctx, state, message: { add: number; remaining?: number }) {
    const amount = await ctx.step("add", async () => message.add);
    const next = { count: state.count + amount };
    if ((message.remaining ?? 0) > 1)
      return ctx.continue(next, { ...message, remaining: message.remaining! - 1 });
    return { state: next, result: next.count };
  },
});
const handle = await counter.start(client, "agent-1");
await handle.send({ add: 1, remaining: 3 }, { id: "request-1" });
const stop = new AbortController();
await new WorkflowWorker(client, [counter], { worker: "worker-1" }).run({ signal: stop.signal });

A runnable worker is database/examples/workflows/counter.mjs. Build the SDK first with npm run build -w @littlebigbrain/client; set LBB_WORKFLOW_START=1 to create and seed its example, or create it in the console and send {"add":1} there.

ctx.step(key, async ({effectId, signal}) => ...) saves its JSON result before proceeding. Put all I/O, clock reads and randomness inside steps. Pass effectId to the downstream service’s idempotency facility: the effect can succeed before its checkpoint, so delivery remains at least once. The key is stable across attempts and recovery, scoped to this namespace/identity/message/step.

ctx.sleep(key, milliseconds) saves a durable timer and releases the worker. When due, the handler restarts with saved step results and skips the elapsed sleep. Await steps and sleeps sequentially; parallel or unawaited steps are rejected. Keys must be unique within the turn. Return JSON values (including null); do not return undefined from a step. Cancellation is cooperative through signal; remote fencing cannot undo external effects.

Unlike a long replay history, the next message starts with saved state and zero steps. There is no user-managed run chain or mandatory Continue-As-New ceremony. For one large batch, return ctx.continue(state, nextMessage) before the turn’s 64-step/256-KiB budget; its successor is appended behind messages already queued. State, reply and successor are committed together. A failed turn blocks the inbox until retry or cancellation; it cannot partially publish new state. A paused workflow accepts messages and allows its already leased turn to finish, but takes no new claims. Cancellation targets an exact turn and leaves saved state unchanged.

Code versions are pinned for an identity. Keep old worker versions available; a differently registered version cannot claim its work. Within a version, changing step order/keys or omitting saved steps is a permanent SDK replay error. This does not snapshot arbitrary JavaScript stacks or infer state migrations. Timers block that identity’s inbox until due; run independently interruptible work as separate identities. The SDK renews leases and abandons fenced attempts; claim transport errors can stop the worker loop, so supervise and restart workers. Ordinary handler errors retry; WorkflowError fails permanently.

LLM runs, signals and progress

On a workflow’s page in the console, Send message queues another turn on the workflow identity. It does not send a chat message to a model and cannot interrupt an existing turn. Use Respond on a turn that waits for a signal, or Send a signal on the current turn, for approvals, tool callbacks and other events an existing turn needs. The SDK keeps handle.send(input, {id}) for inbox admission and adds handle.signal(name, value, {turn, id}) for this distinct operation.

database/examples/workflows/agent-review.mjs is a runnable draft → human review → delivery example. Build the SDK, then run it with LBB_API_KEY and LBB_WORKFLOW_START=1. Without adapter URLs it uses labelled local demo results. Set MODEL_URL and TOOL_URL to your own JSON HTTP adapters to run real operations; provider credentials and provider-specific request mapping belong in those adapters. The console’s New workflow can create agent-review@v1 with initial state {}; then send the message {"task":"Draft a customer response"}. Read the saved draft in the turn’s steps, then respond to approval with {"approved":true} or {"approved":false}.

const review = await ctx.waitForSignal<{ approved: boolean }>("review", {
  name: "approval", timeoutMs: 86_400_000,
});
if (!review.received) return { state, result: "approval timed out" };
if (!review.value.approved) return { state, result: "rejected" };

// From a UI, webhook handler or another authenticated client:
await handle.signal("approval", { approved: true }, { turn: 7, id: "review-event-123" });

Signals target a specific turn; delayed callbacks never apply to a successor. Distinct event IDs permit repeated signals of the same name. Exact retries return the original receipt, including after the turn moved to history; a reused ID with a different name/value conflicts. A turn buffers up to 64 signals, each at most 32 KiB, within the existing 256 KiB turn budget. New signals to completed/cancelled turns conflict. Signals to queued, paused or failed runs buffer until they can run.

A signal wait consumes the oldest unconsumed matching signal in the same conditional write as the checkpoint. If none exists, it releases the worker until a matching signal or timeout. It resumes the same attempt, replaying saved results. Signal receipt strictly before the deadline wins; a signal at/after the deadline stays buffered for a subsequent wait. A timed-out wait returns {received:false,value:null}; JSON null is still a valid received value when received:true. Timeouts are 1 ms–365 days. Signals do not execute concurrent handlers or interrupt a sleeping/running step: the handler explicitly awaits them. Use cancellation for interruption. Targeting a turn is intentional; signals are not automatically transferred across continuations. Unconsumed events remain in that turn’s history, so consume relevant signals before returning.

For long LLM or tool calls, configure a step independently of the worker lease:

const pages = await ctx.step("fetch-documents", async ({ checkpoint, heartbeat, signal, effectId }) => {
  // Your tool can resume from this saved cursor after a retry.
  let cursor = checkpoint;
  for await (const page of fetchPages({ cursor, signal, effectId })) {
    cursor = page.nextCursor;
    await heartbeat({ pages: page.count }, { checkpoint: cursor });
  }
  return { cursor }; // Completed result is checkpointed once.
}, { kind: "tool", timeoutMs: 300_000, heartbeatTimeoutMs: 30_000 });

kind (model, tool, step) is operation metadata, not a provider adapter. Every SDK step has a per-attempt timeout (default 5 minutes; 100 ms–24 hours). Optional heartbeatTimeoutMs requires explicit operation progress within that interval (at least 100 ms, no longer than the execution timeout). Automatic lease renewal only reports worker liveness and extends neither deadline. A hung model request therefore times out even while its worker is alive. Do not enable a short progress timeout for a non-streaming provider that cannot report progress; use its execution timeout instead. Operation failures use the workflow’s attempt limit/backoff and replay prior completed steps.

Progress heartbeats replace the current details (8 KiB) and resume checkpoint (32 KiB). They use increasing sequence numbers per attempt; exact retries do not extend deadlines. The next attempt receives the last checkpoint through the step context. A failed/retrying turn exposes its last operation; completing its step clears active progress. The API/console show operation kind/key, attempt, deadline, last progress heartbeat, details and cursor separately from worker lease status.

The SDK propagates AbortSignal and stops waiting when a lease, operation or progress timeout is lost, even if a provider ignores cancellation. The server fences stale checkpoint/completion writes and schedules retries. Cancellation does not roll back an external tool effect or guarantee that a provider stops billing; pass the effect ID to an adapter that actually enforces idempotency. Cancelling a run is observed on the worker’s next lease renewal (normally every lease/3).

Keep token streaming on a separate ephemeral channel. Coalesce progress updates; do not persist every token or full conversation in heartbeat payloads. Progress replaces the current value, and each acknowledged change is one object write. Large transcripts/results belong in referenced objects. This release does not add provider integrations, token/cost enforcement, streaming transport, child workflows or parallel steps.

Progress convention

Operation progress details follow one shape when a workflow wants clients to draw them. The console draws them as the Running line and a bar, for any workflow.

{"phase": "reading pages", "done": 40, "total": 97, "unit": "pages"}

Field	Rule
phase	Plain words, 1 to 80 characters. Required.
done	A whole number, at least 0. Required.
total	A whole number, or absent when the step does not know it.
unit	documents, pages, mentions, links, proposals, instances, questions, messages, items, or another short word (1 to 32 letters, digits, spaces, - or _). Optional.

Details with a phase key follow the convention. Other keys may sit beside the four fields. The server refuses details with a phase key that break a rule (400). Details without a phase key stay free-form.

The turn and instance views carry a normalized progress: phase, done, total, unit, started_at (when the operation started) and heartbeat_at (the last progress write), both RFC 3339. A turn’s progress reads its operation; a failed or retrying turn keeps its last progress. An instance’s progress is set only while its current turn runs. The views fill it on read; the stored turn does not hold it. Clients read progress and never parse details.

The TypeScript SDK reports it with progress in the step context:

await ctx.step("read", async ({ progress }) => {
  for (let page = 1; page <= pages.length; page++) {
    progress("reading pages", { done: page, total: pages.length, unit: "pages" });
    await readPage(pages[page - 1]);
  }
});

progress is coalesced. A phase change is sent at once. Other values are sent at most once per 3 s, or once per half the step’s heartbeatTimeoutMs when that is shorter (worker option progressEveryMs). The last value wins and is sent when the step returns. Each call keeps the resume checkpoint of the last heartbeat. A value outside the convention fails the step for good. The Python worker reports progress the same way (Run a worker in Python).

Message origins

A message can say where it came from. POST /v1/workflows/instances/message takes an optional origin, at most 1 KiB:

{"workflow_id": "agent", "id": "m-412", "message": {"document": "report.pdf"},
 "origin": {"kind": "commit", "from_seq": 409, "to_seq": 412,
            "fields": ["https://littlebigbrain.com/p/file_sha256"],
            "requests": ["file-upload:3fa2"]}}

Field	Meaning
kind	Required. app (your code), console (a person on the console), manual (Run now), watch, backfill, commit, schema_changed, requeue, migration (a watch), schedule, event (an event of the data plane), signal (a callback)
trigger, reason, event	Identifiers, 1 to 128 ASCII letters, digits, _, - or .
from_seq, to_seq, target_seq	Commit sequences: the range that caused it, or the commit a pass reads up to
fields, requests	At most 32 entries each: the changed fields, and the request ids of the commits
actor	Who asked, at most 128 characters
attempt	How many times the item was sent before
note	A short note, at most 280 characters

One origin covers one message. A message of several instances holds the union of their fields. The engine stores the origin with the turn. The turn, the inbox and the worker’s claimed task show it; the SDK hands it to the handler as ctx.origin. The origin is no part of the message’s identity: a repeated message id with the same message answers the first turn, with the first origin.

A continuation gets {"kind": "continuation", "turn": <n>, "message_id": <id>}, the turn that continued. A sender cannot set the kind continuation. A developer workflow’s trigger messages name their origin from their reason: backfill with target_seq, commit with to_seq, manual for a refresh, schema_changed. A CDC wake is {"kind": "event", "event": "cdc_capture"}. The console’s Send message sends console.

Inbox

GET /v1/workflows/instances/inbox?id=<id> lists the unfinished turns of one instance, oldest first: the turn that runs or runs next, then the queued messages. Each entry has turn, message_id, status (queued, waiting, running, failed), received_at_ms, ready_at_ms, attempt, message_bytes, origin, error and the running turn’s progress. The page also has the instance’s type, version, status, pending_messages, and the oldest_received_at_ms and newest_received_at_ms of every unfinished turn. after is a turn-number cursor, limit is 1 to 100 (default 50), and payloads=true adds each message. It is one read of the instance object.

GET /v1/workflows/inbox?workflow_type=<type>&version=<v> lists the instances of a type (of every version without version) that have unfinished turns, by id: each with its pending_messages, oldest_received_at_ms and inbox head (no payload). identities counts the instances of the type in the namespace index, with_messages those the index lists with a turn. The index lags a write of another node by at most its poll interval, so a listed instance whose inbox emptied since is left out. after is an instance-id cursor, limit is 1 to 100 (default 50), and each listed instance is one read.

Every caller can read both, the reserved lbb.* types included.

Message API

All requests inherit graph/stack authentication. Message IDs are required. While a turn is in the kept history, an identical retry returns that turn. A changed payload conflicts. The continue- prefix is reserved.

Method and path	Behavior
POST /v1/workflows/instances	{id,workflow_type,version,state,lease_ms?,max_attempts?}; identical creation retries are safe
GET /v1/workflows/instances	after ID cursor and limit; returns instances,next_after
GET /v1/workflows/instances/get?id=…	Saved state, status, pending count, completed and history counters, history_pruned_through, current turn, and the running turn’s progress
POST /v1/workflows/instances/message	{workflow_id,id,message,origin?}; durable inbox admission; origin per Message origins
GET /v1/workflows/instances/inbox?id=…	after turn-number cursor, limit 1–100, payloads; the unfinished turns with their origins (Inbox)
GET /v1/workflows/inbox?workflow_type=…	version, after id cursor, limit 1–100; the instances of a type with unfinished turns (Inbox)
POST /v1/workflows/instances/control	{workflow_id,action,turn?}; action is pause, resume, retry (a failed turn), skip (a failed turn), cancel_turn, or wake. retry and skip need turn. wake needs no turn: it moves a turn that waits without an attempt (a retry_at_ms failure or a capacity deferral) to now, and changes nothing when no turn waits so. A step’s sleep or signal wait stays
POST /v1/workflows/instances/delete	{workflow_id}; deletes the instance, its turns and its history. {deleted} is false when no instance had the id. The id can be created again after successful deletion. In a selected environment, 503 workflow_source_closure_pending means cleanup is continuing: retry the same delete after Retry-After
GET /v1/workflows/instances/history?id=…	after turn-number cursor, limit 1–50; turns,next_after from the oldest kept turn
GET /v1/workflows/instances/turn?id=…&turn=…	One turn and its saved steps, reply and state after completion; 404 for a removed turn
POST /v1/workflows/turns/claim	{worker,workflows:[{workflow_type,version}],wait_ms?}; only the inbox head may run
POST /v1/workflows/turns/checkpoint	{workflow_id,turn,token,position,key,kind,output?,delay_ms?}; ordered step or sleep
POST /v1/workflows/turns/complete	{workflow_id,turn,token,state,result?,continuation?:{message},usage?}; exact retries work while the turn is kept. usage is what the turn’s model calls used; the TypeScript SDK sends it
POST /v1/workflows/turns/fail	{workflow_id,turn,token,error,non_retryable?,usage?}
POST /v1/workflows/turns/signal	{workflow_id,turn,id,name,value}; buffered, idempotent event targeting one turn
POST /v1/workflows/turns/wait-signal	{workflow_id,turn,token,position,key,name,timeout_ms}; consume or suspend
POST /v1/workflows/turns/operation	{workflow_id,turn,token,position,key,kind,timeout_ms,heartbeat_timeout_ms?}; start/resume step
POST /v1/workflows/turns/progress	{workflow_id,turn,token,key,sequence,details,checkpoint}; operation progress, independent of lease; details with a phase key follow the progress convention
POST /v1/workflows/turns/heartbeat	{workflow_id,turn,token}; renew a live lease

Defaults: three attempts and a 60-second lease. Failure backoff doubles from one second to 60 seconds; durable sleep does not consume an additional attempt. Limits: 64 sequential steps, 64 KiB saved state/output/reply, 32 KiB message, 256 KiB stored turn, 128 pending messages per identity. IDs/type/version/step keys use the ASCII identifier rules below. Large payloads belong in referenced objects. The combined graph budget is 1,000 DAG runs/message identities and 64 MiB of current state. Writes stop before a budget is exceeded; there is no silent truncation.

A completed turn leaves its identity object and becomes an immutable checksummed history object with a message-ID index object. The history objects are written first; only then does the identity object drop the turn. Completed payloads stop counting against the current-state budget.

After the first turn moved to history, new message admission checks the message-ID index object. A repeated message ID is a duplicate while its turn is kept.

LBB’s own workflow types. A type that starts with lbb. belongs to Little Big Brain: lbb.model-checks, lbb.search-tuning, lbb.model-trials, lbb.calibration, and every first-party type to come. Only Little Big Brain’s own worker runs them. The workflow routes refuse every other caller with 403 reserved_workflow_type, your stack API key and your console session included:

• Creating an instance of a reserved type.
• A claim that names a reserved type. The claim is refused whole.
• A worker’s write to a turn of a reserved instance (checkpoint, complete, fail, operation, progress, heartbeat, wait-signal).

A turn of these types that fails for good never holds the next one. Little Big Brain marks what it worked on as failed, with the turn’s error: the trial (or one new call of it), the session, the test, or the calls of a check. Then it skips the turn. To retry the turn yourself, pause the instance first.

A person can send a message, send a signal, pause, resume, retry, skip, cancel or delete a reserved instance from the console. A stack API key gets 403 reserved_workflow_type for these requests. Every caller can read a reserved instance, its turns and its history. The ids of the server’s own instances (model-checks, search-tuning, model-trials, calibration) are reserved too: only Little Big Brain’s own worker creates an instance with one of them. A stack API key, a console session and a self-hosted server’s owner token get 403 reserved_workflow_id, whatever the type.

Integrations and CDC: sdk.integration and sdk.cdc. The workflows of @littlebigbrain/integrations run in your own workers too, so their types are outside Little Big Brain’s names. They were lbb.integration and lbb.cdc before 2026-10-13. New connections get the new names. A connection made by an older release keeps its old name until Little Big Brain moves it; its state, inbox and history stay, and a sync in progress finishes. The runners of the package claim both names, so update the package in your own workers before your connections move: a worker of an older release claims only the old name and stops finding work.

History retention. Each instance keeps its newest 256 turns. It also keeps every turn that finished in the last 24 hours. Older turns are removed.

• Reading a removed turn answers 404.
• History pages start at the oldest kept turn.
• A message ID whose turn was removed counts as a new message.

Self-hosted servers set the bounds with LBB_WORKFLOW_HISTORY_KEEP_TURNS and LBB_WORKFLOW_HISTORY_KEEP_MS. A value of 0 turns keeps everything.

This model addresses history growth through explicit state and bounded replay. Temporal’s Continue-As-New creates a fresh run/history; its workflow definition rules explain deterministic replay. Here the boundary is every committed message.

Runs

Each attempt of a turn is a run. The engine keeps a small summary of each run for 30 days, beside the turns it keeps for 24 hours. A summary has the run’s origin, outcome, error, the inbox before and after, what it finished (the done and unit of its last progress), a sample, its steps and phases, the turn result’s counters, and the usage of its model calls.

• Name what a turn processed in its result: sample (up to 20 names) and counters (whole numbers by name) go into the run’s summary.
• The TypeScript and Python workers name the turn’s run and the step on each activity call, and add up the calls’ usage for the completion. Your code does nothing for it.
• A run’s page lists the model calls it made.

curl "$LBB/v1/workflows/named/runs?name=rooms&limit=20" \
  -H "Authorization: Bearer $LBB_API_KEY"
curl "$LBB/v1/workflows/instances/runs?id=agent" \
  -H "Authorization: Bearer $LBB_API_KEY"

Each answer is one page in the list envelope, newest first: pass next_cursor as cursor for the next page. /runs/get?run=<run id> returns one run with its model calls, and /runs/stats?hours=<n> the runs by hour. The runs of a named workflow are under /v1/workflows/named/runs; the runs of an instance of no named workflow under /v1/workflows/instances/runs. In the SDKs, client.workflows.named.allRuns("rooms") (TypeScript) and client.workflows.named.all_runs("rooms") (Python) read every page.

Run results

A run also records what it did with each input. Each result names the input, its outcome and its values, in the shape a test gives them.

Outcome	Means
written	The run wrote at least one value.
review	A value waits for a person. The value names its review_item.
no_value	The run read the input and wrote no value.
failed	The run could not do the input. error says why.
duplicate	An earlier attempt of the turn wrote the output.

A value has property, value, the quote that states it, its support, and its outcome. A number the text states in another unit has stated, as 600 mm. A value from a file can name its source and page region.

LBB’s values.extract, documents.parse and documents.link record their results. A worker records them for you:

• writeOwned reports each subject it writes, with the fields and the edges.
• review of an item with a subject reports that input as review.
• requeue reports each instance as failed.

The worker sends the results with the turn’s completion, at most 512 KiB. A step keeps what it reported, so a turn that runs again reports it too.

Roll workers forward only

A step that reported inputs saves them beside its value. An SDK from before run results does not take them out again. On a replay, it gives your code {lbb_step_results, run, results, value}, not the step’s value. This can happen after a rollback, or in a fleet of mixed versions. So do not go back to an earlier SDK while newer turns can still run.

curl "$LBB/v1/workflows/named/runs/results?name=rooms&outcome=review,failed" \
  -H "Authorization: Bearer $LBB_API_KEY"

run=<run id> reads one run, with run_counts for every input by outcome. iri=<input> reads one input’s results. The list pages with cursor and limit, newest run first. A run keeps its first 200 inputs whole. Results are kept 30 days, as the runs are. In the SDKs, client.workflows.named.runResults("rooms") (TypeScript) and client.workflows.named.run_results("rooms") (Python) read one page.

Safe writes, waits and failing instances

These features let a worker write once per turn, wait without a failure, and keep a shard going when one instance fails.

• Fenced commits. writeOwned sends the step’s turn and token as workflow_fence with the commit. A worker whose lease ended gets 409 workflow_fence_lost, and the commit writes nothing. A repeat of the step’s effect id returns the earlier commit. The effect id is new for each message, so a value that changes back (A, then B, then A) is written again.
• Owned properties. A commit under a turn stays inside the properties its workflow lists in owns.properties. A property outside them answers 403 write_outside_owns. The IRIs are in details, and the commit writes nothing. A commit that sets property_merge: "replace" is refused too. The server may first count such commits and write them as before. Make your worker write only the properties in owns. With the next SDK release, a turn that meets this refusal fails at once. The worker does not retry it.
• Waits. A step that meets 429 workflow_budget_exceeded fails the turn with retry_at_ms. The turn waits until the budget is whole again and counts no attempt. Throw WorkflowWait(retryAtMs) to wait for another reason. Run the workflow now, resume it, or call handle.wake() to end the wait. LBB’s own workers wait the same way when the database is busy; such a wait shows reason: "capacity".
• Failing instances. Send the instances a turn could not do back to the watch with requeueFailed(client, ctx, "requeue", failed, { graph }), and complete the turn. The watch sends them again after 1, 4 and 16 minutes, at most 3 times. With on_exhausted: "skip", a turn that fails for good stays in the history, and the next message runs.
• Budgets. budget_usd_per_month on a workflow limits the cost of its model calls. The workflow’s status shows its spend for the month.
• Fan-in. fan_in: "catalogue" adds the identity starter.<name>.catalogue. The shards send it messages with fixed ids, for example generated-<sha256>-<version>, so a repeat is one message. triggeredWorkflow({ onFanIn }) handles them.
• Blobs. A step output holds at most 64 KiB. Put a larger output in a blob (client.workflows.blobs.putJson(value, { workflowId }), at most 8 MiB) and keep its key in the step output. Blobs go with their workflow.

const failed = [{ instance: iri, error: "the model refused the text" }];
await requeueFailed(client, ctx, "requeue", failed, { graph: message.graph });
return { state, result: { counters: { failed: failed.length } } };

Activity graph HTTP API

All routes use the existing bearer-token/stack/graph scope and limits. Add ?graph=<graph> as usual. IDs/names for steps, activities, signals and workers use 1–128 ASCII letters, digits, _, -, .. Run display names may be up to 200 bytes. Times are Unix milliseconds.

Method and path	Request / behavior
POST /v1/workflows	{id,name,input?,steps}; identical ID+definition is idempotent, different definition conflicts
GET /v1/workflows	after exclusive run-ID cursor, limit 1–100 (default 50); returns runs,next_after ordered by ID
GET /v1/workflows/run?id=…	Definition, run state, step states/results, attempt and worker
POST /v1/workflows/claim	{worker,activities,wait_ms?}; task or null; long poll up to 10 seconds
POST /v1/workflows/complete	{run_id,step_id,token,output?,error?,claim_next?}; returns current run and optional next task
POST /v1/workflows/heartbeat	{run_id,step_id,token}; extends a still-live attempt by its configured timeout
POST /v1/workflows/signal	{run_id,name,value}; buffers a declared signal before its step is reached
POST /v1/workflows/cancel?id=…	Idempotently cancels an active run and fences its outstanding tokens
GET /v1/workflows/status	Enabled state, durability (object_storage), active runs and the index generation. The backlog fields are always zero
POST /v1/workflows/archive	Check the scheduling index against the identity objects now and return the status

A step has id, kind (activity, timer, signal), depends_on (default []), name (activity or signal), optional JSON input and delay_ms (default 0). Dependencies must form a DAG. Delay begins when dependencies finish. Activities receive their input, workflow input, and direct dependency outputs. Timer steps complete when due; a named signal is a one-shot value shared by steps with that name. Repeating the same signal/value is idempotent; changing its value conflicts.

Activities default to 3 attempts, 1,000 ms retry delay and 60,000 ms timeout. max_attempts is 1–100, retry_delay_ms at most one day, timeout_ms 100 ms to one hour, and delay_ms at most 365 days. A final activity failure fails the run and cancels its remaining steps. Timer polling is 25 ms; due work is processed in batches of 64 runs. Host clocks must stay synchronized for deadline behavior.

A claimed task has an opaque per-attempt token and stable effect_id. An expired or cancelled attempt cannot complete or heartbeat. Delivery is at least once; pass effect_id to downstream idempotency facilities. Cancellation cannot undo or kill code already executing in a worker. Completing with output: null succeeds; provide error to fail an attempt. Retry the exact completion body after a lost response. Its receipt prevents duplicate advancement and returns the same next task only while that attempt remains valid. claim_next only claims this run; use the normal claim endpoint afterward for other runs.

Storage, recovery and erasure

Namespace: tenants/<tenant>/graphs/<graph>/branches/main/workflows/v2/epoch=<e> (the internal graph object prefix still contains branches/main; <e> is the graph head epoch, which changes when a graph is deleted and created again). Format v2 is isolated from RDF formats.

Object	Key under the namespace	Written
Instance and its current turns	state/instances/<id>.json	conditional write per mutation
Activity-graph run	state/runs/<id>.json	conditional write per mutation
Scheduling index	state/index.json	conditional write when a schedule changes
Turn history	history/i-<id>/turns/<number>.json	once, immutable
Message-ID index	history/i-<id>/messages/<hash>.json	once, immutable

The identity object is the truth. Every mutation reads it, applies the change and writes it back on the condition that nobody else wrote in between; a conflict repeats the step. A claim is such a write, so two nodes cannot assign one turn. Later calls from the worker are checked against the token and deadline in that object.

The index lists every identity with its type, status and next ready time. Workers find work through it. It is a hint: a claim always decides on the identity object and corrects an index entry that is wrong. A node that serves waiting claims checks the index once per LBB_WORKFLOW_POLL_MS. A periodic listing (LBB_WORKFLOW_RECONCILE_MS) repairs an index that missed an update, so the worst case after a lost index write is a delay of that interval.

Core state operations per step: a checkpoint or heartbeat is one read and one write. A claim adds an index write. A completion adds two history writes, one more identity write and an index write. These are not total storage-request counts: authority checks, scheduling records and selected-profile admission metadata add operations.

Limits per graph are 64 steps/run, 64 KiB definition, 256 KiB stored run or turn, 1,000 combined identities/runs and 64 MiB of current state. The identity limit and the state budget are checked against the index, so concurrent creations can pass them by a few. Terminal runs are retained until graph deletion; there is no per-run retention API yet.

There is nothing to recover after a node is lost or replaced: the next request reads the objects from any node. Timers and expired leases are applied by the next claim or read, so they do not depend on a process that stays up.

Graph deletion immediately retires the graph lifetime. Ordinary workflow state and history are removed in that request. In an administratively selected environment, maintenance workers complete workflow cleanup; the response’s workflow_cleanup.pending and workflow_cleanup.job_ids report that work. Environment and account erasure also wait for the required workflow cleanup. Selected cleanup retains scrubbed barriers that prevent delayed writes from restoring deleted payload. A graph created again starts with a new epoch and no workflow state. Erasing workflows does not undo external effects.

Optional database reads from a worker

With workflow execution reads and batch admission enabled on the server, a TypeScript WorkflowWorker can opt in with executionReads: true. Its optional ctx.sparql method runs bounded SELECT or ASK reads on the claimed graph; check that the method exists when accepting older context implementations. Python exposes client.workflow_turn_reader(task) for sync and async clients. Continue heartbeating, checkpointing and completing through the control client. Put a read inside a named checkpoint to reuse its acknowledged result on replay.

The reader refreshes before first use and when its remaining lifetime cannot cover the requested read budget plus a margin. It uses server-supplied lifetime and monotonic elapsed time, including refresh transit, rather than comparing worker and server wall clocks. A locally exhausted lifetime is retryable; it never silently replays a query or falls back to the environment API key.

An invalid or already-expired bearer returns 401; a lost workflow lease or a credential that expires during a read returns 403. Batch saturation returns 429 batch_database_busy; billing refusal is 402 batch_billing_quota_exceeded. Waiting work is bounded per owner and environment as well as globally. These controls use the existing workflows and workers and do not autoscale the worker fleet. Both server gates remain off until the deployment is qualified.

Validation and further work

lbb-workflows/tests covers lease fencing across two nodes, idempotent handoff, timers and signals, message deduplication across history, an index that missed an update, and an interrupted move to history. lbb-sim/tests/integration/workflow_object_store.rs injects failed operations, lost PUT responses and applied-then-conflict responses into two interleaved nodes and checks turn numbering, single holders, state, history and the index. HTTP and console tests cover the product flow.

The design note and the performance ledger in the repository record the protocol and the handoff cost. Future work includes a per-node cache of identity objects that removes the read of most steps, ranged or batched history reads, configurable retention and state migrations. Child workflows and arbitrary-code stack snapshots are not implemented.

Operator recovery modes

Release 3 adds an experimental LBB_MODEL_WORKFLOW_SCHEDULING setting for the existing model-workflow maintenance job: off (default), observe, or recover. Configured producers for lbb.model-checks, lbb.search-tuning, lbb.model-trials and lbb.calibration, at application version v1, publish durable scheduling records before activating work. Current production writers use their canonical IDs: model-checks, search-tuning, model-trials and calibration. Creation is idle; the first message publishes the recovery record. Recovery checks the current source and claim before execution. Existing environments retain ordinary recovery. Selected new environments use the profile described below. Other workflow types and versions keep their existing worker behavior and are outside this advertised recovery mode.

Waiting work and held leases remain recoverable after a delivery disappears, without another incoming message. Canonical next-wake probes remain available; Observe/Recover adds a bounded, read-only lookup through the workflow index. These controls do not provide a fleet recovery-latency or scaling guarantee.

Noncanonical model IDs require the separate LBB_MODEL_WORKFLOW_ARBITRARY_PUBLICATION gate, which defaults to false and also requires Observe/Recover. The gate alone does not certify complete recovery coverage. After qualifying compatible writers, readers and erasers, a service administrator can create a new environment with the selected model profile. This profile covers the four v1 types above, including noncanonical IDs, and requires Recover mode. Its construction proof permits skipping the model recovery scan of graphs and sources while advertised recovery continues. It does not adopt existing environments or change other worker families. Global environment and active-job inventories remain, so this is not zero idle I/O. These controls are implemented locally; production activation remains an operator rollout decision. Once scheduling or selection records exist, turning a gate off does not remove them or make an older writer or purge worker safe to restore.

For these configured producers, a source proved retired or replaced returns 400 workflow_scheduling_not_live; unavailable authority, including a missing environment reverse locator, returns a retryable 503. Empty advertisement directories need one bounded presence list, while ordinary recovery and local orphan checks still run in unselected environments. This does not make idle graph populations free of storage requests. After scheduling records are written, keep compatible producers, readers and purge workers during upgrade or rollback; switching the setting to off does not make an older binary safe. Tenant deletion uses bounded cleanup pages and checkpoints. Inconclusive shared records are retained with a repair diagnostic and increasing worker retry delay; large or blocked inventories can still need more than one cleanup pass.

Service administrators can explicitly submit bounded historical repair through POST /api/admin/workflows/historical-discovery with campaign_id and request_id, and inspect that campaign with GET on the same path. Each POST requests one maintenance quantum; it does not automatically continue the whole inventory. This campaign still selects the four canonical IDs; arbitrary-ID historical sources are not automatically seeded. Completed traversal still reports coverage: false. A historical campaign does not provide the construction proof used by selected new environments, and does not retire the legacy recovery scans for an existing environment.