Compare two versions of a workflow (experimental)

You changed a model or a setting of a workflow, and you want to know whether the change helps. An experiment runs the workflow twice on the same examples, once as it is (A) and once with your change (B). It checks each output against the right values and reports the difference with a 95% interval.

An experiment writes nothing to the graph. It changes no workflow and no model. You decide what to use, and you make that change yourself.

What you need

• A workflow that fills values from text (values.extract).
• A dataset of the right values. Save one from a workflow test after you answered its values.

A is the workflow as it runs now. B changes one part of A:

• the model of one use: extract (the reasoning model) or decide (the decision model); or
• the settings revision, one the workflow keeps (see settings history).

One experiment changes one part, so the difference has one cause. To compare a model and a revision, run two experiments.

In the console

Open Evals, then Experiments. The tab lists the experiments of the graph, newest first. Each row shows the status, outcome, items, difference and spend. A workflow’s Quality tab lists its own experiments. It links to this tab, filtered to the workflow.

New experiment opens one dialog.

1. Pick the workflow and a dataset of values.
2. Pick what B changes: a model of one use, or one kept revision of the settings. For a model, the dialog compares the price per token. For a revision, it lists the settings that differ. B cannot be the model or the revision already in use.
3. Set the number of items and the cost limit.
4. Read the estimate. It shows the cost of A and B and the cost limit. It also shows what is left of this month’s workflow budget. Start stays off while the estimate is over the limit or the budget.

The Spend tab of a workflow has the same dialog. Pick Change on a model, then Compare on a dataset.

An experiment’s page shows A and B side by side. Each column shows the score, precision, recall and cost per item. It also shows the share sent to review and the failures. Under the columns, a line shows the difference of B against A with its 95% interval. The bar is marked on it. Fewer than 10 items give a note that the result is too early.

Open an item to see the dataset’s labels beside the values A and B read. Values of properties the dataset does not label sit in their own list. They count in no score.

Use B asks you to confirm. The dialog states the part that changes. It shows the difference, the interval and the cost ratio. When B misses the bar, it says why. When B is worse, the page says to keep A and offers Use B anyway. If the workflow changed since the experiment started, the page says what moved. It offers Run again.

Estimate the cost first

Each arm runs a model call per item, so an experiment spends model budget. estimate shows what a start would estimate for both arms, the cost limit, and what is left of the environment’s monthly workflow budget. It runs no model and writes nothing.

cURL

curl -X POST "$LBB_URL/v1/evals/experiments/estimate?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{
  "workflow": "sinks",
  "dataset": "ds_1791967648726a1b2c3d4e5f6",
  "b": { "model": { "use": "extract",
                    "model": { "provider": "anthropic", "model": "claude-haiku-5-5" } } }
}'

TypeScript

import { LbbClient } from "@littlebigbrain/client";

const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" });

const request = {
  workflow: "sinks",
  dataset: "ds_1791967648726a1b2c3d4e5f6",
  b: {
    model: {
      use: "extract" as const,
      model: { provider: "anthropic", model: "claude-haiku-5-5" },
    },
  },
};

const cost = await client.evals.experiments.estimate(request);
console.log(cost.items, cost.estimate_micro_usd / 1_000_000);

Python

import os

from lbb import LbbClient

client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender")

request = {
    "workflow": "sinks",
    "dataset": "ds_1791967648726a1b2c3d4e5f6",
    "b": {"model": {"use": "extract", "model": {"provider": "anthropic", "model": "claude-haiku-5-5"}}},
}

cost = client.evals.experiments.estimate(**request)
print(cost.items, cost.estimate_micro_usd / 1_000_000)

Start an experiment

A start takes the same body. It runs the first max_items items of the dataset (at most 200, default 200). The estimate must fit max_cost_usd (default 5, at most 100) and the environment’s monthly workflow budget. Otherwise the start is refused, with invalid_request (param: max_cost_usd) or 429 workflow_budget_exceeded.

The experiment starts queued. The experiments of a graph run one at a time, so a second one waits for the first. Poll get, or wait.

cURL

curl -X POST "$LBB_URL/v1/evals/experiments?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-haiku-1' --data '{
  "workflow": "sinks",
  "dataset": "ds_1791967648726a1b2c3d4e5f6",
  "b": { "model": { "use": "extract",
                    "model": { "provider": "anthropic", "model": "claude-haiku-5-5" } } },
  "max_cost_usd": 2
}'

curl "$LBB_URL/v1/evals/experiments/get?graph=tender&id=exp_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"

TypeScript

const started = await client.evals.experiments.create({
  ...request,
  max_cost_usd: 2,
});
console.log(started.id, started.status);

const done = await client.evals.experiments.wait(started.id);
console.log(done.status, done.finish_reason);

Python

started = client.evals.experiments.create(**request, max_cost_usd=2)
print(started.id, started.status.value)

done = client.evals.experiments.wait(started.id)
print(done.status.value, done.finish_reason)

An experiment ends in one of these states:

Status	Meaning
done	It ended with a report. finish_reason says why: all_items (every item has both answers), max_cost (the cost limit) or budget (the environment’s monthly workflow budget).
failed	It stopped on an error (error). The items compared so far stay.
cancelled	A person stopped it. The items compared so far stay.

The client sends an idempotency key unless you pass one. A start sent again with the same key answers the same experiment. The same key with another body is 409 idempotency_conflict.

Read the result

The report compares B with A over the items both arms answered. Each item is graded against the dataset’s values: precision and recall of the values the arm wrote, and their F1 score, 0 to 1.

Field	Meaning
items	Items compared.
a, b	Per arm: score (the mean item score), precision, recall, review_share, cost_micro_usd_per_item and failed.
delta	b.score − a.score.
ci_low, ci_high	The 95% interval of the difference.
outcome	too_few under 10 items; else better (the interval is above 0), worse (below 0) or same.
qualifies	B meets the bar for use: at least 20 items, the interval’s low end at least −0.05, no more failures than A plus 5 points, and B cheaper per item or better.
cost_ratio	B’s cost per item over A’s.
unlabeled	Values of properties the dataset does not label, on which the arms differ.

With fewer than 10 items the report still shows the difference, but it says too_few: add examples to the dataset and run again.

get also returns one row per item with both arms’ grades, in items. The values themselves are in the arms’ workflow tests (a.test, b.test), where you can read each value with its quote.

TypeScript

const report = done.report;
console.log(report.items, report.delta, report.ci_low, report.ci_high, report.outcome);
for (const item of done.items ?? [])
  console.log(item.item, item.a?.score, item.b?.score);

Python

report = done.report
print(report.items, report.delta, report.ci_low, report.ci_high, report.outcome.value)
for item in done.items:
    print(item.item, item.a.score if item.a else None, item.b.score if item.b else None)

When the dataset changes

The labels are the dataset’s. If you add or correct an example after the experiment ran, a read of the experiment says report.stale: true, and a turn scores the report again with the labels as they are now. The report then names the new labels_etag. A deleted dataset keeps the report as it was last scored (dataset.deleted: true).

Grade the values no label judges

Sometimes the arms differ on a value of a property that the dataset does not label. The report lists it in unlabeled, and no score counts it. A person can grade it. Little Big Brain files one item in the Review inbox for each value. The origin is evals:datasets, and the dataset is the item’s record (eval.id). The title reads “Is width_cm = 60 right for ‘S 4.1’?”. The item shows the quote the arm gave. An experiment on the same dataset that wrote the same value shares the item.

An answer becomes a label of the dataset’s item. right adds the value to values. wrong adds it to leave_out. The label records the answer in source: the item’s id and the time of the answer. can't tell adds no label. The value stays unlabeled. Little Big Brain can ask again 30 days after the answer.

The report reads as stale after an answer, and the next turn scores it again. A wrong on one value judges that value only. Another value of the property stays unlabeled until a person grades it.

List the open items of a dataset with GET /v1/reviews?origin_id=evals:datasets&eval_id=<dataset id>&status=open.

In the console, the experiment’s page links each unlabeled value to its item in Review. The Review inbox lists these items under the source Datasets.

Use B

An experiment does not change anything. To use B, make the change that sets that part, and pass the experiment’s id. The server then checks that the workflow still runs A. If it does not (the workflow was made again, its settings changed, or a model changed since), it answers 409 experiment_outdated, and you run the experiment again.

For a model, choose it for the workflow:

cURL

curl -X PUT "$LBB_URL/v1/models/choices?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{
  "use": "extract",
  "model": { "provider": "anthropic", "model": "claude-haiku-5-5" },
  "workflow": "sinks",
  "experiment": "exp_1791967648726a1b2c3d4e5f6"
}'

TypeScript

const chosen = await client.models.choices.put({
  use: "extract",
  model: { provider: "anthropic", model: "claude-haiku-5-5" },
  workflow: "sinks",
  experiment: done.id,
});
console.log(chosen.model?.model, chosen.experiment);

Python

chosen = client.models.choices.put(
    use="extract",
    model={"provider": "anthropic", "model": "claude-haiku-5-5"},
    workflow="sinks",
    experiment=done.id,
)
print(chosen.model.model, chosen.experiment)

For a settings revision, activate it:

cURL

curl -X POST "$LBB_URL/v1/workflows/named/revisions/activate?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{
  "name": "sinks",
  "id": "wrev_00000000000000000000000000000001",
  "experiment": "exp_1791967648726a1b2c3d4e5f6"
}'

TypeScript

await client.workflows.named.activateRevision(
  "sinks",
  "wrev_00000000000000000000000000000001",
  { experiment: done.id },
);

Python

client.workflows.named.activate_revision(
    "sinks", "wrev_00000000000000000000000000000001", experiment=done.id
)

A repeat of either call changes nothing and succeeds. The experiment then says who applied B and when (applied). The choice row keeps its source person and names the experiment.

Cancel an experiment

cancel stops an experiment that has not ended. It keeps the items it compared, and a batch in flight finishes and counts its cost. If you cancel one of the experiment’s two tests through the tests route (POST /v1/workflows/named/tests/cancel), the experiment is cancelled too.

cURL

curl -X POST "$LBB_URL/v1/evals/experiments/cancel?graph=tender&id=exp_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"

TypeScript

await client.evals.experiments.cancel(started.id);

Python

client.evals.experiments.cancel(started.id)

List experiments

list returns one page of the graph’s experiments, newest first, without their items. Filter by workflow, dataset and status. limit is 1 to 50 (default 20); pass next_cursor as cursor for the next page. all reads every page.

What an experiment costs

Every model call of an experiment goes to the call log with the route tests/<workflow> and counts in the model activity and in the environment’s monthly workflow budget, as a workflow test does. The calls stay out of calibration, the checks sampler and model trials.

Limit	Value
Items	At most 200 (max_items).
Cost	max_cost_usd, default 5, at most 100. The estimate must fit. A run stops before a batch that would pass the limit, and ends done with finish_reason: max_cost.
Budget	A call refused by the environment’s monthly workflow budget ends the experiment done with finish_reason: budget.

Texts of 40,000 characters raise the estimate: 200 such items on Claude Sonnet 5.5 estimate about $6 per arm. Lower max_items or raise max_cost_usd.

Errors

Code	Status	When
experiment_arms_mixed	400	B changes a model and a revision.
experiment_outdated	409	A write with experiment finds that the workflow no longer runs A.
experiments_instance_taken	409	The graph has a workflow instance eval-experiments of another type.
workflow_budget_exceeded	429	The estimate passes the environment’s monthly workflow budget.

Every code and how to fix it.