# Compare two versions of a workflow (experimental) Source: https://docs.littlebigbrain.com/guides/experiments/ You changed a model or a setting of a workflow, and you want to know whether the change helps. An **experiment** runs the workflow twice on the same examples, once as it is (A) and once with your change (B). It checks each output against the right values and reports the difference with a 95% interval. An experiment writes nothing to the graph. It changes no workflow and no model. You decide what to use, and you make that change yourself. ## What you need - A workflow that fills values from text (`values.extract`). - A [dataset](https://docs.littlebigbrain.com/guides/datasets/) of the right values. Save one from a [workflow test](https://docs.littlebigbrain.com/guides/test-a-workflow/) after you answered its values. A is the workflow as it runs now. B changes one part of A: - the model of one use: `extract` (the reasoning model) or `decide` (the decision model); or - the settings revision, one the workflow keeps (see [settings history](https://docs.littlebigbrain.com/guides/workflow-settings-history/)). One experiment changes one part, so the difference has one cause. To compare a model and a revision, run two experiments. ## In the console Open **Evals**, then **Experiments**. The tab lists the experiments of the graph, newest first. Each row shows the status, outcome, items, difference and spend. A workflow's **Quality** tab lists its own experiments. It links to this tab, filtered to the workflow. **New experiment** opens one dialog. 1. Pick the workflow and a dataset of values. 2. Pick what B changes: a model of one use, or one kept revision of the settings. For a model, the dialog compares the price per token. For a revision, it lists the settings that differ. B cannot be the model or the revision already in use. 3. Set the number of items and the cost limit. 4. Read the estimate. It shows the cost of A and B and the cost limit. It also shows what is left of this month's workflow budget. **Start** stays off while the estimate is over the limit or the budget. The Spend tab of a workflow has the same dialog. Pick **Change** on a model, then **Compare on a dataset**. An experiment's page shows A and B side by side. Each column shows the score, precision, recall and cost per item. It also shows the share sent to review and the failures. Under the columns, a line shows the difference of B against A with its 95% interval. The bar is marked on it. Fewer than 10 items give a note that the result is too early. Open an item to see the dataset's labels beside the values A and B read. Values of properties the dataset does not label sit in their own list. They count in no score. **Use B** asks you to confirm. The dialog states the part that changes. It shows the difference, the interval and the cost ratio. When B misses the bar, it says why. When B is worse, the page says to keep A and offers **Use B anyway**. If the workflow changed since the experiment started, the page says what moved. It offers **Run again**. ## Estimate the cost first Each arm runs a model call per item, so an experiment spends model budget. `estimate` shows what a start would estimate for both arms, the cost limit, and what is left of the environment's monthly workflow budget. It runs no model and writes nothing. ### cURL ```sh curl -X POST "$LBB_URL/v1/evals/experiments/estimate?graph=tender" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{ "workflow": "sinks", "dataset": "ds_1791967648726a1b2c3d4e5f6", "b": { "model": { "use": "extract", "model": { "provider": "anthropic", "model": "claude-haiku-5-5" } } } }' ``` ### TypeScript ```ts import { LbbClient } from "@littlebigbrain/client"; const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" }); const request = { workflow: "sinks", dataset: "ds_1791967648726a1b2c3d4e5f6", b: { model: { use: "extract" as const, model: { provider: "anthropic", model: "claude-haiku-5-5" }, }, }, }; const cost = await client.evals.experiments.estimate(request); console.log(cost.items, cost.estimate_micro_usd / 1_000_000); ``` ### Python ```python import os from lbb import LbbClient client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender") request = { "workflow": "sinks", "dataset": "ds_1791967648726a1b2c3d4e5f6", "b": {"model": {"use": "extract", "model": {"provider": "anthropic", "model": "claude-haiku-5-5"}}}, } cost = client.evals.experiments.estimate(**request) print(cost.items, cost.estimate_micro_usd / 1_000_000) ``` ## Start an experiment A start takes the same body. It runs the first `max_items` items of the dataset (at most 200, default 200). The estimate must fit `max_cost_usd` (default 5, at most 100) and the environment's monthly workflow budget. Otherwise the start is refused, with `invalid_request` (`param: max_cost_usd`) or `429 workflow_budget_exceeded`. The experiment starts `queued`. The experiments of a graph run one at a time, so a second one waits for the first. Poll `get`, or `wait`. ### cURL ```sh curl -X POST "$LBB_URL/v1/evals/experiments?graph=tender" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \ -H 'Idempotency-Key: sinks-haiku-1' --data '{ "workflow": "sinks", "dataset": "ds_1791967648726a1b2c3d4e5f6", "b": { "model": { "use": "extract", "model": { "provider": "anthropic", "model": "claude-haiku-5-5" } } }, "max_cost_usd": 2 }' curl "$LBB_URL/v1/evals/experiments/get?graph=tender&id=exp_1791967648726a1b2c3d4e5f6" \ -H "Authorization: Bearer $LBB_API_KEY" ``` ### TypeScript ```ts const started = await client.evals.experiments.create({ ...request, max_cost_usd: 2, }); console.log(started.id, started.status); const done = await client.evals.experiments.wait(started.id); console.log(done.status, done.finish_reason); ``` ### Python ```python started = client.evals.experiments.create(**request, max_cost_usd=2) print(started.id, started.status.value) done = client.evals.experiments.wait(started.id) print(done.status.value, done.finish_reason) ``` An experiment ends in one of these states: | Status | Meaning | | --- | --- | | `done` | It ended with a report. `finish_reason` says why: `all_items` (every item has both answers), `max_cost` (the cost limit) or `budget` (the environment's monthly workflow budget). | | `failed` | It stopped on an error (`error`). The items compared so far stay. | | `cancelled` | A person stopped it. The items compared so far stay. | The client sends an idempotency key unless you pass one. A start sent again with the same key answers the same experiment. The same key with another body is `409 idempotency_conflict`. ## Read the result The report compares B with A over the items both arms answered. Each item is graded against the dataset's values: precision and recall of the values the arm wrote, and their F1 score, 0 to 1. | Field | Meaning | | --- | --- | | `items` | Items compared. | | `a`, `b` | Per arm: `score` (the mean item score), `precision`, `recall`, `review_share`, `cost_micro_usd_per_item` and `failed`. | | `delta` | `b.score − a.score`. | | `ci_low`, `ci_high` | The 95% interval of the difference. | | `outcome` | `too_few` under 10 items; else `better` (the interval is above 0), `worse` (below 0) or `same`. | | `qualifies` | B meets the bar for use: at least 20 items, the interval's low end at least −0.05, no more failures than A plus 5 points, and B cheaper per item or `better`. | | `cost_ratio` | B's cost per item over A's. | | `unlabeled` | Values of properties the dataset does not label, on which the arms differ. | With fewer than 10 items the report still shows the difference, but it says `too_few`: add examples to the dataset and run again. `get` also returns one row per item with both arms' grades, in `items`. The values themselves are in the arms' workflow tests (`a.test`, `b.test`), where you can read each value with its quote. ### TypeScript ```ts const report = done.report; console.log(report.items, report.delta, report.ci_low, report.ci_high, report.outcome); for (const item of done.items ?? []) console.log(item.item, item.a?.score, item.b?.score); ``` ### Python ```python report = done.report print(report.items, report.delta, report.ci_low, report.ci_high, report.outcome.value) for item in done.items: print(item.item, item.a.score if item.a else None, item.b.score if item.b else None) ``` ### When the dataset changes The labels are the dataset's. If you add or correct an example after the experiment ran, a read of the experiment says `report.stale: true`, and a turn scores the report again with the labels as they are now. The report then names the new `labels_etag`. A deleted dataset keeps the report as it was last scored (`dataset.deleted: true`). ### Grade the values no label judges Sometimes the arms differ on a value of a property that the dataset does not label. The report lists it in `unlabeled`, and no score counts it. A person can grade it. Little Big Brain files one item in the Review inbox for each value. The origin is `evals:datasets`, and the dataset is the item's record (`eval.id`). The title reads "Is width_cm = 60 right for 'S 4.1'?". The item shows the quote the arm gave. An experiment on the same dataset that wrote the same value shares the item. An answer becomes a label of the dataset's item. `right` adds the value to `values`. `wrong` adds it to `leave_out`. The label records the answer in `source`: the item's id and the time of the answer. `can't tell` adds no label. The value stays unlabeled. Little Big Brain can ask again 30 days after the answer. The report reads as stale after an answer, and the next turn scores it again. A `wrong` on one value judges that value only. Another value of the property stays unlabeled until a person grades it. List the open items of a dataset with `GET /v1/reviews?origin_id=evals:datasets&eval_id=&status=open`. In the console, the experiment's page links each unlabeled value to its item in Review. The Review inbox lists these items under the source **Datasets**. ## Use B An experiment does not change anything. To use B, make the change that sets that part, and pass the experiment's id. The server then checks that the workflow still runs A. If it does not (the workflow was made again, its settings changed, or a model changed since), it answers `409 experiment_outdated`, and you run the experiment again. For a model, choose it for the workflow: ### cURL ```sh curl -X PUT "$LBB_URL/v1/models/choices?graph=tender" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{ "use": "extract", "model": { "provider": "anthropic", "model": "claude-haiku-5-5" }, "workflow": "sinks", "experiment": "exp_1791967648726a1b2c3d4e5f6" }' ``` ### TypeScript ```ts const chosen = await client.models.choices.put({ use: "extract", model: { provider: "anthropic", model: "claude-haiku-5-5" }, workflow: "sinks", experiment: done.id, }); console.log(chosen.model?.model, chosen.experiment); ``` ### Python ```python chosen = client.models.choices.put( use="extract", model={"provider": "anthropic", "model": "claude-haiku-5-5"}, workflow="sinks", experiment=done.id, ) print(chosen.model.model, chosen.experiment) ``` For a settings revision, activate it: ### cURL ```sh curl -X POST "$LBB_URL/v1/workflows/named/revisions/activate?graph=tender" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' --data '{ "name": "sinks", "id": "wrev_00000000000000000000000000000001", "experiment": "exp_1791967648726a1b2c3d4e5f6" }' ``` ### TypeScript ```ts await client.workflows.named.activateRevision( "sinks", "wrev_00000000000000000000000000000001", { experiment: done.id }, ); ``` ### Python ```python client.workflows.named.activate_revision( "sinks", "wrev_00000000000000000000000000000001", experiment=done.id ) ``` A repeat of either call changes nothing and succeeds. The experiment then says who applied B and when (`applied`). The choice row keeps its source `person` and names the experiment. ## Cancel an experiment `cancel` stops an experiment that has not ended. It keeps the items it compared, and a batch in flight finishes and counts its cost. If you cancel one of the experiment's two tests through the tests route (`POST /v1/workflows/named/tests/cancel`), the experiment is cancelled too. ### cURL ```sh curl -X POST "$LBB_URL/v1/evals/experiments/cancel?graph=tender&id=exp_1791967648726a1b2c3d4e5f6" \ -H "Authorization: Bearer $LBB_API_KEY" ``` ### TypeScript ```ts await client.evals.experiments.cancel(started.id); ``` ### Python ```python client.evals.experiments.cancel(started.id) ``` ## List experiments `list` returns one page of the graph's experiments, newest first, without their items. Filter by `workflow`, `dataset` and `status`. `limit` is 1 to 50 (default 20); pass `next_cursor` as `cursor` for the next page. `all` reads every page. ## What an experiment costs Every model call of an experiment goes to the call log with the route `tests/` and counts in the model activity and in the environment's monthly workflow budget, as a workflow test does. The calls stay out of calibration, the checks sampler and model trials. | Limit | Value | | --- | --- | | Items | At most 200 (`max_items`). | | Cost | `max_cost_usd`, default 5, at most 100. The estimate must fit. A run stops before a batch that would pass the limit, and ends `done` with `finish_reason: max_cost`. | | Budget | A call refused by the environment's monthly workflow budget ends the experiment `done` with `finish_reason: budget`. | Texts of 40,000 characters raise the estimate: 200 such items on Claude Sonnet 5.5 estimate about $6 per arm. Lower `max_items` or raise `max_cost_usd`. ## Errors | Code | Status | When | | --- | --- | --- | | `experiment_arms_mixed` | 400 | B changes a model and a revision. | | `experiment_outdated` | 409 | A write with `experiment` finds that the workflow no longer runs A. | | `experiments_instance_taken` | 409 | The graph has a workflow instance `eval-experiments` of another type. | | `workflow_budget_exceeded` | 429 | The estimate passes the environment's monthly workflow budget. | [Every code and how to fix it](https://docs.littlebigbrain.com/sdks/errors/).