# Test a workflow before it writes (experimental)

Source: https://docs.littlebigbrain.com/guides/test-a-workflow/

A workflow that fills values from text writes facts to your graph. Before
it does, you want to see what it reads on a few real records: the value,
the words of the text that state it, and what the workflow does with it.

A **test** runs a `values.extract` workflow on chosen inputs and writes
nothing to the graph. You read each input's values with their quotes,
answer each value, and save the answers as the test's
[dataset](https://docs.littlebigbrain.com/guides/datasets/). The dataset has the shape of a template's
answer key, so you can compare versions of the workflow on the same
examples later.

## Start a test

In the console, open a values workflow and choose the **Test** tab. Pick a
sample of the class, or paste the IRIs of chosen instances (at most 20),
and choose **Start test**. To test a workflow before you create it, choose
**Test on examples** in the Use template dialog.

Name the workflow in `name`. Send `iris` (at most 20) to choose the inputs,
or a `sample` of the first instances of the watched class (1 to 20,
default 6).

### cURL

```sh
curl -X POST "$LBB_URL/v1/workflows/named/tests?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-test-1' --data '{"name": "sinks", "sample": 6}'

curl "$LBB_URL/v1/workflows/named/tests/get?graph=tender&id=wtest_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

### TypeScript

```ts
import { LbbClient } from "@littlebigbrain/client";

const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" });

const started = await client.workflows.tests.create({ name: "sinks", sample: 6 });
const test = await client.workflows.tests.wait(started.id);
console.log(test.status, test.counts);
```

### Python

```python
import os

from lbb import LbbClient

client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender")

started = client.workflows.tests.create(name="sinks", sample=6)
test = client.workflows.tests.wait(started.id)
print(test.status, test.counts)
```

To test a workflow that does not exist yet, send `install` in place of
`name`, with the body of its create (`PUT /v1/workflows/triggered`).

To test earlier settings of a workflow, send `revision` with `name`: the id
of a revision from its
[settings history](https://docs.littlebigbrain.com/guides/workflow-settings-history/). The test runs on
those settings and the workflow keeps the ones it uses, so you can compare
two versions on the same inputs. The test's `revision` names the settings it
ran on.

To test other models, send `models` with the model of each step you want to
change: `{"extract": {"provider": "anthropic", "model": "claude-haiku-5-5",
"effort": "low"}}`. The test runs on those models and the workflow keeps
its own. The test's `models` names the models it ran on. Choose a model for
the workflow afterwards with
[model choices](https://docs.littlebigbrain.com/guides/check-model-quality/#choose-the-model-of-a-workflow-step).

A test goes from `queued` to `running`, then ends `done`, `failed` or
`cancelled`. `wait` reads the test until it ends. When the test is still
open after 10 minutes, `wait` raises a `TimeoutError` (TypeScript:
`timeoutMs`, Python: `timeout` in seconds). If the server stops during a
test, the test goes on where it stopped: an input that has a result is not
read again. `cancel` stops a test, and the inputs it read keep their
results.

`list` returns one page of the graph's tests, newest first, and `name`
keeps the tests of one workflow. `all` reads every page:

### cURL

```sh
curl "$LBB_URL/v1/workflows/named/tests?graph=tender&name=sinks&limit=10" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

### TypeScript

```ts
for await (const earlier of client.workflows.tests.all({ name: "sinks" })) {
  console.log(earlier.id, earlier.status, earlier.dataset);
}
```

### Python

```python
for earlier in client.workflows.tests.all(name="sinks"):
    print(earlier.id, earlier.status, earlier.dataset)
```

## Read the results

For each input the test keeps the text the workflow read, and for each
property:

- the value, in the property's unit;
- the quote: the words of the text that state the value;
- the support: how well the quote supports the value, from 0 to 1;
- what the live workflow does with it: **Checks passed** (it writes the
  value), **Needs review** (it gives the value to a person), or the check
  that dropped it;
- the conversion of a number the text states in another unit:
  `600 mm → 60 cm`.

The summary counts the values: Checks passed, Needs review, No value and
Failed.

## Answer each value

Answer a value with **Accept**, **Correct** (and the right value) or
**Reject**. For a property without a value, **Accept** says the text states
none, and **Correct** gives the value it states. With the API, the answer
`cant_tell` keeps the property out of the dataset, and `answer: null`
removes an answer.

### cURL

```sh
curl -X POST "$LBB_URL/v1/workflows/named/tests/answer?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-answer-1' --data '{
  "id": "wtest_1791967648726a1b2c3d4e5f6",
  "iri": "https://example.com/sink/s-4-1",
  "property": "width_cm",
  "answer": { "decision": "correct", "value": 45 }
}'
```

### TypeScript

```ts
await client.workflows.tests.answer({
  id: test.id,
  iri: test.inputs[0]!.iri,
  property: "width_cm",
  answer: { decision: "correct", value: 45 },
});
```

### Python

```python
test = client.workflows.tests.answer(
    test.id, test.inputs[0].iri, "width_cm", {"decision": "correct", "value": 45}
)
```

An answer writes nothing to the graph.

## Save the answers as a dataset

**Save as dataset** keeps the answers as an Evals dataset. Each answered
input becomes an example: its text, the values a correct run writes, and
those it leaves out. Only the properties you answered are in it.

A test has one dataset. The first save creates it, and the test names it
in `dataset`. A later save brings that same dataset up to date with the
answers, and returns it with the same id. In the console the button then
reads **Update dataset**. **Download** saves the answers the same way and
downloads the dataset's two files: `records` (each input's name and text)
and `expected` (the answer key).

### cURL

```sh
curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-save-1' --data '{
  "name": "sinks answers",
  "kind": "values",
  "source": { "kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6" }
}'
```

### TypeScript

```ts
const dataset = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: test.id },
});
const files = await client.evals.datasets.export(dataset.id);
console.log(dataset.item_count, files.records.length);
```

### Python

```python
dataset = client.evals.datasets.create("sinks answers", source={"kind": "test", "id": test.id})
files = client.evals.datasets.export(dataset.id)
print(dataset.item_count, len(files.records))
```

Answer more values and save again: the dataset keeps its id.

### TypeScript

```ts
const saved = await client.workflows.tests.get(test.id);
const again = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: test.id },
});
console.log(saved.dataset === again.id);
```

### Python

```python
saved = client.workflows.tests.get(test.id)
again = client.evals.datasets.create("sinks answers", source={"kind": "test", "id": test.id})
print(saved.dataset == again.id)
```

When you delete the dataset, the next save of the test creates a new one.
See [Datasets of examples](https://docs.littlebigbrain.com/guides/datasets/) to list, read, export, add
to and delete datasets.

## What a test costs

A test calls the same models as the workflow. The calls count in your
environment's monthly workflow budget, and they show in the call log with
the route `tests/<name>`. A test that may cost more than the budget has
left is refused with `429 workflow_budget_exceeded`. The calls of a test do
not count as the workflow's own work: model checks, calibration and model
trials leave them out.

## Test a template on your graph

A template your worker runs has `npm run test:examples`. It runs the
workflow on instances of your graph with the real models, checks each
write with the commit's dry run, and prints what the workflow would write.
It writes nothing.

```sh
LBB_URL=... LBB_API_KEY=... LBB_GRAPH=tender npm run test:examples
```

In your own code, `captureActivities(lbbActivities(client))` gives the
same activities: the models and the reads run, and the writes and review
items are kept instead of written. Its model calls name the test, so the
call log shows them as a test's and model checks leave them out. This needs
a server with workflow tests.

In Python, `capture_activities(lbb_activities(client))` gives the same
activities. See
[Run a worker in Python](https://docs.littlebigbrain.com/guides/python-worker/#test-a-workflow-before-it-writes).