Test a workflow before it writes (experimental)

A workflow that fills values from text writes facts to your graph. Before it does, you want to see what it reads on a few real records: the value, the words of the text that state it, and what the workflow does with it.

A test runs a values.extract workflow on chosen inputs and writes nothing to the graph. You read each input’s values with their quotes, answer each value, and save the answers as the test’s dataset. The dataset has the shape of a template’s answer key, so you can compare versions of the workflow on the same examples later.

Start a test

In the console, open a values workflow and choose the Test tab. Pick a sample of the class, or paste the IRIs of chosen instances (at most 20), and choose Start test. To test a workflow before you create it, choose Test on examples in the Use template dialog.

Name the workflow in name. Send iris (at most 20) to choose the inputs, or a sample of the first instances of the watched class (1 to 20, default 6).

cURL

curl -X POST "$LBB_URL/v1/workflows/named/tests?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-test-1' --data '{"name": "sinks", "sample": 6}'

curl "$LBB_URL/v1/workflows/named/tests/get?graph=tender&id=wtest_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"

TypeScript

import { LbbClient } from "@littlebigbrain/client";

const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" });

const started = await client.workflows.tests.create({ name: "sinks", sample: 6 });
const test = await client.workflows.tests.wait(started.id);
console.log(test.status, test.counts);

Python

import os

from lbb import LbbClient

client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender")

started = client.workflows.tests.create(name="sinks", sample=6)
test = client.workflows.tests.wait(started.id)
print(test.status, test.counts)

To test a workflow that does not exist yet, send install in place of name, with the body of its create (PUT /v1/workflows/triggered).

To test earlier settings of a workflow, send revision with name: the id of a revision from its settings history. The test runs on those settings and the workflow keeps the ones it uses, so you can compare two versions on the same inputs. The test’s revision names the settings it ran on.

To test other models, send models with the model of each step you want to change: {"extract": {"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"}}. The test runs on those models and the workflow keeps its own. The test’s models names the models it ran on. Choose a model for the workflow afterwards with model choices.

A test goes from queued to running, then ends done, failed or cancelled. wait reads the test until it ends. When the test is still open after 10 minutes, wait raises a TimeoutError (TypeScript: timeoutMs, Python: timeout in seconds). If the server stops during a test, the test goes on where it stopped: an input that has a result is not read again. cancel stops a test, and the inputs it read keep their results.

list returns one page of the graph’s tests, newest first, and name keeps the tests of one workflow. all reads every page:

cURL

curl "$LBB_URL/v1/workflows/named/tests?graph=tender&name=sinks&limit=10" \
  -H "Authorization: Bearer $LBB_API_KEY"

TypeScript

for await (const earlier of client.workflows.tests.all({ name: "sinks" })) {
  console.log(earlier.id, earlier.status, earlier.dataset);
}

Python

for earlier in client.workflows.tests.all(name="sinks"):
    print(earlier.id, earlier.status, earlier.dataset)

Read the results

For each input the test keeps the text the workflow read, and for each property:

• the value, in the property’s unit;
• the quote: the words of the text that state the value;
• the support: how well the quote supports the value, from 0 to 1;
• what the live workflow does with it: Checks passed (it writes the value), Needs review (it gives the value to a person), or the check that dropped it;
• the conversion of a number the text states in another unit: 600 mm → 60 cm.

The summary counts the values: Checks passed, Needs review, No value and Failed.

Answer each value

Answer a value with Accept, Correct (and the right value) or Reject. For a property without a value, Accept says the text states none, and Correct gives the value it states. With the API, the answer cant_tell keeps the property out of the dataset, and answer: null removes an answer.

cURL

curl -X POST "$LBB_URL/v1/workflows/named/tests/answer?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-answer-1' --data '{
  "id": "wtest_1791967648726a1b2c3d4e5f6",
  "iri": "https://example.com/sink/s-4-1",
  "property": "width_cm",
  "answer": { "decision": "correct", "value": 45 }
}'

TypeScript

await client.workflows.tests.answer({
  id: test.id,
  iri: test.inputs[0]!.iri,
  property: "width_cm",
  answer: { decision: "correct", value: 45 },
});

Python

test = client.workflows.tests.answer(
    test.id, test.inputs[0].iri, "width_cm", {"decision": "correct", "value": 45}
)

An answer writes nothing to the graph.

Save the answers as a dataset

Save as dataset keeps the answers as an Evals dataset. Each answered input becomes an example: its text, the values a correct run writes, and those it leaves out. Only the properties you answered are in it.

A test has one dataset. The first save creates it, and the test names it in dataset. A later save brings that same dataset up to date with the answers, and returns it with the same id. In the console the button then reads Update dataset. Download saves the answers the same way and downloads the dataset’s two files: records (each input’s name and text) and expected (the answer key).

cURL

curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-save-1' --data '{
  "name": "sinks answers",
  "kind": "values",
  "source": { "kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6" }
}'

TypeScript

const dataset = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: test.id },
});
const files = await client.evals.datasets.export(dataset.id);
console.log(dataset.item_count, files.records.length);

Python

dataset = client.evals.datasets.create("sinks answers", source={"kind": "test", "id": test.id})
files = client.evals.datasets.export(dataset.id)
print(dataset.item_count, len(files.records))

Answer more values and save again: the dataset keeps its id.

TypeScript

const saved = await client.workflows.tests.get(test.id);
const again = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: test.id },
});
console.log(saved.dataset === again.id);

Python

saved = client.workflows.tests.get(test.id)
again = client.evals.datasets.create("sinks answers", source={"kind": "test", "id": test.id})
print(saved.dataset == again.id)

When you delete the dataset, the next save of the test creates a new one. See Datasets of examples to list, read, export, add to and delete datasets.

What a test costs

A test calls the same models as the workflow. The calls count in your environment’s monthly workflow budget, and they show in the call log with the route tests/<name>. A test that may cost more than the budget has left is refused with 429 workflow_budget_exceeded. The calls of a test do not count as the workflow’s own work: model checks, calibration and model trials leave them out.

Test a template on your graph

A template your worker runs has npm run test:examples. It runs the workflow on instances of your graph with the real models, checks each write with the commit’s dry run, and prints what the workflow would write. It writes nothing.

LBB_URL=... LBB_API_KEY=... LBB_GRAPH=tender npm run test:examples

In your own code, captureActivities(lbbActivities(client)) gives the same activities: the models and the reads run, and the writes and review items are kept instead of written. Its model calls name the test, so the call log shows them as a test’s and model checks leave them out. This needs a server with workflow tests.

In Python, capture_activities(lbb_activities(client)) gives the same activities. See Run a worker in Python.