Datasets of examples (experimental)

View Markdown

To compare two versions of a workflow, you run both on the same examples and check each output against the right one. A dataset holds those examples. Each example has an input text and the values a correct run writes from it.

The first kind of dataset is values, for workflows that fill values from text (values.extract and its template). A dataset belongs to a graph, and deleting the graph deletes its datasets. In the console, datasets are under Evals → Datasets.

What a dataset holds

A dataset has an id (ds_…), object: "eval_dataset", created, a name, its kind and item_count. It keeps the settings it is graded with in install: the value type of each property (text, keyword, i64, f64, bool), the units and the context. about describes the dataset in sentences.

Each example (an item) has:

name: the record’s name or label, unique in the dataset;
text: the input, at most 40,000 characters;
iri: the instance the text was read from, when it came from a graph;
values: what a correct run writes, each with its property, value, and optionally the quote that states it, the number as the text stated it (450 mm for a width_cm of 45), and other spellings a text value may take (accept);
leave_out: what a correct run does not write. A value of null means no value at all for the property.

These are the fields of a template’s answer key. A dataset exports as the two sample files of a template, and an answer key loads back as the same items.

Save a dataset from a workflow test

A workflow test runs a workflow on chosen inputs, and a person answers each value. Save the answers with source: {kind: "test", id}. Each answered input becomes an item, and the test’s settings become the install.

A test has one dataset. The first save creates it, and the test names it in its dataset field. A later save of the same test brings that dataset up to date with the answers and returns it with the same id. The items from the test change to its current answers. Items you added yourself stay. A delete takes the dataset off its test, so the next save makes a new one.

curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-save-1' --data '{
  "name": "sinks answers",
  "kind": "values",
  "source": { "kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6" }
}'

import { LbbClient } from "@littlebigbrain/client";

const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" });

const saved = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: "wtest_1791967648726a1b2c3d4e5f6" },
});
console.log(saved.id, saved.item_count);

import os

from lbb import LbbClient

client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender")

saved = client.evals.datasets.create(
    "sinks answers", source={"kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6"}
)
print(saved.id, saved.item_count)

Create a dataset with your own examples

Send a name, the install and the first items. When install names properties, each item names only those.

curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-by-hand-1' --data '{
  "name": "sinks by hand",
  "kind": "values",
  "install": {
    "properties": { "width_cm": "f64", "depth_cm": "f64" },
    "units": { "width_cm": "cm", "depth_cm": "cm" }
  },
  "items": [{
    "name": "S 1",
    "text": "S 1 Waschtisch aus Keramik, 60 cm breit.",
    "values": [{ "property": "width_cm", "value": 60, "quote": "60 cm breit" }],
    "leave_out": [{ "property": "depth_cm", "value": null }]
  }]
}'

const dataset = await client.evals.datasets.create({
  name: "sinks by hand",
  kind: "values",
  install: {
    properties: { width_cm: "f64", depth_cm: "f64" },
    units: { width_cm: "cm", depth_cm: "cm" },
  },
  items: [
    {
      name: "S 1",
      text: "S 1 Waschtisch aus Keramik, 60 cm breit.",
      values: [{ property: "width_cm", value: 60, quote: "60 cm breit" }],
      leave_out: [{ property: "depth_cm", value: null }],
    },
  ],
});

dataset = client.evals.datasets.create(
    "sinks by hand",
    install={
        "properties": {"width_cm": "f64", "depth_cm": "f64"},
        "units": {"width_cm": "cm", "depth_cm": "cm"},
    },
    items=[
        {
            "name": "S 1",
            "text": "S 1 Waschtisch aus Keramik, 60 cm breit.",
            "values": [{"property": "width_cm", "value": 60, "quote": "60 cm breit"}],
            "leave_out": [{"property": "depth_cm", "value": None}],
        }
    ],
)

List and read datasets

list returns one page of the graph’s datasets, newest first, without their items (limit 1 to 50, default 20; pass next_cursor as cursor for the next page). all reads every page. get returns one dataset with its items, in the order they were added.

curl "$LBB_URL/v1/evals/datasets?graph=tender&limit=20" \
  -H "Authorization: Bearer $LBB_API_KEY"

curl "$LBB_URL/v1/evals/datasets/get?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"

for await (const each of client.evals.datasets.all()) {
  console.log(each.id, each.name, each.item_count);
}
const one = await client.evals.datasets.get(dataset.id);
console.log(one.items?.map((item) => item.name));

for each in client.evals.datasets.all():
    print(each.id, each.name, each.item_count)
one = client.evals.datasets.get(dataset.id)
print([item.name for item in one.items or []])

Add examples

Add items to a dataset. An item with the name of an item the dataset has replaces it in its place.

curl -X POST "$LBB_URL/v1/evals/datasets/items?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-by-hand-2' --data '{
  "id": "ds_1791967648726a1b2c3d4e5f6",
  "items": [{
    "name": "S 2",
    "text": "S 2 Ausgussbecken, 50x35 cm.",
    "values": [
      { "property": "width_cm", "value": 50, "quote": "50x35 cm" },
      { "property": "depth_cm", "value": 35, "quote": "50x35 cm" }
    ]
  }]
}'

await client.evals.datasets.addItems(dataset.id, [
  {
    name: "S 2",
    text: "S 2 Ausgussbecken, 50x35 cm.",
    values: [
      { property: "width_cm", value: 50, quote: "50x35 cm" },
      { property: "depth_cm", value: 35, quote: "50x35 cm" },
    ],
  },
]);

dataset = client.evals.datasets.add_items(
    dataset.id,
    [
        {
            "name": "S 2",
            "text": "S 2 Ausgussbecken, 50x35 cm.",
            "values": [
                {"property": "width_cm", "value": 50, "quote": "50x35 cm"},
                {"property": "depth_cm", "value": 35, "quote": "50x35 cm"},
            ],
        }
    ],
)
print(dataset.item_count)

Export a dataset as a template’s samples

export returns the two files of a template’s samples. records (records.json) holds each input’s name, text and iri. expected (expected.json) holds about, install and the items without their text. A template’s evals read these files.

curl "$LBB_URL/v1/evals/datasets/export?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY" > export.json

import { writeFileSync } from "node:fs";

const files = await client.evals.datasets.export(dataset.id);
writeFileSync("records.json", JSON.stringify(files.records, null, 2));
writeFileSync("expected.json", JSON.stringify(files.expected, null, 2));

import json

files = client.evals.datasets.export(dataset.id)
with open("records.json", "w") as out:
    json.dump([record.model_dump(exclude_none=True) for record in files.records], out, indent=2)
with open("expected.json", "w") as out:
    json.dump(files.expected.model_dump(exclude_none=True), out, indent=2)

Delete a dataset

A delete removes the dataset and its items. After it, a get or an export of the dataset answers 404, and the list leaves it out. A repeated delete returns the same answer. When the dataset came from a test, the next save of that test creates a new dataset.

curl -X DELETE "$LBB_URL/v1/evals/datasets?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"

const deleted = await client.evals.datasets.delete(dataset.id);
console.log(deleted); // { id: "ds_…", object: "eval_dataset", deleted: true }

deleted = client.evals.datasets.delete(dataset.id)
print(deleted.deleted)

Limits and retries

A dataset holds at most 500 items and 8 MiB of items. The body of one write is at most 1 MiB, so send more items in more writes.
Every item needs its text and at least one entry in values or leave_out.
When install.properties names properties, an item that names another property is refused with 400 invalid_request (param: "items").
Each write takes an Idempotency-Key, and the SDKs send one for you. A retry with the same key answers what the first request did. The same key with another body answers 409 idempotency_conflict.
A request without id where one is needed answers 400 with param: "id".