# Datasets of examples (experimental)

Source: https://docs.littlebigbrain.com/guides/datasets/

To compare two versions of a workflow, you run both on the same examples
and check each output against the right one. A **dataset** holds those
examples. Each example has an input text and the values a correct run
writes from it.

The first kind of dataset is `values`, for workflows that fill values from
text (`values.extract` and its template). A dataset belongs to a graph, and
deleting the graph deletes its datasets. In the console, datasets are under
**Evals → Datasets**.

## What a dataset holds

A dataset has an id (`ds_…`), `object: "eval_dataset"`, `created`, a
`name`, its `kind` and `item_count`. It keeps the settings it is graded
with in `install`: the value type of each property (`text`, `keyword`,
`i64`, `f64`, `bool`), the units and the context. `about` describes the
dataset in sentences.

Each example (an item) has:

- `name`: the record's name or label, unique in the dataset;
- `text`: the input, at most 40,000 characters;
- `iri`: the instance the text was read from, when it came from a graph;
- `values`: what a correct run writes, each with its `property`, `value`,
  and optionally the `quote` that states it, the number as the text
  `stated` it (`450 mm` for a `width_cm` of 45), and other spellings a
  text value may take (`accept`);
- `leave_out`: what a correct run does not write. A `value` of null means
  no value at all for the property.

These are the fields of a template's answer key. A dataset exports as the
two sample files of a template, and an answer key loads back as the same
items.

## Save a dataset from a workflow test

A [workflow test](https://docs.littlebigbrain.com/guides/test-a-workflow/) runs a workflow on chosen
inputs, and a person answers each value. Save the answers with
`source: {kind: "test", id}`. Each answered input becomes an item, and the
test's settings become the `install`.

A test has one dataset. The first save creates it, and the test names it
in its `dataset` field. A later save of the same test brings that dataset
up to date with the answers and returns it with the same id. The items
from the test change to its current answers. Items you added yourself
stay. A delete takes the dataset off its test, so the next save makes a
new one.

### cURL

```sh
curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-save-1' --data '{
  "name": "sinks answers",
  "kind": "values",
  "source": { "kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6" }
}'
```

### TypeScript

```ts
import { LbbClient } from "@littlebigbrain/client";

const client = new LbbClient({ baseUrl: process.env.LBB_URL!, apiKey: process.env.LBB_API_KEY, graph: "tender" });

const saved = await client.evals.datasets.create({
  name: "sinks answers",
  kind: "values",
  source: { kind: "test", id: "wtest_1791967648726a1b2c3d4e5f6" },
});
console.log(saved.id, saved.item_count);
```

### Python

```python
import os

from lbb import LbbClient

client = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="tender")

saved = client.evals.datasets.create(
    "sinks answers", source={"kind": "test", "id": "wtest_1791967648726a1b2c3d4e5f6"}
)
print(saved.id, saved.item_count)
```

## Create a dataset with your own examples

Send a `name`, the `install` and the first `items`. When `install` names
properties, each item names only those.

### cURL

```sh
curl -X POST "$LBB_URL/v1/evals/datasets?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-by-hand-1' --data '{
  "name": "sinks by hand",
  "kind": "values",
  "install": {
    "properties": { "width_cm": "f64", "depth_cm": "f64" },
    "units": { "width_cm": "cm", "depth_cm": "cm" }
  },
  "items": [{
    "name": "S 1",
    "text": "S 1 Waschtisch aus Keramik, 60 cm breit.",
    "values": [{ "property": "width_cm", "value": 60, "quote": "60 cm breit" }],
    "leave_out": [{ "property": "depth_cm", "value": null }]
  }]
}'
```

### TypeScript

```ts
const dataset = await client.evals.datasets.create({
  name: "sinks by hand",
  kind: "values",
  install: {
    properties: { width_cm: "f64", depth_cm: "f64" },
    units: { width_cm: "cm", depth_cm: "cm" },
  },
  items: [
    {
      name: "S 1",
      text: "S 1 Waschtisch aus Keramik, 60 cm breit.",
      values: [{ property: "width_cm", value: 60, quote: "60 cm breit" }],
      leave_out: [{ property: "depth_cm", value: null }],
    },
  ],
});
```

### Python

```python
dataset = client.evals.datasets.create(
    "sinks by hand",
    install={
        "properties": {"width_cm": "f64", "depth_cm": "f64"},
        "units": {"width_cm": "cm", "depth_cm": "cm"},
    },
    items=[
        {
            "name": "S 1",
            "text": "S 1 Waschtisch aus Keramik, 60 cm breit.",
            "values": [{"property": "width_cm", "value": 60, "quote": "60 cm breit"}],
            "leave_out": [{"property": "depth_cm", "value": None}],
        }
    ],
)
```

## List and read datasets

`list` returns one page of the graph's datasets, newest first, without
their items (`limit` 1 to 50, default 20; pass `next_cursor` as `cursor`
for the next page). `all` reads every page. `get` returns one dataset with
its items, in the order they were added.

### cURL

```sh
curl "$LBB_URL/v1/evals/datasets?graph=tender&limit=20" \
  -H "Authorization: Bearer $LBB_API_KEY"

curl "$LBB_URL/v1/evals/datasets/get?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

### TypeScript

```ts
for await (const each of client.evals.datasets.all()) {
  console.log(each.id, each.name, each.item_count);
}
const one = await client.evals.datasets.get(dataset.id);
console.log(one.items?.map((item) => item.name));
```

### Python

```python
for each in client.evals.datasets.all():
    print(each.id, each.name, each.item_count)
one = client.evals.datasets.get(dataset.id)
print([item.name for item in one.items or []])
```

## Add examples

Add items to a dataset. An item with the name of an item the dataset has
replaces it in its place.

### cURL

```sh
curl -X POST "$LBB_URL/v1/evals/datasets/items?graph=tender" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: sinks-by-hand-2' --data '{
  "id": "ds_1791967648726a1b2c3d4e5f6",
  "items": [{
    "name": "S 2",
    "text": "S 2 Ausgussbecken, 50x35 cm.",
    "values": [
      { "property": "width_cm", "value": 50, "quote": "50x35 cm" },
      { "property": "depth_cm", "value": 35, "quote": "50x35 cm" }
    ]
  }]
}'
```

### TypeScript

```ts
await client.evals.datasets.addItems(dataset.id, [
  {
    name: "S 2",
    text: "S 2 Ausgussbecken, 50x35 cm.",
    values: [
      { property: "width_cm", value: 50, quote: "50x35 cm" },
      { property: "depth_cm", value: 35, quote: "50x35 cm" },
    ],
  },
]);
```

### Python

```python
dataset = client.evals.datasets.add_items(
    dataset.id,
    [
        {
            "name": "S 2",
            "text": "S 2 Ausgussbecken, 50x35 cm.",
            "values": [
                {"property": "width_cm", "value": 50, "quote": "50x35 cm"},
                {"property": "depth_cm", "value": 35, "quote": "50x35 cm"},
            ],
        }
    ],
)
print(dataset.item_count)
```

## Export a dataset as a template's samples

`export` returns the two files of a template's samples. `records`
(`records.json`) holds each input's `name`, `text` and `iri`. `expected`
(`expected.json`) holds `about`, `install` and the items without their
text. A template's evals read these files.

### cURL

```sh
curl "$LBB_URL/v1/evals/datasets/export?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY" > export.json
```

### TypeScript

```ts
import { writeFileSync } from "node:fs";

const files = await client.evals.datasets.export(dataset.id);
writeFileSync("records.json", JSON.stringify(files.records, null, 2));
writeFileSync("expected.json", JSON.stringify(files.expected, null, 2));
```

### Python

```python
import json

files = client.evals.datasets.export(dataset.id)
with open("records.json", "w") as out:
    json.dump([record.model_dump(exclude_none=True) for record in files.records], out, indent=2)
with open("expected.json", "w") as out:
    json.dump(files.expected.model_dump(exclude_none=True), out, indent=2)
```

## Delete a dataset

A delete removes the dataset and its items. After it, a get or an export
of the dataset answers `404`, and the list leaves it out. A repeated
delete returns the same answer. When the dataset came from a test, the
next save of that test creates a new dataset.

### cURL

```sh
curl -X DELETE "$LBB_URL/v1/evals/datasets?graph=tender&id=ds_1791967648726a1b2c3d4e5f6" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

### TypeScript

```ts
const deleted = await client.evals.datasets.delete(dataset.id);
console.log(deleted); // { id: "ds_…", object: "eval_dataset", deleted: true }
```

### Python

```python
deleted = client.evals.datasets.delete(dataset.id)
print(deleted.deleted)
```

## Limits and retries

- A dataset holds at most 500 items and 8 MiB of items. The body of one
  write is at most 1 MiB, so send more items in more writes.
- Every item needs its text and at least one entry in `values` or
  `leave_out`.
- When `install.properties` names properties, an item that names another
  property is refused with `400 invalid_request` (`param: "items"`).
- Each write takes an `Idempotency-Key`, and the SDKs send one for you. A
  retry with the same key answers what the first request did. The same key
  with another body answers `409 idempotency_conflict`.
- A request without `id` where one is needed answers `400` with
  `param: "id"`.