Workflow templates (experimental)

Most starter workflows run in LBB. You set the watch and the settings, and LBB runs the work. When you want to change the work itself (a prompt, a rule, what gets written), start from a template: the code of a starter workflow, written for your own workflow worker. The worker calls LBB’s models through activities, so your code needs no model keys and LBB meters the calls.

Template	Does	LBB-run version
documents-link	Connects mentions of the same thing across the documents: each reference links to where that thing is defined or described	documents.link
values-extract	Fills properties of a class from the text of its instances, each value with a quote	values.extract
ontology-fit	Proposes the ontology changes the texts of a class support, as review items	ontology.fit
documents-map	Maps the links of documents-link into your own ontology, as fact proposals	documents.map
review	Writes the fact proposals at or above a gate as facts, and sends the others to a person	review
link	Connects the mentions of one thing across your records (contacts, leads, deals, notes) into one thing, and proposes records of one thing as an identity for a person to accept	none yet
rows	Reads each CSV, TSV or XLSX file into one instance per row, each linked to the file with its row as the evidence	none
sections	Reads each Markdown file into its sections, each linked to its parent with its lines as the evidence	none
documents-check	Checks PDFs against each other: SHACL rules over the values values.extract reads, and a review item with the quotes of both documents for each difference	none

A template’s id is the id of its starter workflow, with dashes for dots. values-extract also has a Python port, in examples/values_extract of the Python SDK. It has the same rules and the same samples.

documents-check is an example that joins LBB’s own parts into one pipeline. Check documents against each other walks through it.

Embeddings (search.embed) and the parse of PDF files (documents.parse) have no template: they stay LBB-run.

A template is a starting point. The LBB-run version is a separate implementation that also handles large graphs and long documents; each template’s README names what that version adds. values-extract, review and ontology-fit follow the rules of their LBB-run version: the same prompts, checks, review items and counters. An eval runs each of them and its LBB-run version on the same recorded model answers and finds the same outputs, and each README names where the two differ.

Create a project from a template

npm create @littlebigbrain -- --template documents-link my-links
cd my-links
npm install
npm test

The project holds the template’s code in src/, its samples in samples/, and three commands:

Command	Does
npm test	Runs the workflow on its samples with mock models. No server.
npm run try	Creates a graph of your stack (LBB_GRAPH, default try-<template>), puts the samples on it, and creates the workflow there
npm run setup, npm start	Creates the workflow on your graph, then serves it

npm run try, npm run setup and npm start read LBB_URL (your stack’s endpoint) and LBB_API_KEY (a key of the stack). The server needs the workflow engine, and the models need a plan with managed models.

--lang python creates the Python port of a template instead. values-extract has one. Run a worker in Python shows its commands.

The link template has a fourth command: npm run grade -- --gold <file> grades the run on a graph against your gold groups (link precision and recall, cluster purity and completeness, identity precision). Its npm run setup creates one workflow per class it reads, and the parts that group the links into things. By default no model reads a record value: an equal email, web address, phone number or code links the mentions, and every other candidate goes to a person. Set modelsReadRecords: true in the settings to let the decision model decide.

Read files by their type

rows and sections start when a file arrives. Each is a named workflow with a content trigger: LBB sends it each upload of its media types within seconds, one file per message. rows takes text/csv and text/tab-separated-values. sections takes text/markdown, and text/plain when the name ends with .md. Both are in the starter workflow catalog, under Documents, with the media types of their trigger. In the console, their Use template shows the code and the command below.

npm create @littlebigbrain -- --template rows my-rows
cd my-rows && npm install
npm test
LBB_GRAPH=try-rows npm run try && LBB_GRAPH=try-rows npm start

rows writes one instance per row of the class its settings name (the samples’ Customer). A column whose header is a property of the graph fills that property: Customer ID fills customer_id. Each instance has an edge READ_FROM to the file’s Document. The edge’s evidence is the row’s text, and its source is the file’s IRI with the row: https://littlebigbrain.com/e/41ab…#row=5, as RFC 7111 counts rows (the header is row 1). sections writes one instance per heading, with its title, level, text and path of headings, and an edge PART_OF to its parent section or the file. Its evidence names the lines: #line=39,44 is lines 40 to 44 (RFC 5147). The console opens the file at those places.

Both read a file once per version and settings. Before a file, the workflow checks the file’s derivation with client.workflows.derivations.check, and skips the file when one exists. After its last write it records one, for the upload the check named. The same bytes under a second name are then read once. A file deleted while a run reads it gets no derivation from that run. The run counts it as skipped. The run of a new upload reads the file again. Say the file was deleted and not uploaded again. Then the run ends at the next read (404), and counts the file as skipped too. rows reads a file by byte range (client.files.content(sha256, { range })), at most 4 MiB per read. Each turn reads on from the byte where the last one stopped, so a large file is read about once over all its turns, and a quoted field that spans two reads is still one row. With the setting represents: { class: "Report" }, the workflow also writes Document REPRESENTS <report> when the file’s name is the label of exactly one report: q3-customers.csv and the report Q3 customers.

Read the rows of a file with SPARQL:

PREFIX p: <https://littlebigbrain.com/p/>
PREFIX r: <https://littlebigbrain.com/r/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX lbb: <https://littlebigbrain.com/ns#>
SELECT ?customer ?email ?source WHERE {
  ?doc p:name "q3-customers.csv" .
  ?customer r:read_from ?doc .
  OPTIONAL { ?customer p:email ?email }
  ?st rdf:subject ?customer ; rdf:predicate r:read_from ; lbb:evidenceSource ?source .
}

The activities

lbbActivities(client) from @littlebigbrain/client returns the nine calls a workflow makes. The first three are model calls:

Activity	Does
reason(request, { effectId, label, instance, attempt })	The reasoning model answers a prompt with JSON that follows a JSON Schema. attempt (1 to 10) names the call’s attempt in the call log: send 2 for a retry of a failed answer, under its own effectId
decide({ state, questions }, { effectId, label, instance })	The decision model answers choice questions about a state, each with a choice, a confidence and the probability of each option
embed({ texts }, { effectId, label, instance })	The embedding model embeds up to 256 texts
pages(sha256, { from, to })	The parsed pages of a file, each line and word with its box
sparql(query, options)	A SPARQL query, its rows flattened
writeOwned(ctx, stepKey, write)	An owned write, committed inside a step
review({ workflow, title, rationale, facts, change })	A review item for a person to accept
evolve(ops, { dryRun })	An ontology change, or what it would do
ontology()	The graph’s classes, relations and properties

A model call runs inside a step and passes the step’s effectId. When a worker stops after the model answered and before the step was saved, the next attempt sends the same effect id, and LBB returns the first answer at no cost:

import { isActivityRefused, lbbActivities, splitInHalves } from "@littlebigbrain/client";

const lbb = lbbActivities(client);
const answer = await ctx.step("define-p3", (s) =>
  lbb.reason<{ definitions: { row_id: string; identifier: string }[] }>(
    { prompt, schema },
    { effectId: s.effectId, label: "equipment list p3" },
  ),
);

Each model call names the turn’s run, so the run’s page lists it, and the worker adds up the calls’ usage for the turn’s run summary. A call also names its step (workflow.step, the step’s key). That is the step whose effectId the call passes, or whose effectId starts the call’s id, as in ${s.effectId}.2. Pass instance in the options (an instance IRI or a document’s name) to show what a call works on.

The calls count against the stack’s model budget for workflows. A request that is too large for the model fails with 422 activity_request_refused: isActivityRefused(error) names that error, and splitInHalves(items) splits the items of the request so each half goes in its own call. In Python, lbb_activities(client) of lbb.workflows has the same calls, with is_activity_refused and split_in_halves (Run a worker in Python).

Walk through documents-link

The samples are three PDFs of an invented school building tender: an equipment list that defines codes such as S 1.1, a specification with numbered sections, and two room sheets that use the codes, some in other spellings (S1.1, S.2.2).

1. npm run try creates the graph try-documents-link, creates LBB’s documents.parse workflow there, uploads the three PDFs and waits until they are parsed. Then it adds the classes DocEntry and DocMention to the ontology and creates the triggered workflow documents-link, which watches Document and its field parse_status.
2. LBB_GRAPH=try-documents-link npm start serves the workflow. The first trigger message names the three documents.
3. The workflow reads each document in windows of pages, one window per turn. It reads the definitions of the equipment list and the specification first and writes them as DocEntry entities. Then, for the room sheets, it asks which spans are codes, finds candidates for each span among the entries (by characters and by meaning), asks the decision model which entry each span names, and writes each mention as a DocMention with a REFERS_TO link to the entry it names.
4. A link whose confidence is below the gate (0.8) goes to the review inbox instead of the graph.

Read the links with SPARQL:

PREFIX p: <https://littlebigbrain.com/p/>
PREFIX r: <https://littlebigbrain.com/r/>
SELECT ?span ?page ?identifier WHERE {
  ?doc p:name "raumblaetter.pdf" .
  ?m r:mentioned_in ?doc ; p:span ?span ; p:page ?page ; r:refers_to ?entry .
  ?entry p:identifier ?identifier .
}
ORDER BY ?page ?span

The answer key of the samples lists 15 links, among them S1.1 to S 1.1 and S.2.2 to S 2.2. npm test runs the same workflow with mock models and checks that it makes those 15 links and no other.

To change the work, edit src/workflow.ts (the steps and the writes) or src/link.ts (the prompts, the candidates and the link rule), run npm test, and restart the worker. A turn that runs again replays its saved steps in their order, and fails when the code’s steps differ: change the order of the steps while the workflow has no turn in progress.

Do not run a template and its LBB-run version on one graph: they own the same relations, and the second one is refused with 409 workflow_owns_conflict. rows and sections can both own REPRESENTS on one graph: no upload reaches both of their content triggers.

Map the links, then review the facts

A link says “this line cites that entry”. documents-map turns the links into facts of your own ontology, such as room 0.12 HAS_EQUIPMENT T 1. For each entry, the decision model picks the instance it is, or a new instance and its class. For each link, it picks the statement the line makes. Each statement becomes a FactProposal with a confidence and its evidence.

review then decides. A proposal at or above the gate (0.8) becomes a fact, with the proposal as its evidence. A proposal below the gate goes to a person. So does every statement about a new instance, by default. The person gets one review item per instance, with each fact and its line. A statement the graph cannot hold is refused, with a note.

The two templates chain after documents-link, or after LBB’s own workflows. You can mix them with LBB’s own versions.

npm create @littlebigbrain -- --template documents-map my-map
npm create @littlebigbrain -- --template review my-review

In documents-map, npm run try uploads the three sample PDFs and lets LBB parse them. Then it commits the links a correct documents-link run makes on them. So the map runs on known links, without a link run first. The answer key holds 15 facts and 15 new instances: two rooms, eleven pieces of equipment and two sections of the specification.

In review, npm run try commits a small graph and 14 proposals, one for each case. The answer key gives the status each proposal gets: written, in_review or refused, and the three review items, one per instance.

The templates write through POST /v1/graph/commit. So they take the relations, properties and classes of the records ontology, and labels. Each README lists what the LBB-run version adds.