Check documents against each other (experimental)

One purchase leaves several documents. The order says 10 kg at 12.50 EUR. The delivery note says 8 kg arrived. The invoice bills 10 kg at 14.00 EUR.

Each document is correct on its own. The differences show only when someone reads two of them side by side.

This guide builds that check from LBB’s own parts. Upload the documents. LBB reads the values, with the quote that states each. SHACL shapes state the rules between the documents.

Each difference becomes a review item for a person. The item shows the quotes of both documents.

The output is a list of candidates for a person to check. The pipeline makes no payment, approval or compliance decision. It writes no fact about the purchase. A person reads the quotes and decides what each difference means.

The pipeline is the documents-check template. All names and numbers on this page are invented.

The pipeline

Step	Part	Runs in
1	Files: an upload of a PDF is a Document	LBB
2	documents.parse reads the PDF into pages, with every line and its box	LBB
3	documents-texts writes the text of each parsed PDF as an instance of a class	your worker
4	values.extract reads the text of each instance and writes its values, each with a quote	LBB
5	SHACL shapes state the rules between the documents; LBB validates them after each publication	LBB
6	documents-check files one review item per difference	your worker

Your worker runs two small workflows. Everything else is LBB-run.

documents-texts is glue. It finds the class of a PDF in its first line. The line reads “PURCHASE ORDER”, “DELIVERY NOTE” or “INVOICE”. documents-check reads the conformance report of the shapes.

Run it on a local stack

In the LBB repository, one command starts a stack with mock models. It needs no keys. Step 2 needs pdftotext from Poppler.

npm run dev:stack -- --worker documents-check

The command starts a data plane and creates the project from the template. It puts the pipeline on graph main and uploads three sample PDFs. Then it starts the worker.

After about a minute, open the console and choose Review. Three items wait for you.

From your own project, use any stack that has the workflow engine and models:

npm create @littlebigbrain -- --template documents-check my-checks
cd my-checks && npm install
npm test                                      # both workflows on the samples, with mock models
LBB_GRAPH=try-documents-check npm run try     # the pipeline and the sample PDFs on your stack
LBB_GRAPH=try-documents-check npm start       # serve the two workflows

npm run setup creates the pipeline on a graph of yours. It uploads nothing.

What setup changes

Setup publishes shapes.ttl as the graph’s shape set. Publishing replaces the whole set, and it can change the enforcement mode. So setup reads the graph’s shapes first.

• A graph with shapes of its own stops the setup before anything is written. The error names the shapes. Merge shapes.ttl into yours and publish the union in the mode you want.
• After the merge, set publishShapes: false in the settings. Setup then reads and writes no shape set, and it creates the rest.
• A graph that holds these shapes keeps its mode.
• A graph with no shapes gets warn.

The workflows own the classes PurchaseOrder, DeliveryNote and Invoice. They own the properties document_text and read_sha256, and the relation TEXT_OF. They drop only the instances they wrote.

The rows and sections templates own other names (READ_FROM, text). All three can run on one graph.

The sample documents

Three PDFs, one page each. The lines have the form label: value, which the mock models read.

Document	Holds
purchase-order.pdf	reference: PO-1001, quantity: 10 kg, price: 12.50 EUR, total: 125.00 EUR
delivery-note.pdf	reference: DN-1001, order: PO-1001, quantity: 8000 g
invoice.pdf	reference: INV-1001, order: PO-1001, delivery: DN-1001, quantity: 10 kg, price: 14.00 EUR, total: 139.50 EUR

The delivery note gives its weight in grams. The ontology declares kg as the unit of quantity. So values.extract writes 8, with the quote quantity: 8000 g.

The two quantities compare in one unit without any code of yours.

The rules

A shape selects the documents of a class. It reports a result for each one that breaks the rule.

A rule that compares two documents is a SHACL-SPARQL constraint (sh:sparql). Its sh:select returns two variables:

• $this, the document that breaks the rule;
• ?value, the document it was compared with.

This rule says an invoice must not bill more than the delivery note it names shows:

ex:InvoiceQuantityVsDelivery a sh:NodeShape ;
  sh:targetClass dclass:invoice ;
  sh:severity sh:Warning ;
  sh:sparql [
    sh:message "The invoice bills more than the delivery note shows" ;
    sh:select """
      PREFIX dclass: <https://littlebigbrain.com/class/>
      PREFIX dp: <https://littlebigbrain.com/p/>
      SELECT $this ?value WHERE {
        $this dp:delivery ?reference ; dp:quantity ?billed .
        ?value a dclass:deliverynote ; dp:reference ?reference ; dp:quantity ?delivered .
        FILTER (?billed > ?delivered)
      }""" ] .

A document names another by its reference. The invoice says delivery: DN-1001, and the delivery note has reference: DN-1001. The shape joins on that value.

The template has four rules:

Rule	Reports
invoice-quantity-vs-delivery	The invoice bills more than the delivery note shows, to the gram
invoice-quantity-vs-order	The invoice bills more than the order asks for, to the gram
invoice-price-vs-order	The invoice price differs from the order price, compared in cents
total-matches-quantity-times-price	A total differs from quantity times price by more than one cent

The last rule has a tolerance. A total that is a cent off from rounding passes.

The shapes compare money in whole cents and quantities to the gram. So a floating-point error never decides. The samples break three rules and satisfy the other two on purpose.

The shapes are published in warn mode:

import { readFileSync } from "node:fs";

await lbb.schema.publish({
  desired_mode: "warn",
  shapes: { source: readFileSync("shapes.ttl", "utf8"), format: "turtle" },
});

In warn mode every write is accepted. The conformance report lists the differences.

A rule with sh:sparql has no bounded set of affected records. On a records graph, reject mode would check the whole graph on each commit. An RDF-native graph refuses reject mode for such a rule.

Use warn for rules that compare documents. See Published SHACL conformance.

What lands in the Review inbox

documents-check runs in three cases:

• a field of a document changes;
• a document is removed;
• a person answers one of its items.

It watches the class BusinessDocument, which the three document classes extend.

Each run reads the conformance report. When the message is about a commit, the run waits until the report covers that commit. The wait is a durable timer, so no worker is held.

The run then finds each quote in the parsed pages of its file. It files one item of kind choice per result.

{
  "kind": "choice",
  "title": "Invoice INV-1001 bills 10 kg. Delivery note DN-1001 shows 8 kg.",
  "subject": { "type": "Invoice", "name": "invoice.pdf" },
  "question": {
    "options": [
      { "id": "discrepancy", "label": "A real difference: follow it up" },
      { "id": "misread", "label": "A value was read wrongly: correct it" },
      { "id": "acceptable", "label": "Acceptable: no action" }
    ],
    "proposed": "discrepancy"
  },
  "evidence": [
    {
      "quote": "quantity: 10 kg",
      "source_id": "https://littlebigbrain.com/e/357244d5…#page=1",
      "region": { "page": 0, "x0": 0.121, "y0": 0.3232, "x1": 0.2413, "y1": 0.3352, "confidence": 1 }
    },
    {
      "quote": "quantity: 8000 g",
      "source_id": "https://littlebigbrain.com/e/19a8c14d…#page=1",
      "region": { "page": 0, "x0": 0.121, "y0": 0.2947, "x1": 0.2526, "y1": 0.3067, "confidence": 1 }
    }
  ]
}

Each quote names its file and page, and the box of its line. The inbox shows each quote with its source. A client of your own can open the page at the box.

List the items from code:

const { data } = await lbb.reviews.list({
  originId: "starter:documents-check",
  status: "open",
  kind: "choice",
});
for (const item of data) console.log(item.title);

curl "$LBB_URL/v1/reviews?graph=main&kind=choice&origin_id=starter:documents-check" \
  -H "Authorization: Bearer $LBB_API_KEY"

Answer an item

A person chooses one option, or Can’t tell. A choice is a label. It writes no fact.

Answer	Meaning
A real difference	The person follows it up outside LBB.
A value was read wrongly	The model read a document wrongly. The person corrects the value in the value item that values.extract filed, when the value was below the gate, or deletes the file and uploads the right one.
Acceptable	The difference is known and needs no action.

A change of a value, or the deletion of a file, wakes documents-check again. When the report no longer lists a difference, the workflow withdraws its open item. An item that a person answered stays as answered.

The key of an item holds a hash of the document IRIs and the quotes:

• The same difference is the same item, however often the check runs.
• A difference with other quotes is a new item.
• A difference that returns reopens its withdrawn item.

Use your own documents

Change four things, all in the template:

1. The kinds. KINDS in texts.ts lists each class. It also lists the first line that names the class, and the properties values.extract fills. PROPERTIES in workflow.ts gives their types and units.
2. The rules. Write a shape in shapes.ttl. Add its entry to checks in checks.ts. The entry holds the sentences of the review item and the quotes it shows. A test fails when a shape has no entry or an entry has no shape.
3. The context. defaults.context in workflow.ts is one sentence about your documents, for the model.
4. The samples. npm test reads three things. They are samples/records.json (the text of each document), the parsed pages in samples/pages/ and the answer key samples/expected.json. Replace the three together with your own.

With real models, the quote of a value can differ from the quote in the key.

The records of samples/expected.json have the shape of the values-extract answer key. They also load as an Evals dataset of kind values. Use it to compare models on your documents.

Limits

• Each class has one value per property. A document with several line items needs one instance per line. Change documents-texts to write one instance per row.
• A rule skips a document whose reference names no document of the graph. An invoice that names no delivery note is never compared with one. The same holds when the delivery note does not exist.
• The first line of a PDF must equal the title of a kind, ignoring case. documents-texts skips “INVOICE REMINDER”.
• The price is read as a price per kg, as the samples say. The shapes do not compare currencies.
• The text of a document is the lines of its pages, up to 20,000 characters.
• A quantity is compared in the unit the ontology declares. values.extract converts a quantity in another unit of the same kind. It refuses a unit of another kind.
• The shapes compare what the model read. A wrongly read value can start a difference or hide one. The quotes let a person see which.
• A value below the gate is a value item, as in any values.extract workflow. The shapes see the value only after it is written.
• A turn files at most 25 new items (candidatesPerTurn, at most 40). A continuation files the rest, in up to 20 more turns. An item that exists already, open or answered, takes no share of a turn. A turn reads a report window of 150 results. The state counts the results past it as skipped.
• A turn that leaves a result out withdraws nothing. A value with no quote gives no item, so it leaves a result out too. Withdrawals wait for a turn that builds every item.
• A turn waits 60 seconds for the report to cover its commit. If it does not, the message continues in a new turn. After 30 minutes in all, the message ends as not checked. Nothing is filed or withdrawn, and the state counts it as unchecked. The next change checks again.
• The check workflow skips a turn that failed for good (on_exhausted: skip). The next message runs the whole check again.
• values.extract does not clean up after a deleted file. Its ValueSource entities stay, and so do the open value items of the deleted file’s instance. Answer or withdraw those items by hand.