# Add links with an LLM Source: https://docs.littlebigbrain.com/guides/add-links-with-an-llm/ Some facts live only in text. A manual says which account a booking rule is for. An advisory says which library a bug is in. An LLM can read each section and write those facts as links. This guide continues [Load your data](https://docs.littlebigbrain.com/start/load-data/). It uses the same graph, `kb`, the same client and the same files. ## How it works 1. You send the model one section and the records it may link to. 2. The model answers with links as JSON. 3. Your code drops each link to a record you do not have. 4. A dry run checks the rest against the schema. Then you write them. The schema refuses unknown types, links and fields. It cannot know which records your model meant. Step 3 checks that. ## Pick your use case The examples follow the use case you pick. The choice is the same as on [Load your data](https://docs.littlebigbrain.com/start/load-data/#pick-your-use-case). ### Support **Your sources.** A ticket database and your release notes. **The question.** Which release fixed the login bug? **The graph.** ```text Ticket → REPORTED_AS → Bug ← COVERS ← Section → PART_OF → Document ``` ### Accounting **Your sources.** A ledger database and your accounting manual. **The question.** Why was JE-1042 booked to account 6815? **The graph.** ```text JournalEntry → BOOKED_TO → Account ← COVERS ← Section → PART_OF → Document ``` ### Legal **Your sources.** A contract database and the signed contract files. **The question.** Which contracts need consent after a merger? **The graph.** ```text Party ← HAS_PARTY ← Contract ← COVERS ← Section → PART_OF → Document ``` ### Life sciences **Your sources.** A batch record database and your deviation reports. **The question.** Which batches were made from the lot that failed testing? **The graph.** ```text Batch → MADE_FROM → Lot ← COVERS ← Section → PART_OF → Document ``` ### Security **Your sources.** A service catalog and your security advisories. **The question.** Which services are exposed to the log4j bug? **The graph.** ```text Service → DEPENDS_ON → Library ← COVERS ← Section → PART_OF → Document ``` ## 1. Ask the model Send one prompt per section. The prompt names the link, the records the link may point to, and the section: ```text You link document sections to records in a knowledge graph. Add a COVERS link from the section to each record below that the section states a rule or a fact about. Use only these records: Section key: Section name: Section text: Answer with JSON only, in this form: {"triplets": [{"source": {"type": "Section", "key": "", "name": ""}, "relation": "COVERS", "target": {"type": "", "key": "", "name": ""}, "evidence": ""}]} Answer {"triplets": []} when the section is about none of these records. ``` `candidates.json` lists the records the link may point to. Your database already has them, so export them from there. Use any model that can answer with JSON. Collect the answers for all sections in one file, `links.json`. A model can make mistakes. In each example below, one link points to a record that does not exist: ### Support ```json title="candidates.json" [ { "type": "Bug", "key": "AUTH-112", "name": "session token expiry" }, { "type": "Bug", "key": "PDF-31", "name": "PDF fonts missing" } ] ``` ```json title="links.json" { "triplets": [ { "source": { "type": "Section", "key": "release-notes-3.2#1", "name": "Sign-in fixes" }, "relation": "COVERS", "target": { "type": "Bug", "key": "AUTH-112", "name": "session token expiry" }, "evidence": "release-notes-3.2#1" }, { "source": { "type": "Section", "key": "release-notes-3.2#2", "name": "Faster exports" }, "relation": "COVERS", "target": { "type": "Bug", "key": "EXPORT-7", "name": "export runs on one thread" }, "evidence": "release-notes-3.2#2" } ] } ``` The last link is wrong: EXPORT-7 is not in your bugs table. ### Accounting ```json title="candidates.json" [ { "type": "Account", "key": "6815", "name": "Low-value office equipment" }, { "type": "Account", "key": "4400", "name": "Software subscriptions" } ] ``` ```json title="links.json" { "triplets": [ { "source": { "type": "Section", "key": "accounting-manual#3.2", "name": "3.2 Low-value equipment" }, "relation": "COVERS", "target": { "type": "Account", "key": "6815", "name": "Low-value office equipment" }, "evidence": "accounting-manual#3.2" }, { "source": { "type": "Section", "key": "accounting-manual#3.4", "name": "3.4 Software" }, "relation": "COVERS", "target": { "type": "Account", "key": "4400", "name": "Software subscriptions" }, "evidence": "accounting-manual#3.4" }, { "source": { "type": "Section", "key": "accounting-manual#3.4", "name": "3.4 Software" }, "relation": "COVERS", "target": { "type": "Account", "key": "1790", "name": "Prepaid expenses" }, "evidence": "accounting-manual#3.4" } ] } ``` The last link is wrong: account 1790 is not in your accounts table. ### Legal ```json title="candidates.json" [ { "type": "Contract", "key": "MSA-204", "name": "MSA-204: master services agreement" }, { "type": "Contract", "key": "NDA-311", "name": "NDA-311: mutual NDA" } ] ``` ```json title="links.json" { "triplets": [ { "source": { "type": "Section", "key": "MSA-204.pdf#14", "name": "14. Assignment" }, "relation": "COVERS", "target": { "type": "Contract", "key": "MSA-204", "name": "MSA-204: master services agreement" }, "evidence": "MSA-204.pdf#14" }, { "source": { "type": "Section", "key": "MSA-204.pdf#15", "name": "15. Term" }, "relation": "COVERS", "target": { "type": "Contract", "key": "MSA-204", "name": "MSA-204: master services agreement" }, "evidence": "MSA-204.pdf#15" }, { "source": { "type": "Section", "key": "MSA-204.pdf#14", "name": "14. Assignment" }, "relation": "COVERS", "target": { "type": "Contract", "key": "MSA-205", "name": "MSA-205: master services agreement" }, "evidence": "MSA-204.pdf#14" } ] } ``` The last link is wrong: MSA-205 is not in your contracts table. ### Life sciences ```json title="candidates.json" [ { "type": "Lot", "key": "73091", "name": "Lot 73091" }, { "type": "Lot", "key": "73112", "name": "Lot 73112" } ] ``` ```json title="links.json" { "triplets": [ { "source": { "type": "Section", "key": "DR-77#2", "name": "2. Finding" }, "relation": "COVERS", "target": { "type": "Lot", "key": "73091", "name": "Lot 73091" }, "evidence": "DR-77#2" }, { "source": { "type": "Section", "key": "DR-77#3", "name": "3. Action" }, "relation": "COVERS", "target": { "type": "Lot", "key": "73090", "name": "Lot 73090" }, "evidence": "DR-77#3" } ] } ``` The last link is wrong: lot 73090 is not in your lots table. ### Security ```json title="candidates.json" [ { "type": "Library", "key": "log4j-core", "name": "log4j-core" }, { "type": "Library", "key": "jackson-databind", "name": "jackson-databind" } ] ``` ```json title="links.json" { "triplets": [ { "source": { "type": "Section", "key": "SA-2021-44#1", "name": "Affected versions" }, "relation": "COVERS", "target": { "type": "Library", "key": "log4j-core", "name": "log4j-core" }, "evidence": "SA-2021-44#1" }, { "source": { "type": "Section", "key": "SA-2021-44#2", "name": "Fix" }, "relation": "COVERS", "target": { "type": "Library", "key": "log4j-core", "name": "log4j-core" }, "evidence": "SA-2021-44#2" }, { "source": { "type": "Section", "key": "SA-2021-44#2", "name": "Fix" }, "relation": "COVERS", "target": { "type": "Library", "key": "log4j-api", "name": "log4j-api" }, "evidence": "SA-2021-44#2" } ] } ``` The last link is wrong: log4j-api is not in your service catalog. ## 2. Drop links to unknown records A link to an unknown `key` creates a new record. So keep only the links whose target is in `candidates.json`: ### TypeScript ```ts const known = new Set(read("candidates.json").map((r: any) => `${r.type}/${r.key}`)); const links = read("links.json"); links.triplets = links.triplets.filter((t: any) => known.has(`${t.target.type}/${t.target.key}`), ); console.log(links.triplets.length, "links kept"); ``` ### Python ```python known = {(r["type"], r["key"]) for r in read("candidates.json")} links = read("links.json") links["triplets"] = [ t for t in links["triplets"] if (t["target"]["type"], t["target"]["key"]) in known ] print(len(links["triplets"]), "links kept") ``` ### cURL ```sh jq --slurpfile known candidates.json \ '.triplets |= map(select(.target as $t | $known[0] | any(.type == $t.type and .key == $t.key)))' \ links.json > checked.json jq '.triplets | length' checked.json ``` ## 3. Check, then write A dry run checks the links against the schema and writes nothing. It fails like a real write on an unknown type, link or field. Say the model names a link `EXPLAINS`. The error lists the links your schema allows: ### Support ```text unknown relation 'EXPLAINS'; allowed relations: REPORTED_AS, PART_OF, COVERS ``` ### Accounting ```text unknown relation 'EXPLAINS'; allowed relations: BOOKED_TO, PART_OF, COVERS ``` ### Legal ```text unknown relation 'EXPLAINS'; allowed relations: HAS_PARTY, PART_OF, COVERS ``` ### Life sciences ```text unknown relation 'EXPLAINS'; allowed relations: MADE_FROM, PART_OF, COVERS ``` ### Security ```text unknown relation 'EXPLAINS'; allowed relations: DEPENDS_ON, PART_OF, COVERS ``` Send such an error back to the model and ask again. When the dry run passes, it reports the number of links it would write. Then write them: ### TypeScript ```ts await lbb.commitDryRun(links); const written = await lbb.graph("kb").facts.create(links, { idempotencyKey: "kb-links-1" }); await lbb.waitForPublished(written.commit_seq); ``` ### Python ```python lbb.commit_dry_run(links) written = lbb.graph("kb").facts.create(links, idempotency_key="kb-links-1") lbb.wait_for_published(written["commit_seq"]) ``` ### cURL ```sh curl --fail-with-body -sS -X POST "$LBB_URL/v1/graph/commit?graph=kb&dry_run=true" \ -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/json" \ -d @checked.json curl --fail-with-body -sS -X POST "$LBB_URL/v1/graph/commit?graph=kb" \ -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/json" \ -H "Idempotency-Key: kb-links-1" -d @checked.json until curl -sS "$LBB_URL/v1/graph/publication-status?graph=kb" \ -H "Authorization: Bearer $LBB_API_KEY" | grep -q '"state": *"current"'; do sleep 1; done ``` Each link keeps the section key in `evidence`. An answer can then cite the section it came from. ## 4. Follow the path Your database records and your document sections are one graph now. This query starts at a record and follows its links to the section that explains it: ### Support ```sparql title="path.rq" PREFIX dclass: PREFIX dr: PREFIX dp: PREFIX rdfs: SELECT ?ticket ?bug ?section ?document WHERE { ?t a dclass:ticket ; rdfs:label ?ticket ; dr:reported_as ?b . ?b rdfs:label ?bug . ?s dr:covers ?b ; rdfs:label ?section ; dr:part_of ?d . ?d rdfs:label ?document . } ``` The result has 1 row: | ticket | bug | section | document | | --- | --- | --- | --- | | `login fails after 3.1` | `session token expiry` | `Sign-in fixes` | `Release notes 3.2` | ### Accounting ```sparql title="path.rq" PREFIX dclass: PREFIX dr: PREFIX dp: PREFIX rdfs: SELECT ?entry ?account ?rule ?text WHERE { ?e a dclass:journalentry ; rdfs:label ?entry ; dr:booked_to ?a . ?a rdfs:label ?account . ?s dr:covers ?a ; rdfs:label ?rule ; dp:text ?text . } ORDER BY ?entry ``` The result has 2 rows: | entry | account | rule | text | | --- | --- | --- | --- | | `JE-1042: office chairs` | `Low-value office equipment` | `3.2 Low-value equipment` | Book office equipment that costs less than 800 EUR per item to account 6815. Expense it in the month of purchase. Keep the receipt with the entry. | | `JE-1043: design software` | `Software subscriptions` | `3.4 Software` | Book software subscriptions to account 4400. Spread an annual plan over twelve months. | ### Legal ```sparql title="path.rq" PREFIX dclass: PREFIX dr: PREFIX dp: PREFIX rdfs: SELECT ?contract ?party ?clause WHERE { ?c a dclass:contract ; rdfs:label ?contract ; dr:has_party ?p . ?p rdfs:label ?party . ?s dr:covers ?c ; rdfs:label ?clause ; dp:text ?text . FILTER(CONTAINS(LCASE(?text), "change of control")) } ``` The result has 1 row: | contract | party | clause | | --- | --- | --- | | `MSA-204: master services agreement` | `Meridian Cloud` | `14. Assignment` | ### Life sciences ```sparql title="path.rq" PREFIX dclass: PREFIX dr: PREFIX dp: PREFIX rdfs: SELECT ?batch ?lot ?finding WHERE { ?b a dclass:batch ; rdfs:label ?batch ; dr:made_from ?l . ?l rdfs:label ?lot . ?s dr:covers ?l ; rdfs:label ?finding . } ``` The result has 1 row: | batch | lot | finding | | --- | --- | --- | | `Batch CVX-A17` | `Lot 73091` | `2. Finding` | ### Security ```sparql title="path.rq" PREFIX dclass: PREFIX dr: PREFIX dp: PREFIX rdfs: SELECT ?service ?library ?advisory WHERE { ?svc a dclass:service ; rdfs:label ?service ; dr:depends_on ?lib . ?lib rdfs:label ?library . ?s dr:covers ?lib ; rdfs:label ?advisory . } ORDER BY ?advisory ``` The result has 2 rows: | service | library | advisory | | --- | --- | --- | | `Checkout API` | `log4j-core` | `Affected versions` | | `Checkout API` | `log4j-core` | `Fix` | Run the query: ### TypeScript ```ts const { rows } = await lbb.sparqlRows({ query: readFileSync("path.rq", "utf8") }); console.log(rows); ``` ### Python ```python with open("path.rq") as f: for row in lbb.sparql(f.read()): print(row) ``` ### cURL ```sh curl --fail-with-body -sS -G "$LBB_URL/sparql?graph=kb" \ -H "Authorization: Bearer $LBB_API_KEY" -H "Accept: text/csv" \ --data-urlencode query@path.rq ``` Search finds a section by meaning. This query then proves which records it is about. [How search works](https://docs.littlebigbrain.com/start/platform/) shows the two steps together. ## Run it again The same files give the same links. A rerun with the same idempotency key returns the first write and stores nothing new. When a document changes, load its sections again and ask the model again. Use a new idempotency key for the new links. ## Related - [Load your data](https://docs.littlebigbrain.com/start/load-data/): the schema, the database and the documents. - [Search by meaning](https://docs.littlebigbrain.com/guides/search-by-meaning/): filter hits by type or by a link such as `COVERS`. - [Query with SPARQL](https://docs.littlebigbrain.com/guides/sparql-and-shacl/): filters, aggregates and longer paths.