Skip to content
little big braindocs

Load RDF data

View Markdown

This page is for data that is RDF already. Examples are Turtle or N-Triples files, a dump from another triple store, or a public dataset. The import stores each statement as written. Your SPARQL queries use your own predicate IRIs.

If your data is JSON, rows or documents, start with Load your data. The RDF words on this page are explained in core concepts.

The first commit selects the write model

Section titled “The first commit selects the write model”

A graph has one of two write models. Its first commit selects the model, and the model never changes. Schema calls such as define and evolve do not select it.

First commit Model (truth) Later writes
RDF import or SPARQL Update rdf_native RDF import and SPARQL Update INSERT DATA only
JSON records (commit or NDJSON import) property_graph JSON records and RDF import

An RDF-native graph refuses the JSON record routes. Each returns 400 with the error code rdf_native_graph:

Refused on an RDF-native graph Use instead
POST /v1/graph/commit without replace_owned, and ?dry_run=true POST /v1/graph/import/rdf, or SPARQL Update INSERT DATA
POST /v1/graph/import POST /v1/graph/import/rdf
POST /v1/graph/reload Delete the graph and load it again
POST /v1/graph/retract Delete the graph and load it again

POST /v1/graph/import-jobs accepts the upload with 202. The job then fails at its first commit with import_infrastructure_failure. A retry fails the same way. SPARQL queries, POST /v1/ontology/evolve and schema publishing work on both models.

SPARQL Update accepts INSERT DATA only. A writer that replaces its own facts can do so: files, the document workflows, an accepted review item and a commit with replace_owned remove what they wrote before and no longer write. To remove other statements, delete the graph with POST /v1/graph/delete?graph=research&confirm=research and load the corrected data. Synchronous RDF import can recreate the deleted graph. If the new load uses durable NDJSON POST /v1/graph/import-jobs, explicitly recreate it with POST /v1/graph/create?graph=research, install the JSON-record schema, and use a new Idempotency-Key. Durable submission refuses a retained deletion tombstone with 409 graph_epoch_changed; old jobs cannot adopt the recreated graph. See Bulk NDJSON ingest.

To mix JSON records and RDF in one graph, write the JSON records first. See Add RDF to a JSON-records graph.

GET /v1/status reports the model as graph_status.truth. Before the first commit, it reads property_graph.

Terminal window
curl --fail-with-body -sS "$LBB_URL/v1/status?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY" | jq '.graph_status.truth'

Load a file

Section titled “Load a file”

The examples load this Turtle file into a new graph named research. Save it as dataset.ttl:

@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix dcterms: <http://purl.org/dc/terms/> .


<ada> a foaf:Person ;
  foaf:name "Ada Lovelace" .


<babbage> a foaf:Person ;
  foaf:name "Charles Babbage" .


<notes> dcterms:title "Notes on the Analytical Engine"@en ;
  dcterms:creator <ada> ;
  dcterms:references <engine> .


<engine> dcterms:title "The Analytical Engine"@en ;
  dcterms:creator <babbage> .

The IRIs <ada> and <notes> are relative. base_iri resolves them to https://example.org/data/ada and so on. The first import creates the graph, and the graph is then RDF-native.

After the import, wait until the commit is published. Queries read published data. This usually takes a second or two.

import { readFileSync } from "node:fs";
import { LbbClient } from "@littlebigbrain/client";


const lbb = new LbbClient({
  baseUrl: process.env.LBB_URL!,
  apiKey: process.env.LBB_API_KEY!,
  graph: "research",
});


const loaded = await lbb.graph("research").facts.importRdf(readFileSync("dataset.ttl", "utf8"), {
  format: "turtle",
  baseIri: "https://example.org/data/",
  idempotencyKey: "research-load-1",
});
await lbb.waitForPublished(loaded.committed_commit_seq!);
console.log(loaded.triples_read, "statements in commit", loaded.committed_commit_seq);

On a new graph, the SDK examples print 9 statements in commit 1.

The response counts what the import read and stored:

Field Meaning
graph_created true when this import created the graph.
lines_read Non-empty lines in the body.
triples_read Statements parsed.
resource_triples, literal_triples Statements whose object is an IRI or blank node, and statements whose object is a literal.
duplicate_resource_triples, duplicate_literal_triples Statements repeated inside this request. Each is stored once.
imported_triplets Statements sent to storage.
batches Internal commits this request made.
committed_commit_seq The last commit this request made. Wait for this one.
predicate_count, predicates Each predicate IRI in the request, with its statement counts.
error_count, errors Lines that failed to parse. errors lists the first 100 messages.
published_generation The publication this request queued. Absent with build=false.

Formats and parameters

Section titled “Formats and parameters”

POST /v1/graph/import/rdf takes the RDF document as the request body.

Format Content-Type ?format=
N-Triples application/n-triples ntriples, nt, n-triples
Turtle text/turtle turtle, ttl
N-Quads application/n-quads nquads, nq, n-quads
TriG application/trig trig

?format= overrides the content type. Any other content type is read as N-Triples. JSON-LD and RDF/XML are not supported, and ?format=jsonld returns 400. Convert such files to N-Triples or Turtle first.

Named graphs are not supported. Each graph serves the default RDF graph only. N-Quads and TriG can name a graph per statement. Such a statement is reported in errors and not stored. With strict=true, it fails the request.

Parameter Default Effect
graph main The graph to write.
base_iri none Base IRI for relative IRIs in Turtle and TriG.
strict false true fails the request with 400 at the first parse error. By default, a bad line is skipped and reported in errors.
batch 100000 Statements per internal commit, from 1 to 1,000,000.
build true false stores the statements without queuing publication. See Load a large dataset.
blank_node_scope none Keeps blank nodes from different source documents apart. At most 255 ASCII letters, digits, -, _, ., : or /.
resource_type Resource Name of the record type the import registers in the schema.
observed_at commit time An RFC 3339 time stored with the imported links on a JSON-records graph. RDF-native graphs ignore it.
edge_idempotency skip_unchanged Accepts skip_unchanged or append. It does not change an RDF import.
graph_uri none Refused with 400. Import into the default graph.

The Idempotency-Key header is optional on this route. The SDKs send one.

A request with more statements than batch makes several internal commits. It is not one transaction. With strict=true, batches before the error stay committed.

What the import stores

Section titled “What the import stores”

The import stores each statement as written. Queries use your predicate IRIs, such as foaf:name, and your subject IRIs. A statement that is already stored does not appear twice in query results.

The import does not add one link type to the schema per predicate. A dataset with a million predicates keeps a small schema. The import registers a fixed set instead. It adds the record types Resource and RdfLiteral and one link, RDF_TRIPLE.

Each entry in predicates has these fields:

Field Meaning
uri, lbb_predicate_iri The source predicate IRI.
relation, lbb_relation_iri The fixed link, RDF_TRIPLE, and its IRI.
resource_triples, literal_triples, triples Statement counts for this predicate.

Load a large dataset

Section titled “Load a large dataset”

A request can be at most 64 MiB, headers included. Send a larger dataset as several requests.

  • N-Triples and N-Quads have one statement per line. Split them on line boundaries.
  • Turtle and TriG are documents. Each chunk must be a complete Turtle or TriG document, with its own prefixes.

Each request publishes its commits by default. For a load of many chunks, send build=false on every chunk except the last. The last chunk then queues one publication for the whole load.

A build=false stream still starts publication in one case. That is when 8 commits, or 32 MiB of statements, wait for it.

Terminal window
split -l 200000 dataset.nt chunk-
set -- chunk-*
n=0
for file in "$@"; do
  n=$((n + 1))
  build=false; [ "$n" -eq "$#" ] && build=true
  curl --fail-with-body -sS -X POST "$LBB_URL/v1/graph/import/rdf?graph=research&build=$build" \
    -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/n-triples" \
    -H "Idempotency-Key: research-bulk-$file" \
    --data-binary @"$file"
done

Pick a line count that keeps each file under 64 MiB.

The SDK helpers do the same. They send build=false on every document except the last. They add :1, :2 and so on to your idempotency key.

import { readdirSync } from "node:fs";


const chunks = readdirSync(".").filter((file) => file.startsWith("chunk-")).sort();
const bulk = await lbb.graph("research").facts.importRdfMany(
  chunks.map((file) => ({ rdf: readFileSync(file, "utf8") })),
  { format: "ntriples", idempotencyKey: "research-bulk-1" },
);
await lbb.waitForPublished(bulk.finalSequence!, { timeoutMs: 600_000 });
console.log(bulk.imports.length, "chunks, last commit", bulk.finalSequence);

These snippets continue the code from Load a file. Add them to the end of the same file.

Retries and idempotency keys

Section titled “Retries and idempotency keys”

Give each chunk its own Idempotency-Key. A retry with the same key writes nothing. It returns the first result, with the same committed_commit_seq.

The key identifies the request, and the body is not compared. A reused key with a changed body also returns the first result and stores nothing new. Use a new key for new data.

Blank nodes across chunks

Section titled “Blank nodes across chunks”

A blank node label keeps its identity across requests. Splitting one N-Triples file keeps _:b0 the same node in every chunk.

Two different source documents can both use _:b0 for different nodes. Send each document’s chunks with its own blank_node_scope, such as blank_node_scope=doc-2. The two nodes then stay distinct.

Wait for publication

Section titled “Wait for publication”

Every accepted import is stored at once. Queries by default read the latest published commit, so a new load is visible after publication.

GET /v1/graph/publication-status reports progress. It works before the first publication too.

Terminal window
curl --fail-with-body -sS "$LBB_URL/v1/graph/publication-status?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY"
Field Meaning
state current, queued, compacting, building, publishing or blocked.
head_seq The newest commit.
target_seq The commit publication is working toward.
published_seq The newest published commit.
lag_commits Commits written but not yet published.
current_stage The step in progress, when one runs.
last_progress_at_micros When publication last made progress.
retry retry_after_ms, eventual_read_available and a message.

Wait until state is current and published_seq is at least your committed_commit_seq. The SDKs do this with waitForPublished(seq) and wait_for_published(seq). Both fail on blocked and after 30 seconds by default. Pass timeoutMs or timeout for a large load.

A read can also skip the wait. consistency=strong applies the newest writes at once. min_indexed_seq sets the oldest commit a read accepts. See core concepts.

Query the data

Section titled “Query the data”

Query with the IRIs from your data. This query lists each work with its author:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX dcterms: <http://purl.org/dc/terms/>


SELECT ?title ?author WHERE {
  ?work dcterms:title ?title ; dcterms:creator ?person .
  ?person foaf:name ?author .
}
ORDER BY ?title

In the code, worksQuery (works_query in Python) holds this query text. The snippets use the lbb client from Load a file.

const works = await lbb.sparqlRows({ query: worksQuery });
for (const row of works.rows) console.log(row.title, "by", row.author);

For cURL, save the query as works.rq. Each example prints two lines:

Notes on the Analytical Engine by Ada Lovelace
The Analytical Engine by Charles Babbage

SPARQL tools such as YASGUI, Protégé and RDFLib connect to the /sparql endpoint. They use the stack API key. See Query and validate.

Validate RDF writes

Section titled “Validate RDF writes”

A SHACL shape set in reject mode also checks RDF imports and SPARQL Update. The platform validates each write before it is stored. A violation returns 400, and the graph does not change.

The check covers the records the write can affect. That includes records the write does not name. Say a shape requires each donation’s donor to have a name. A new statement about a donor is then checked against every donation that names that donor.

The cost follows the size of the write. A large graph does not slow a small write.

POST /v1/schema/publish reports in write_enforcement whether each shape can be checked this way. Two error codes belong to this check:

Code Cause What to do
shapes_not_write_enforceable The shape set has an sh:sparql constraint or a recursive shape reference. The affected records of a write cannot be bounded. Publish the shape set in warn mode and read the conformance report. On an RDF-native graph, publishing such a set in reject mode returns this code.
shacl_write_validation_budget One write affects more than 250,000 records, or needs more than 2,000,000 index lookups to find them. Split the write into smaller requests, or publish the shape set in warn mode for this load.

In reject mode, a bulk load moves at the pace of publication. Say more than 64 MiB of statements wait for publication. The next write then returns a retryable 429 strong_read_pending. Retry after the Retry-After delay.

Add RDF to a JSON-records graph

Section titled “Add RDF to a JSON-records graph”

A graph built from JSON records accepts RDF import too. The statements keep their source predicates, next to the links you declared in JSON.

This example adds a description to the Bug type of the quickstart graph:

import { LbbClient } from "@littlebigbrain/client";


const support = new LbbClient({
  baseUrl: process.env.LBB_URL!,
  apiKey: process.env.LBB_API_KEY!,
  graph: "support",
});
const added = await support.graph("support").facts.importRdf(
  '<https://littlebigbrain.com/class/bug> <http://www.w3.org/2000/01/rdf-schema#comment> "A defect in the product." .\n',
  { idempotencyKey: "support-rdf-1" },
);
console.log("commit", added.committed_commit_seq);

The graph stays property_graph. JSON commits keep working after the import.

The import adds Resource, RdfLiteral and RDF_TRIPLE to the graph’s schema. A later define without them fails with 400 ontology_identity_breaking_change. Use evolve for later schema changes.

Errors

Section titled “Errors”
Response Cause
400 rdf_native_graph A JSON record route on a graph whose first write was RDF.
400 reserved_namespace A statement names https://littlebigbrain.com/s/… as its subject, or a predicate in https://littlebigbrain.com/ns# or https://littlebigbrain.com/ep/. The platform writes those for link evidence and link properties. Nothing was stored.
400 invalid_request An empty body, an unsupported format, graph_uri, an unknown edge_idempotency value, or a parse error with strict=true.
400 with request exceeds 67108864 byte limit The request is larger than 64 MiB. Split it.
400 with schema validation failed in reject mode The write breaks a reject shape. Nothing was stored.
400 shapes_not_write_enforceable, 400 shacl_write_validation_budget See Validate RDF writes.
429 with Retry-After The platform is busy, or publication is catching up. Retry after the delay.
Section titled “Related”