# Load RDF data

Source: https://docs.littlebigbrain.com/guides/load-rdf/

This page is for data that is RDF already. Examples are Turtle or N-Triples
files, a dump from another triple store, or a public dataset. The import stores
each statement as written. Your SPARQL queries use your own predicate IRIs.

If your data is JSON, rows or documents, start with
[Load your data](https://docs.littlebigbrain.com/start/load-data/). The RDF words on this page are explained
in [core concepts](https://docs.littlebigbrain.com/start/concepts/#rdf-under-the-hood).

## The first commit selects the write model

A graph has one of two write models. Its first commit selects the model, and
the model never changes. Schema calls such as `define` and `evolve` do not
select it.

| First commit | Model (`truth`) | Later writes |
| --- | --- | --- |
| RDF import or SPARQL Update | `rdf_native` | RDF import and SPARQL Update `INSERT DATA` only |
| JSON records (commit or NDJSON import) | `property_graph` | JSON records and RDF import |

An RDF-native graph refuses the JSON record routes. Each returns `400` with the
error code `rdf_native_graph`:

| Refused on an RDF-native graph | Use instead |
| --- | --- |
| `POST /v1/graph/commit` without `replace_owned`, and `?dry_run=true` | `POST /v1/graph/import/rdf`, or SPARQL Update `INSERT DATA` |
| `POST /v1/graph/import` | `POST /v1/graph/import/rdf` |
| `POST /v1/graph/reload` | Delete the graph and load it again |
| `POST /v1/graph/retract` | Delete the graph and load it again |

`POST /v1/graph/import-jobs` accepts the upload with `202`. The job then fails
at its first commit with `import_infrastructure_failure`. A retry fails the
same way. SPARQL queries, `POST /v1/ontology/evolve` and schema publishing
work on both models.

SPARQL Update accepts `INSERT DATA` only. A writer that replaces its own
facts can do so: files, the document workflows, an accepted review item and
a commit with `replace_owned` remove what they wrote before and no longer
write. To remove other statements, delete the graph with
`POST /v1/graph/delete?graph=research&confirm=research` and load the corrected
data. Synchronous RDF import can recreate the deleted graph. If the new load
uses durable NDJSON `POST /v1/graph/import-jobs`, explicitly recreate it with
`POST /v1/graph/create?graph=research`, install the JSON-record schema, and use
a new `Idempotency-Key`. Durable submission refuses a retained deletion
tombstone with `409 graph_epoch_changed`; old jobs cannot adopt the recreated
graph. See [Bulk NDJSON ingest](https://docs.littlebigbrain.com/sdks/http-api/#bulk-ndjson-ingest).

To mix JSON records and RDF in one graph, write the JSON records first. See
[Add RDF to a JSON-records graph](#add-rdf-to-a-json-records-graph).

`GET /v1/status` reports the model as `graph_status.truth`. Before the first
commit, it reads `property_graph`.

```sh
curl --fail-with-body -sS "$LBB_URL/v1/status?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY" | jq '.graph_status.truth'
```

## Load a file

The examples load this Turtle file into a new graph named `research`. Save it
as `dataset.ttl`:

```turtle
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix dcterms: <http://purl.org/dc/terms/> .

<ada> a foaf:Person ;
  foaf:name "Ada Lovelace" .

<babbage> a foaf:Person ;
  foaf:name "Charles Babbage" .

<notes> dcterms:title "Notes on the Analytical Engine"@en ;
  dcterms:creator <ada> ;
  dcterms:references <engine> .

<engine> dcterms:title "The Analytical Engine"@en ;
  dcterms:creator <babbage> .
```

The IRIs `<ada>` and `<notes>` are relative. `base_iri` resolves them to
`https://example.org/data/ada` and so on. The first import creates the graph,
and the graph is then RDF-native.

After the import, wait until the commit is published. Queries read published
data. This usually takes a second or two.

### TypeScript

```ts
import { readFileSync } from "node:fs";
import { LbbClient } from "@littlebigbrain/client";

const lbb = new LbbClient({
  baseUrl: process.env.LBB_URL!,
  apiKey: process.env.LBB_API_KEY!,
  graph: "research",
});

const loaded = await lbb.graph("research").facts.importRdf(readFileSync("dataset.ttl", "utf8"), {
  format: "turtle",
  baseIri: "https://example.org/data/",
  idempotencyKey: "research-load-1",
});
await lbb.waitForPublished(loaded.committed_commit_seq!);
console.log(loaded.triples_read, "statements in commit", loaded.committed_commit_seq);
```

### Python

```python
import os

from lbb import LbbClient

lbb = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="research")
graph = lbb.graph("research")

with open("dataset.ttl") as f:
    loaded = graph.facts.import_rdf(
        f.read(),
        format="turtle",
        base_iri="https://example.org/data/",
        idempotency_key="research-load-1",
    )
graph.wait_for_published(loaded["committed_commit_seq"])
print(loaded["triples_read"], "statements in commit", loaded["committed_commit_seq"])
```

### cURL

```sh
curl --fail-with-body -sS -X POST \
  "$LBB_URL/v1/graph/import/rdf?graph=research&base_iri=https://example.org/data/" \
  -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: text/turtle" \
  -H "Idempotency-Key: research-load-1" \
  --data-binary @dataset.ttl

until curl -sS "$LBB_URL/v1/graph/publication-status?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY" | grep -q '"state": *"current"'; do sleep 1; done
```

On a new graph, the SDK examples print `9 statements in commit 1`.

The response counts what the import read and stored:

| Field | Meaning |
| --- | --- |
| `graph_created` | `true` when this import created the graph. |
| `lines_read` | Non-empty lines in the body. |
| `triples_read` | Statements parsed. |
| `resource_triples`, `literal_triples` | Statements whose object is an IRI or blank node, and statements whose object is a literal. |
| `duplicate_resource_triples`, `duplicate_literal_triples` | Statements repeated inside this request. Each is stored once. |
| `imported_triplets` | Statements sent to storage. |
| `batches` | Internal commits this request made. |
| `committed_commit_seq` | The last commit this request made. Wait for this one. |
| `predicate_count`, `predicates` | Each predicate IRI in the request, with its statement counts. |
| `error_count`, `errors` | Lines that failed to parse. `errors` lists the first 100 messages. |
| `published_generation` | The publication this request queued. Absent with `build=false`. |

## Formats and parameters

`POST /v1/graph/import/rdf` takes the RDF document as the request body.

| Format | `Content-Type` | `?format=` |
| --- | --- | --- |
| N-Triples | `application/n-triples` | `ntriples`, `nt`, `n-triples` |
| Turtle | `text/turtle` | `turtle`, `ttl` |
| N-Quads | `application/n-quads` | `nquads`, `nq`, `n-quads` |
| TriG | `application/trig` | `trig` |

`?format=` overrides the content type. Any other content type is read as
N-Triples. JSON-LD and RDF/XML are not supported, and `?format=jsonld` returns
`400`. Convert such files to N-Triples or Turtle first.

Named graphs are not supported. Each graph serves the default RDF graph only.
N-Quads and TriG can name a graph per statement. Such a statement is reported
in `errors` and not stored. With `strict=true`, it fails the request.

| Parameter | Default | Effect |
| --- | --- | --- |
| `graph` | `main` | The graph to write. |
| `base_iri` | none | Base IRI for relative IRIs in Turtle and TriG. |
| `strict` | `false` | `true` fails the request with `400` at the first parse error. By default, a bad line is skipped and reported in `errors`. |
| `batch` | `100000` | Statements per internal commit, from 1 to 1,000,000. |
| `build` | `true` | `false` stores the statements without queuing publication. See [Load a large dataset](#load-a-large-dataset). |
| `blank_node_scope` | none | Keeps blank nodes from different source documents apart. At most 255 ASCII letters, digits, `-`, `_`, `.`, `:` or `/`. |
| `resource_type` | `Resource` | Name of the record type the import registers in the schema. |
| `observed_at` | commit time | An RFC 3339 time stored with the imported links on a JSON-records graph. RDF-native graphs ignore it. |
| `edge_idempotency` | `skip_unchanged` | Accepts `skip_unchanged` or `append`. It does not change an RDF import. |
| `graph_uri` | none | Refused with `400`. Import into the default graph. |

The `Idempotency-Key` header is optional on this route. The SDKs send one.

A request with more statements than `batch` makes several internal commits. It
is not one transaction. With `strict=true`, batches before the error stay
committed.

## What the import stores

The import stores each statement as written. Queries use your predicate IRIs,
such as `foaf:name`, and your subject IRIs. A statement that is already stored
does not appear twice in query results.

The import does not add one link type to the schema per predicate. A dataset
with a million predicates keeps a small schema. The import registers a fixed
set instead. It adds the record types `Resource` and `RdfLiteral` and one
link, `RDF_TRIPLE`.

Each entry in `predicates` has these fields:

| Field | Meaning |
| --- | --- |
| `uri`, `lbb_predicate_iri` | The source predicate IRI. |
| `relation`, `lbb_relation_iri` | The fixed link, `RDF_TRIPLE`, and its IRI. |
| `resource_triples`, `literal_triples`, `triples` | Statement counts for this predicate. |

## Load a large dataset

A request can be at most 64 MiB, headers included. Send a larger dataset as
several requests.

- **N-Triples and N-Quads** have one statement per line. Split them on line
  boundaries.
- **Turtle and TriG** are documents. Each chunk must be a complete Turtle or
  TriG document, with its own prefixes.

Each request publishes its commits by default. For a load of many chunks, send
`build=false` on every chunk except the last. The last chunk then queues one
publication for the whole load.

A `build=false` stream still starts publication in one case. That is when 8
commits, or 32 MiB of statements, wait for it.

```sh
split -l 200000 dataset.nt chunk-
set -- chunk-*
n=0
for file in "$@"; do
  n=$((n + 1))
  build=false; [ "$n" -eq "$#" ] && build=true
  curl --fail-with-body -sS -X POST "$LBB_URL/v1/graph/import/rdf?graph=research&build=$build" \
    -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/n-triples" \
    -H "Idempotency-Key: research-bulk-$file" \
    --data-binary @"$file"
done
```

Pick a line count that keeps each file under 64 MiB.

The SDK helpers do the same. They send `build=false` on every document except
the last. They add `:1`, `:2` and so on to your idempotency key.

### TypeScript

```ts
import { readdirSync } from "node:fs";

const chunks = readdirSync(".").filter((file) => file.startsWith("chunk-")).sort();
const bulk = await lbb.graph("research").facts.importRdfMany(
  chunks.map((file) => ({ rdf: readFileSync(file, "utf8") })),
  { format: "ntriples", idempotencyKey: "research-bulk-1" },
);
await lbb.waitForPublished(bulk.finalSequence!, { timeoutMs: 600_000 });
console.log(bulk.imports.length, "chunks, last commit", bulk.finalSequence);
```

### Python

```python
from pathlib import Path

chunks = sorted(Path(".").glob("chunk-*"))
bulk = graph.facts.import_rdf_many(
    [chunk.read_text() for chunk in chunks],
    format="ntriples",
    idempotency_key="research-bulk-1",
)
graph.wait_for_published(bulk["final_sequence"], timeout=600)
print(len(bulk["imports"]), "chunks, last commit", bulk["final_sequence"])
```

These snippets continue the code from [Load a file](#load-a-file). Add them to
the end of the same file.

### Retries and idempotency keys

Give each chunk its own `Idempotency-Key`. A retry with the same key writes
nothing. It returns the first result, with the same `committed_commit_seq`.

The key identifies the request, and the body is not compared. A reused key with
a changed body also returns the first result and stores nothing new. Use a new
key for new data.

### Blank nodes across chunks

A blank node label keeps its identity across requests. Splitting one N-Triples
file keeps `_:b0` the same node in every chunk.

Two different source documents can both use `_:b0` for different nodes. Send
each document's chunks with its own `blank_node_scope`, such as
`blank_node_scope=doc-2`. The two nodes then stay distinct.

## Wait for publication

Every accepted import is stored at once. Queries by default read the latest
published commit, so a new load is visible after publication.

`GET /v1/graph/publication-status` reports progress. It works before the first
publication too.

```sh
curl --fail-with-body -sS "$LBB_URL/v1/graph/publication-status?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY"
```

| Field | Meaning |
| --- | --- |
| `state` | `current`, `queued`, `compacting`, `building`, `publishing` or `blocked`. |
| `head_seq` | The newest commit. |
| `target_seq` | The commit publication is working toward. |
| `published_seq` | The newest published commit. |
| `lag_commits` | Commits written but not yet published. |
| `current_stage` | The step in progress, when one runs. |
| `last_progress_at_micros` | When publication last made progress. |
| `retry` | `retry_after_ms`, `eventual_read_available` and a `message`. |

Wait until `state` is `current` and `published_seq` is at least your
`committed_commit_seq`. The SDKs do this with `waitForPublished(seq)` and
`wait_for_published(seq)`. Both fail on `blocked` and after 30 seconds by
default. Pass `timeoutMs` or `timeout` for a large load.

A read can also skip the wait. `consistency=strong` applies the newest writes
at once. `min_indexed_seq` sets the oldest commit a read accepts. See
[core concepts](https://docs.littlebigbrain.com/start/concepts/#when-a-read-sees-your-write).

## Query the data

Query with the IRIs from your data. This query lists each work with its
author:

```sparql
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX dcterms: <http://purl.org/dc/terms/>

SELECT ?title ?author WHERE {
  ?work dcterms:title ?title ; dcterms:creator ?person .
  ?person foaf:name ?author .
}
ORDER BY ?title
```

In the code, `worksQuery` (`works_query` in Python) holds this query text. The
snippets use the `lbb` client from [Load a file](#load-a-file).

### TypeScript

```ts
const works = await lbb.sparqlRows({ query: worksQuery });
for (const row of works.rows) console.log(row.title, "by", row.author);
```

### Python

```python
for row in lbb.sparql(works_query):
    print(row["title"], "by", row["author"])
```

### cURL

```sh
curl --fail-with-body -sS -X POST "$LBB_URL/v1/query/sparql-text?graph=research" \
  -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/json" \
  -d "{\"query\": $(jq -Rs . < works.rq)}" \
  | jq -r '.results | fromjson | .results.bindings[] | "\(.title.value) by \(.author.value)"'
```

For cURL, save the query as `works.rq`. Each example prints two lines:

```text
Notes on the Analytical Engine by Ada Lovelace
The Analytical Engine by Charles Babbage
```

SPARQL tools such as YASGUI, Protégé and RDFLib connect to the `/sparql`
endpoint. They use the stack API key. See
[Query and validate](https://docs.littlebigbrain.com/guides/sparql-and-shacl/).

## Validate RDF writes

A SHACL shape set in `reject` mode also checks RDF imports and SPARQL Update.
The platform validates each write before it is stored. A violation returns
`400`, and the graph does not change.

The check covers the records the write can affect. That includes records the
write does not name. Say a shape requires each donation's donor to have a name.
A new statement about a donor is then checked against every donation that
names that donor.

The cost follows the size of the write. A large graph does not slow a small
write.

`POST /v1/schema/publish` reports in `write_enforcement` whether each shape can
be checked this way. Two error codes belong to this check:

| Code | Cause | What to do |
| --- | --- | --- |
| `shapes_not_write_enforceable` | The shape set has an `sh:sparql` constraint or a recursive shape reference. The affected records of a write cannot be bounded. | Publish the shape set in `warn` mode and read the conformance report. On an RDF-native graph, publishing such a set in `reject` mode returns this code. |
| `shacl_write_validation_budget` | One write affects more than 250,000 records, or needs more than 2,000,000 index lookups to find them. | Split the write into smaller requests, or publish the shape set in `warn` mode for this load. |

In `reject` mode, a bulk load moves at the pace of publication. Say more than
64 MiB of statements wait for publication. The next write then returns a
retryable `429 strong_read_pending`. Retry after the `Retry-After` delay.

## Add RDF to a JSON-records graph

A graph built from JSON records accepts RDF import too. The statements keep
their source predicates, next to the links you declared in JSON.

This example adds a description to the `Bug` type of the
[quickstart](https://docs.littlebigbrain.com/start/quickstart/) graph:

### TypeScript

```ts
import { LbbClient } from "@littlebigbrain/client";

const support = new LbbClient({
  baseUrl: process.env.LBB_URL!,
  apiKey: process.env.LBB_API_KEY!,
  graph: "support",
});
const added = await support.graph("support").facts.importRdf(
  '<https://littlebigbrain.com/class/bug> <http://www.w3.org/2000/01/rdf-schema#comment> "A defect in the product." .\n',
  { idempotencyKey: "support-rdf-1" },
);
console.log("commit", added.committed_commit_seq);
```

### Python

```python
import os

from lbb import LbbClient

support = LbbClient(os.environ["LBB_URL"], api_key=os.environ["LBB_API_KEY"], graph="support")
added = support.graph("support").facts.import_rdf(
    '<https://littlebigbrain.com/class/bug> <http://www.w3.org/2000/01/rdf-schema#comment> "A defect in the product." .\n',
    idempotency_key="support-rdf-1",
)
print("commit", added["committed_commit_seq"])
```

### cURL

```sh
curl --fail-with-body -sS -X POST "$LBB_URL/v1/graph/import/rdf?graph=support" \
  -H "Authorization: Bearer $LBB_API_KEY" -H "Content-Type: application/n-triples" \
  -H "Idempotency-Key: support-rdf-1" \
  --data-binary '<https://littlebigbrain.com/class/bug> <http://www.w3.org/2000/01/rdf-schema#comment> "A defect in the product." .'
```

The graph stays `property_graph`. JSON commits keep working after the import.

The import adds `Resource`, `RdfLiteral` and `RDF_TRIPLE` to the graph's
schema. A later `define` without them fails with
`400 ontology_identity_breaking_change`. Use `evolve` for later schema changes.

## Errors

| Response | Cause |
| --- | --- |
| `400 rdf_native_graph` | A JSON record route on a graph whose first write was RDF. |
| `400 reserved_namespace` | A statement names `https://littlebigbrain.com/s/…` as its subject, or a predicate in `https://littlebigbrain.com/ns#` or `https://littlebigbrain.com/ep/`. The platform writes those for link evidence and link properties. Nothing was stored. |
| `400 invalid_request` | An empty body, an unsupported format, `graph_uri`, an unknown `edge_idempotency` value, or a parse error with `strict=true`. |
| `400` with `request exceeds 67108864 byte limit` | The request is larger than 64 MiB. Split it. |
| `400` with `schema validation failed in reject mode` | The write breaks a `reject` shape. Nothing was stored. |
| `400 shapes_not_write_enforceable`, `400 shacl_write_validation_budget` | See [Validate RDF writes](#validate-rdf-writes). |
| `429` with `Retry-After` | The platform is busy, or publication is catching up. Retry after the delay. |

## Related

- [Query and validate](https://docs.littlebigbrain.com/guides/sparql-and-shacl/): SPARQL, SHACL shapes and
  the conformance report.
- [Reasoning & inference](https://docs.littlebigbrain.com/guides/reasoning/): answer with facts that follow
  from your schema.
- [Replay answers and history](https://docs.littlebigbrain.com/guides/time-travel-audit/): read a graph as
  it was at an earlier commit.
- [HTTP API](https://docs.littlebigbrain.com/sdks/http-api/): every route and parameter.