Check model quality (experimental)

LBB calls models for its own work on your graph. A rerank model orders search hits. A router and an answer model answer questions from the graph. The fit maps integration streams to your classes. Model checks show how often those calls are right.

A judge model checks a sample of the calls. You can agree with a check or correct it. Your correction becomes the ground truth of the call, and the judge’s own verdict stays beside it. The share of reviews that agree with the judge is the judge’s own score.

In the console, model checks are on the Evals pages.

LBB runs the judge with its own key. A check sends the call’s input and output to the judge’s provider (Anthropic for Claude Opus 5.5). They hold text from your graph: search texts and hits, questions, and samples of the connected source. LBB keeps the calls and the checks for the graph, and deletes them when you delete the graph.

What gets checked

Job	Model	What the judge does
Search rerank	Jev	grades every hit 0 to 3; the score is the nDCG@10 of the model’s order
Query routing	Jev	answers the same choice; right when it picks the same option
Question answers	Claude Sonnet 5.5	rates the answer against the question and the rows it read; writes the right answer or query when it is wrong
Query rewriting	Claude Sonnet 5.5	rates the query, and writes a corrected one when it is wrong
Ontology fit	Jev	answers the same choice questions
Ontology proposals	Claude Sonnet 5.5	rates the proposed changes
Eval labels	the eval judge	grades every labeled result 0 to 3
Document mapping	Jev	answers the same choice questions (when the mapping workflow runs)

The calls of your workflows are logged and not checked: the definitions, mentions and links of documents, the values from text, and the calls of a workflow’s own code. The Models page counts them in its Workflows group, per workflow and job, with their calls, tokens, cost and model.

A score of 0.8 or more is right, below 0.5 is wrong, and between is partly. Calls are sampled per graph and day, under a daily budget per environment.

In the console

Open Evals. The overview sums the month for the selected graph: the calls checked, the share right, the reviews, the judge’s agreement with people, and the cost of the checks. Each job has a row; a row opens its checks. The month, cost and cards load independently. Cost shows … while loading and n/a when unavailable. A past month’s results can appear while the current month is still loading.

On Checks, filter by job, or show the wrong, the corrected or the not reviewed checks. A routing row reads “lookup ✓” or “lookup → history”: the model’s pick, then the judge’s when it differs. Open a check to see the model and the judge side by side:

• a routing or fit call: the model’s pick and the judge’s pick, each with how sure it is, and both probabilities of every option as bars;
• a rerank call: the hits in the model’s order, each with the model’s score and the judge’s grade, and how close the order is to the best one;
• a question’s answer: the model’s answer beside the judge’s, the steps the model took, its query beside the judge’s, and the rows it read;
• a rewrite: the model’s query beside the judge’s corrected query.

The judge’s reason follows in a few lines. Details holds the whole call: its prompt, its input and output, and the judge’s cost.

Agree keeps the judge’s verdict. Correct sets the right verdict. For a rerank call you can grade each hit again, and for a choice you can pick the right option. Your grades also count in search tuning.

All calls lists every logged call of the month. Check asks the judge to check one now.

Checks and calls load independently of the month’s summary. You can open and filter the rows while the summary is loading; if the summary fails, its error appears above the list and the rows remain usable.

Answer in Review

The checks also wait in Review, the inbox for every task a person does. Each check is one item: the call, the judge’s verdict with its reason, and Agree, the two other verdicts or Can’t tell. Your answer there is the same review as on the Checks page, and a review on the Checks page answers the item. Review holds up to 100 open checks per graph. The rest wait, and come in as you answer, the oldest first. The Checks page shows how many are open and links to them.

Calibration questions and golden query labels come into Review the same way.

The Models page shows the judge’s mean score per model: “94% score” on a model’s row, and the judge’s agreement on its own row. The score is not the share right. For a choice, such as a routing or a mapping answer, the score is the judge’s probability for the model’s answer. The verdict is right when the judge’s own first choice is the model’s answer.

Tune search

Search tuning runs a session on the graph’s own searches. The judge proposes search settings (the rerank, how many candidates it reads, the blend of rerank and similarity, and how many clusters a search reads), runs them, grades the hits, and scores each variant with nDCG@10, MRR@10 and recall@10. The best variant then runs against the current settings on searches it did not see.

A session proposes settings only when they rank better and the 95% interval of the gain is above zero. The proposal names the settings, the gain, and the change of latency and cost per search. Nothing changes until you press Apply.

Measure a workflow gate

Three document workflows send their unsure answers to the inbox: Link documents, Review facts and Values from text. An answer at or above the workflow’s gate (0.8 by default) is written at once. An answer below it becomes a review item. Each time you accept or dismiss one, you tell LBB whether the model’s answer was right at that confidence.

Evals → Workflow gates counts those decisions per confidence band below each gate (0 to 0.5, then 0.1 wide): how many, how many were right, the precision and its 95% interval. Then it tries the gates below the current one, 0.05 apart, from the highest down. A gate passes when the decisions from it up to the current gate number at least 30, are right at least as often as the target precision (95% by default), and have their interval start at most 10 points below the target. The decisions just above it (the next 0.1) must also hold one decision, and from 10 decisions be right at least the target less 10 points of the time, so a bad region does not hide under good decisions above it. The card proposes the lowest gate that passes and shows, for each gate, what holds it.

The inbox only sees answers below the gate, so the page cannot measure the bands above it, and a proposal only lowers a gate, never below 0.5. Measure above the gate on a card starts a calibration session of that gate, which checks the answers above the gate too (see below). For Link documents, an item where the model chose no entry counts as right when you dismiss it: nothing is written, as the gate would do.

Nothing changes until you press Apply and confirm. You can pick the proposed gate or a higher one. The workflow then reads its instances again under the new gate, and its row on the Workflows page shows “gate 0.65 · 41 decisions · measured 2026-10-12”. An API key cannot apply a gate.

Test another model

A model trial tests another model on the jobs above. It answers the calls the judge already checked again with that model, for example Claude Haiku 5.5 for question answers or query routing. Then it scores both models against the same ground truth: your corrections where you made them, the judge’s verdicts otherwise.

• A routing or fit call scores 1 when the model picks the right option.
• A rerank or label call scores the nDCG@10 of the model’s order under the ground-truth grades.
• A question’s answer is graded by the judge, with the known right answer as its reference. The new model’s queries read the same commit as the first answer, when the graph still has it.

In the console, open Models, then Model choice. Each use of a model (question answers, query routing, search rerank, …) has a card with its current model. Pick another model (and its effort) in the card’s model menu to test it. The trial runs by itself: it answers the checked calls, newest first, and adds each new check of that use for 14 days or until it has 40 calls. The card’s chart shows every model’s score against its cost per call, with the 95% interval of each tested model. Pick Time above the chart to see the time per call instead. Open a trial to see each call with both answers and the ground truth side by side.

A model meets the bar when at least 20 calls were compared, it is at most 5 points worse at the low end of the 95% interval, it fails at most 5 points more often, and it is cheaper or better. A trial changes no model by itself. One trial spends at most $3.00 of model and judge cost.

When a model meets the bar, Switch moves that use to the model on this graph from the next call on. Question answers switch to a Claude model and its effort, query routing to a Jev model; the other uses are tested only. Revert goes back to LBB’s model in one step.

A trial always compares a model with the model in use now. After a switch, most checked calls come from the model before. A new trial has the model in use answer those calls again, so both models answer the same calls. Trials that compared with the model before end, and their models show Test again instead of Switch.

Choose the model of a workflow step

The AI steps of workflows run two models. The Extract model reads values from text and answers the JSON prompts of a workflow’s code (the reason activity). The Decide model answers choice questions: whether the text supports a value, and the decide activity. LBB runs Claude Sonnet 5.5 for Extract and Jev for Decide. Embed keeps its model, since vectors of two models do not compare.

You can choose another model for the whole environment, or for one workflow. A workflow takes its own choice first, then the environment’s, then LBB’s model. Each model call says in the call log which choice it followed (choice).

These calls have no checked calls, so a trial cannot test them. Test the model on the workflow’s own inputs instead:

1. Open the workflow, then Spend. Models by step lists each step with its model, where the model comes from, and this month’s calls and cost.
2. Change a step. Pick a model and its effort. The dialog says how the price per token moves, for example “about 20 times cheaper per token”.
3. Test first runs the workflow on 6 inputs with that model. It writes nothing. Compare its values with an earlier test on the Test tab.
4. Set for this workflow makes the next step run the model. Use the environment’s goes back in one step.

For the environment, open Models, then Model choice, and pick a step under Workflow steps. Its card lists the workflows that run a model of their own.

A change takes effect from the next step of a workflow. Other servers take it within 30 seconds. A cheaper model lowers the cost per call from its first call, and the budgets count each call at its own price.

Calibrate thresholds and workflow gates

Jev answers each routing question and each fit question with a probability. A threshold sets which answers LBB takes at once. Ask takes the router’s route when its confidence is at least the threshold; below it, the answer loop reads only the probabilities. Both jobs start at 0.70, a value nobody measured on your data.

The document workflows use a gate the same way. Link documents writes a link, Values from text a value and Review facts a mapped fact when its confidence is at least the workflow’s gate (0.80 by default). Below the gate the decision waits in the inbox for a person. Nobody sees the decisions above the gate.

A calibration session measures, on the graph’s own calls, how often the model is right at each confidence level. A judge checks a sample of the calls, and you check a part of the judge’s verdicts. The session then proposes a threshold for a target precision. For a workflow, the sample holds decisions above the gate too, so the session tells you how often the decisions written without a person are right.

Open Evals, then Calibration. Query routing and Ontology fit each have a card with the threshold that applies now, whether people validated it, and the last session. Below them, each workflow with a gate has a card with its gate and the last session that measured it. New calibration (or Calibrate on a card) asks for:

• the job, or the workflow when you start from its card;
• the target precision: 99% for the fit, 95% for routing, which writes no data. A workflow’s gate starts at its own target, 95% unless the workflow sets another. The inbox measurement of the gate uses the same target. You can change it for one session;
• the most questions to answer (40);
• a budget for the judge in US dollars (10). It does not count against the daily check budget;
• the window of calls to read, in days (30).

While the session runs, its page shows the decisions it found, sampled and checked, and the judge’s cost against the budget. Then Review shows one decision at a time: what the model read, what it chose and how sure it was, and the judge’s check with its reason. A workflow’s decision that was written without a person says so. When the two picks are the same, the card asks whether that pick is right: Yes, No or Can’t tell (keys Y, N and C). When they differ, it asks which is right: the model’s pick (key 1), the judge’s pick (key 2), Something else (key N) or Can’t tell (key C). After No or Something else you can name the right option. Disputed decisions come first, and you can change an answer later. The review asks for the answers the session still needs before it stops at its most questions. The same questions wait in Review, one item per question, and Answer on the session page opens them there; an answer in either place is the session’s answer.

The report starts with the model: how often it is right over all its decisions, and per confidence band. For a workflow it says how often the decisions above the gate are right, and how often the ones below it are. Then it says how far to trust these numbers. The judge checked the sample, and your answers check the judge: for example, people checked 16 of its verdicts and agreed with 15 (94%, with a 95% range of 72% to 99%). A session is validated when the range is at most 10 points on each side (or the most questions have answers) and starts at 80% or higher.

The table reads the thresholds from 0.95 down. At each one it shows the estimated accuracy and where its 95% interval starts, the share of decisions at or above it, the known decisions there, and the next 0.1 above it. A threshold meets the rule when at least 10 known decisions lie at or above it, they reach the target, and their interval starts at most 10 points below the target. A threshold below the target stops the walk. So does a next 0.1 that holds decisions but no known one, or that is more than 10 points below the target. The proposal is the lowest threshold that meets the rule. The workflow gates from the inbox use the same rule.

Apply writes a threshold: the proposal, a row of the table, or a value from 0.50 to 0.99. The page asks first and says what the threshold does. A session that is not validated can still be applied, with a warning. The router uses a new threshold within a minute. The integrations do not read the fit threshold yet. For a workflow, Apply sets its gate, and the workflow reads its instances again with it. The workflow keeps where its gate came from, and its row on Workflows says “from a session”. The Models pages for question answering and the fit show the same threshold, with a Calibrate link.

With the SDKs

The TypeScript and Python clients wrap trials and switches under models. wait reads the trial until it compared the checked calls.

const { trials: [trial] } = await lbb.models.trials.create({
  candidate: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" },
  jobs: ["ask"],
});
const result = await lbb.models.trials.wait(trial.id);
if (result.report.qualifies) await lbb.models.switches.create({ trial: result.id });

started = lbb.models.trials.create(
    candidate={"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"},
    jobs=["ask"],
)
trial = lbb.models.trials.wait(started.trials[0].id)
if trial.report.qualifies:
    lbb.models.switches.create(trial=trial.id)

The clients choose the model of a workflow step with models.choices, and test a model first with models on a workflow test:

await lbb.workflows.tests.create({
  name: "products",
  sample: 6,
  models: { extract: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" } },
});
await lbb.models.choices.put({
  use: "extract",
  model: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" },
  workflow: "products",
});

lbb.models.choices.put(
    use="extract",
    model={"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"},
    workflow="products",
)
lbb.models.choices.delete(use="extract", workflow="products")

The MCP server has the same: lbb_model_choice reads (choices lists the model choices), and lbb_model_choice_manage starts, stops, switches and reverts, and sets or removes a choice (put_choice, delete_choice).

Over HTTP

# The month's checks of a graph by job, and the judge's agreement
curl "$LBB_URL/v1/models/checks/summary?graph=crm" \
  -H "Authorization: Bearer $LBB_API_KEY"

# Test Claude Haiku 5.5 on question answers and query routing
curl -X POST "$LBB_URL/v1/models/trials?graph=crm" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  --data '{"candidate": {"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"}, "jobs": ["ask", "route"]}'

# Correct a check: the right verdict and a note
curl -X POST "$LBB_URL/v1/models/checks/review?graph=crm&id=$CALL_ID" \
  -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \
  --data '{"agree": false, "verdict": "wrong", "note": "Contacts are companies."}'

Route	What it does
GET /v1/models/checks/summary	the month per job and model, the judge’s agreement, today’s budget
GET /v1/models/checks	the month’s checks, newest first; job, verdict, reviewed filter
POST /v1/models/checks/review	{agree: true}, or a correction with verdict, reference and note
GET /v1/models/checks/export	the month’s checks as JSON lines: the call, the judge and the review
GET /v1/models/calls	the call log, newest first
GET /v1/models/calls/get	one call with its input, output and check
POST /v1/models/calls/check	check one call now
POST /v1/search/tuning	start a tuning session
GET /v1/search/tuning	the sessions, newest first
POST /v1/search/tuning/apply	apply a session’s proposal
GET /v1/workflows/triggered/calibration	the workflow gates measured from inbox decisions: bands, labels and a proposal
POST /v1/workflows/triggered/calibration/apply	apply a gate’s proposal (a console session only): {starter, name, gate?}
GET /v1/models/trials/options	per job: the current model, its checked calls, the models a trial can test
POST /v1/models/trials	test a model: {candidate: {provider, model, effort?}, jobs?}
GET /v1/models/trials	the trials, newest first, with their reports
GET /v1/models/trials/get	one trial with its compared calls
POST /v1/models/trials/stop	stop a trial
POST /v1/models/switches	switch a task to a trial’s model: {trial}
POST /v1/models/switches/revert	back to LBB’s model: ?job=ask
GET /v1/models/choices	the model of each workflow use: the environment’s rows, and with workflow the rows in force for one workflow
PUT /v1/models/choices	choose a model: {use, model: {provider, model, effort?}, workflow?}
DELETE /v1/models/choices	remove a choice: ?use=extract&workflow=products; answers {id, object, deleted: true}
POST /v1/models/calibration	start a calibration session: {"job": "route"}, or a workflow’s gate: {"job": "resolve", "workflow": "documents-link"} (infer for Values from text, map for Review facts)
GET /v1/models/calibration	the calibration sessions, newest first
GET /v1/models/calibration/get	one session with its questions and report
POST /v1/models/calibration/answer	answer a question: yes, no (with an optional right) or cant_tell
POST /v1/models/calibration/apply	apply the proposal, or a threshold from 0.50 to 0.99; for a workflow, its new gate
GET /v1/models/thresholds	the applied thresholds of route and fit

A rerank correction takes reference: {"grades": {"<hit id>": 0..3}}, and a choice correction reference: {"picks": {"<question id>": "<option>"}}.

From the SDKs and MCP

The TypeScript client has checks and embeddings.searchTuning. The Python client has checks and embeddings.search_tuning. See TypeScript and Python. Through MCP, lbb_evals reads the checks with checks_summary and checks, and records a review with review_check. See MCP.