# Check model quality (experimental) Source: https://docs.littlebigbrain.com/guides/check-model-quality/ LBB calls models for its own work on your graph. A rerank model orders search hits. A router and an answer model answer questions from the graph. The fit maps integration streams to your classes. Model checks show how often those calls are right. A judge model checks a sample of the calls. You can agree with a check or correct it. Your correction becomes the ground truth of the call, and the judge's own verdict stays beside it. The share of reviews that agree with the judge is the judge's own score. In the console, model checks are on the **Evals** pages. LBB runs the judge with its own key. A check sends the call's input and output to the judge's provider (Anthropic for Claude Opus 5.5). They hold text from your graph: search texts and hits, questions, and samples of the connected source. LBB keeps the calls and the checks for the graph, and deletes them when you delete the graph. ## What gets checked | Job | Model | What the judge does | | --- | --- | --- | | Search rerank | Jev | grades every hit 0 to 3; the score is the nDCG@10 of the model's order | | Query routing | Jev | answers the same choice; right when it picks the same option | | Question answers | Claude Sonnet 5.5 | rates the answer against the question and the rows it read; writes the right answer or query when it is wrong | | Query rewriting | Claude Sonnet 5.5 | rates the query, and writes a corrected one when it is wrong | | Ontology fit | Jev | answers the same choice questions | | Ontology proposals | Claude Sonnet 5.5 | rates the proposed changes | | Eval labels | the eval judge | grades every labeled result 0 to 3 | | Document mapping | Jev | answers the same choice questions (when the mapping workflow runs) | The calls of your workflows are logged and not checked: the definitions, mentions and links of documents, the values from text, and the calls of a workflow's own code. The **Models** page counts them in its **Workflows** group, per workflow and job, with their calls, tokens, cost and model. A score of 0.8 or more is **right**, below 0.5 is **wrong**, and between is **partly**. Calls are sampled per graph and day, under a daily budget per environment. ## In the console Open **Evals**. The overview sums the month for the selected graph: the calls checked, the share right, the reviews, the judge's agreement with people, and the cost of the checks. Each job has a row; a row opens its checks. The month, cost and cards load independently. Cost shows `…` while loading and `n/a` when unavailable. A past month's results can appear while the current month is still loading. On **Checks**, filter by job, or show the wrong, the corrected or the not reviewed checks. A routing row reads "lookup ✓" or "lookup → history": the model's pick, then the judge's when it differs. Open a check to see the model and the judge side by side: - a routing or fit call: the model's pick and the judge's pick, each with how sure it is, and both probabilities of every option as bars; - a rerank call: the hits in the model's order, each with the model's score and the judge's grade, and how close the order is to the best one; - a question's answer: the model's answer beside the judge's, the steps the model took, its query beside the judge's, and the rows it read; - a rewrite: the model's query beside the judge's corrected query. The judge's reason follows in a few lines. **Details** holds the whole call: its prompt, its input and output, and the judge's cost. **Agree** keeps the judge's verdict. **Correct** sets the right verdict. For a rerank call you can grade each hit again, and for a choice you can pick the right option. Your grades also count in search tuning. **All calls** lists every logged call of the month. **Check** asks the judge to check one now. Checks and calls load independently of the month's summary. You can open and filter the rows while the summary is loading; if the summary fails, its error appears above the list and the rows remain usable. ### Answer in Review The checks also wait in **Review**, the inbox for every task a person does. Each check is one item: the call, the judge's verdict with its reason, and **Agree**, the two other verdicts or **Can't tell**. Your answer there is the same review as on the Checks page, and a review on the Checks page answers the item. Review holds up to 100 open checks per graph. The rest wait, and come in as you answer, the oldest first. The Checks page shows how many are open and links to them. Calibration questions and golden query labels come into Review the same way. The **Models** page shows the judge's mean score per model: "94% score" on a model's row, and the judge's agreement on its own row. The score is not the share right. For a choice, such as a routing or a mapping answer, the score is the judge's probability for the model's answer. The verdict is right when the judge's own first choice is the model's answer. ## Tune search **Search tuning** runs a session on the graph's own searches. The judge proposes search settings (the rerank, how many candidates it reads, the blend of rerank and similarity, and how many clusters a search reads), runs them, grades the hits, and scores each variant with nDCG@10, MRR@10 and recall@10. The best variant then runs against the current settings on searches it did not see. A session proposes settings only when they rank better and the 95% interval of the gain is above zero. The proposal names the settings, the gain, and the change of latency and cost per search. Nothing changes until you press **Apply**. ## Measure a workflow gate Three document workflows send their unsure answers to the inbox: **Link documents**, **Review facts** and **Values from text**. An answer at or above the workflow's gate (0.8 by default) is written at once. An answer below it becomes a review item. Each time you accept or dismiss one, you tell LBB whether the model's answer was right at that confidence. **Evals → Workflow gates** counts those decisions per confidence band below each gate (0 to 0.5, then 0.1 wide): how many, how many were right, the precision and its 95% interval. Then it tries the gates below the current one, 0.05 apart, from the highest down. A gate passes when the decisions from it up to the current gate number at least 30, are right at least as often as the target precision (95% by default), and have their interval start at most 10 points below the target. The decisions just above it (the next 0.1) must also hold one decision, and from 10 decisions be right at least the target less 10 points of the time, so a bad region does not hide under good decisions above it. The card proposes the lowest gate that passes and shows, for each gate, what holds it. The inbox only sees answers below the gate, so the page cannot measure the bands above it, and a proposal only lowers a gate, never below 0.5. **Measure above the gate** on a card starts a calibration session of that gate, which checks the answers above the gate too (see below). For **Link documents**, an item where the model chose no entry counts as right when you dismiss it: nothing is written, as the gate would do. Nothing changes until you press **Apply** and confirm. You can pick the proposed gate or a higher one. The workflow then reads its instances again under the new gate, and its row on the Workflows page shows "gate 0.65 · 41 decisions · measured 2026-10-12". An API key cannot apply a gate. ## Test another model A model trial tests another model on the jobs above. It answers the calls the judge already checked again with that model, for example Claude Haiku 5.5 for question answers or query routing. Then it scores both models against the same ground truth: your corrections where you made them, the judge's verdicts otherwise. - A routing or fit call scores 1 when the model picks the right option. - A rerank or label call scores the nDCG@10 of the model's order under the ground-truth grades. - A question's answer is graded by the judge, with the known right answer as its reference. The new model's queries read the same commit as the first answer, when the graph still has it. In the console, open **Models**, then **Model choice**. Each use of a model (question answers, query routing, search rerank, …) has a card with its current model. Pick another model (and its effort) in the card's model menu to test it. The trial runs by itself: it answers the checked calls, newest first, and adds each new check of that use for 14 days or until it has 40 calls. The card's chart shows every model's score against its cost per call, with the 95% interval of each tested model. Pick **Time** above the chart to see the time per call instead. Open a trial to see each call with both answers and the ground truth side by side. A model **meets the bar** when at least 20 calls were compared, it is at most 5 points worse at the low end of the 95% interval, it fails at most 5 points more often, and it is cheaper or better. A trial changes no model by itself. One trial spends at most $3.00 of model and judge cost. When a model meets the bar, **Switch** moves that use to the model on this graph from the next call on. Question answers switch to a Claude model and its effort, query routing to a Jev model; the other uses are tested only. **Revert** goes back to LBB's model in one step. A trial always compares a model with the model in use now. After a switch, most checked calls come from the model before. A new trial has the model in use answer those calls again, so both models answer the same calls. Trials that compared with the model before end, and their models show **Test again** instead of **Switch**. ## Choose the model of a workflow step The AI steps of workflows run two models. The **Extract** model reads values from text and answers the JSON prompts of a workflow's code (the `reason` activity). The **Decide** model answers choice questions: whether the text supports a value, and the `decide` activity. LBB runs Claude Sonnet 5.5 for Extract and Jev for Decide. **Embed** keeps its model, since vectors of two models do not compare. You can choose another model for the whole environment, or for one workflow. A workflow takes its own choice first, then the environment's, then LBB's model. Each model call says in the call log which choice it followed (`choice`). These calls have no checked calls, so a trial cannot test them. Test the model on the workflow's own inputs instead: 1. Open the workflow, then **Spend**. **Models by step** lists each step with its model, where the model comes from, and this month's calls and cost. 2. **Change** a step. Pick a model and its effort. The dialog says how the price per token moves, for example "about 20 times cheaper per token". 3. **Test first** runs the workflow on 6 inputs with that model. It writes nothing. Compare its values with an earlier test on the **Test** tab. 4. **Set for this workflow** makes the next step run the model. **Use the environment's** goes back in one step. For the environment, open **Models**, then **Model choice**, and pick a step under **Workflow steps**. Its card lists the workflows that run a model of their own. A change takes effect from the next step of a workflow. Other servers take it within 30 seconds. A cheaper model lowers the cost per call from its first call, and the budgets count each call at its own price. ## Calibrate thresholds and workflow gates Jev answers each routing question and each fit question with a probability. A threshold sets which answers LBB takes at once. Ask takes the router's route when its confidence is at least the threshold; below it, the answer loop reads only the probabilities. Both jobs start at 0.70, a value nobody measured on your data. The document workflows use a gate the same way. **Link documents** writes a link, **Values from text** a value and **Review facts** a mapped fact when its confidence is at least the workflow's gate (0.80 by default). Below the gate the decision waits in the inbox for a person. Nobody sees the decisions above the gate. A calibration session measures, on the graph's own calls, how often the model is right at each confidence level. A judge checks a sample of the calls, and you check a part of the judge's verdicts. The session then proposes a threshold for a target precision. For a workflow, the sample holds decisions above the gate too, so the session tells you how often the decisions written without a person are right. Open **Evals**, then **Calibration**. **Query routing** and **Ontology fit** each have a card with the threshold that applies now, whether people validated it, and the last session. Below them, each workflow with a gate has a card with its gate and the last session that measured it. **New calibration** (or **Calibrate** on a card) asks for: - the job, or the workflow when you start from its card; - the target precision: 99% for the fit, 95% for routing, which writes no data. A workflow's gate starts at its own target, 95% unless the workflow sets another. The inbox measurement of the gate uses the same target. You can change it for one session; - the most questions to answer (40); - a budget for the judge in US dollars (10). It does not count against the daily check budget; - the window of calls to read, in days (30). While the session runs, its page shows the decisions it found, sampled and checked, and the judge's cost against the budget. Then **Review** shows one decision at a time: what the model read, what it chose and how sure it was, and the judge's check with its reason. A workflow's decision that was written without a person says so. When the two picks are the same, the card asks whether that pick is right: **Yes**, **No** or **Can't tell** (keys Y, N and C). When they differ, it asks which is right: the model's pick (key 1), the judge's pick (key 2), **Something else** (key N) or **Can't tell** (key C). After **No** or **Something else** you can name the right option. Disputed decisions come first, and you can change an answer later. The review asks for the answers the session still needs before it stops at its most questions. The same questions wait in **Review**, one item per question, and **Answer** on the session page opens them there; an answer in either place is the session's answer. The report starts with the model: how often it is right over all its decisions, and per confidence band. For a workflow it says how often the decisions above the gate are right, and how often the ones below it are. Then it says how far to trust these numbers. The judge checked the sample, and your answers check the judge: for example, people checked 16 of its verdicts and agreed with 15 (94%, with a 95% range of 72% to 99%). A session is **validated** when the range is at most 10 points on each side (or the most questions have answers) and starts at 80% or higher. The table reads the thresholds from 0.95 down. At each one it shows the estimated accuracy and where its 95% interval starts, the share of decisions at or above it, the known decisions there, and the next 0.1 above it. A threshold meets the rule when at least 10 known decisions lie at or above it, they reach the target, and their interval starts at most 10 points below the target. A threshold below the target stops the walk. So does a next 0.1 that holds decisions but no known one, or that is more than 10 points below the target. The proposal is the lowest threshold that meets the rule. The workflow gates from the inbox use the same rule. **Apply** writes a threshold: the proposal, a row of the table, or a value from 0.50 to 0.99. The page asks first and says what the threshold does. A session that is not validated can still be applied, with a warning. The router uses a new threshold within a minute. The integrations do not read the fit threshold yet. For a workflow, **Apply** sets its gate, and the workflow reads its instances again with it. The workflow keeps where its gate came from, and its row on **Workflows** says "from a session". The **Models** pages for question answering and the fit show the same threshold, with a **Calibrate** link. ## With the SDKs The TypeScript and Python clients wrap trials and switches under `models`. `wait` reads the trial until it compared the checked calls. ```ts const { trials: [trial] } = await lbb.models.trials.create({ candidate: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" }, jobs: ["ask"], }); const result = await lbb.models.trials.wait(trial.id); if (result.report.qualifies) await lbb.models.switches.create({ trial: result.id }); ``` ```python started = lbb.models.trials.create( candidate={"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"}, jobs=["ask"], ) trial = lbb.models.trials.wait(started.trials[0].id) if trial.report.qualifies: lbb.models.switches.create(trial=trial.id) ``` The clients choose the model of a workflow step with `models.choices`, and test a model first with `models` on a workflow test: ```ts await lbb.workflows.tests.create({ name: "products", sample: 6, models: { extract: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" } }, }); await lbb.models.choices.put({ use: "extract", model: { provider: "anthropic", model: "claude-haiku-5-5", effort: "low" }, workflow: "products", }); ``` ```python lbb.models.choices.put( use="extract", model={"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"}, workflow="products", ) lbb.models.choices.delete(use="extract", workflow="products") ``` The MCP server has the same: `lbb_model_choice` reads (`choices` lists the model choices), and `lbb_model_choice_manage` starts, stops, switches and reverts, and sets or removes a choice (`put_choice`, `delete_choice`). ## Over HTTP ```sh # The month's checks of a graph by job, and the judge's agreement curl "$LBB_URL/v1/models/checks/summary?graph=crm" \ -H "Authorization: Bearer $LBB_API_KEY" # Test Claude Haiku 5.5 on question answers and query routing curl -X POST "$LBB_URL/v1/models/trials?graph=crm" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \ --data '{"candidate": {"provider": "anthropic", "model": "claude-haiku-5-5", "effort": "low"}, "jobs": ["ask", "route"]}' # Correct a check: the right verdict and a note curl -X POST "$LBB_URL/v1/models/checks/review?graph=crm&id=$CALL_ID" \ -H "Authorization: Bearer $LBB_API_KEY" -H 'Content-Type: application/json' \ --data '{"agree": false, "verdict": "wrong", "note": "Contacts are companies."}' ``` | Route | What it does | | --- | --- | | `GET /v1/models/checks/summary` | the month per job and model, the judge's agreement, today's budget | | `GET /v1/models/checks` | the month's checks, newest first; `job`, `verdict`, `reviewed` filter | | `POST /v1/models/checks/review` | `{agree: true}`, or a correction with `verdict`, `reference` and `note` | | `GET /v1/models/checks/export` | the month's checks as JSON lines: the call, the judge and the review | | `GET /v1/models/calls` | the call log, newest first | | `GET /v1/models/calls/get` | one call with its input, output and check | | `POST /v1/models/calls/check` | check one call now | | `POST /v1/search/tuning` | start a tuning session | | `GET /v1/search/tuning` | the sessions, newest first | | `POST /v1/search/tuning/apply` | apply a session's proposal | | `GET /v1/workflows/triggered/calibration` | the workflow gates measured from inbox decisions: bands, labels and a proposal | | `POST /v1/workflows/triggered/calibration/apply` | apply a gate's proposal (a console session only): `{starter, name, gate?}` | | `GET /v1/models/trials/options` | per job: the current model, its checked calls, the models a trial can test | | `POST /v1/models/trials` | test a model: `{candidate: {provider, model, effort?}, jobs?}` | | `GET /v1/models/trials` | the trials, newest first, with their reports | | `GET /v1/models/trials/get` | one trial with its compared calls | | `POST /v1/models/trials/stop` | stop a trial | | `POST /v1/models/switches` | switch a task to a trial's model: `{trial}` | | `POST /v1/models/switches/revert` | back to LBB's model: `?job=ask` | | `GET /v1/models/choices` | the model of each workflow use: the environment's rows, and with `workflow` the rows in force for one workflow | | `PUT /v1/models/choices` | choose a model: `{use, model: {provider, model, effort?}, workflow?}` | | `DELETE /v1/models/choices` | remove a choice: `?use=extract&workflow=products`; answers `{id, object, deleted: true}` | | `POST /v1/models/calibration` | start a calibration session: `{"job": "route"}`, or a workflow's gate: `{"job": "resolve", "workflow": "documents-link"}` (`infer` for Values from text, `map` for Review facts) | | `GET /v1/models/calibration` | the calibration sessions, newest first | | `GET /v1/models/calibration/get` | one session with its questions and report | | `POST /v1/models/calibration/answer` | answer a question: `yes`, `no` (with an optional `right`) or `cant_tell` | | `POST /v1/models/calibration/apply` | apply the proposal, or a `threshold` from 0.50 to 0.99; for a workflow, its new gate | | `GET /v1/models/thresholds` | the applied thresholds of `route` and `fit` | A rerank correction takes `reference: {"grades": {"": 0..3}}`, and a choice correction `reference: {"picks": {"": "