# Calibre

> Calibre supplies expert-written and expert-reviewed LLM training data (SFT examples, preference ranking, red-teaming and evaluation sets) for AI labs and enterprise AI teams. Two experts label every item, a third decides when they disagree, and every row ships with its QA trail. Beyond the data, Calibre trains the model to an agreed target, runs it, and has experts review what it says in production, so every miss becomes the next batch.

Last updated: 2026-10-25. Features marked "in pilot" are not generally available yet. Numbers on the site are sample data unless stated otherwise.

## The problem Calibre solves

- Most data vendors sell one lap: the data ships and the vendor leaves. The people who labelled it never see how the model did, so the same gaps get labelled the same way next time.
- A model's misses in evaluations and production stay on a dashboard instead of becoming the next training batch.
- Guidelines drift as the labelling team grows, so batch 1 and batch 40 are graded against slightly different readings.

## The loop: annotate → train → deploy → distil → evaluate → repeat

### Annotate
- Two experts label each item blind. Calibre scores how closely they agree; anything below the customer's bar goes to a third expert (an adjudicator), and items are spot-checked against a hidden gold set.
- Specialists are domain experts (medicine, law, finance, code, maths), matched to the task by domain. Each passes a paid trial task scored against a gold set before touching client data.
- Every row is traceable: who wrote it, who checked it, where they disagreed, who adjudicated, and which guideline version applied. Delivered as JSONL with the QA fields on each record.

### Train (in pilot)
- The customer sets the target score. Calibre adds a training stage only when the last one falls short, so the customer doesn't pay for stages they don't need.
- Every run starts as a plan the customer approves: data snapshot, guideline version, target and compute budget. If the plan changes after approval, the run won't start until it is approved again.
- Every training decision and every failure is written to a run log the customer can read.

### Deploy
- Experts sample live model outputs at a rate the customer chooses, mark each right or wrong, and write the fix into the next lap's training data.
- Confidence-based sampling (in pilot) sends experts the outputs the model was least sure of first, instead of random samples that mostly find answers that were fine.
- Every fix cites the policy clause it enforces, so it can be checked, appealed and reused. Appeals must cite a clause too.

### Distil (in pilot)
- Calibre distils the best model into smaller students and scores each against the same gold set. The customer sets a quality floor; the smallest model that clears it ships.
- On-policy distillation: the smaller model answers for itself and the larger one corrects it as it goes, with experts spot-checking the teacher.
- Smaller models are cheaper and faster to serve. Sample comparison on one item: 3.7× faster and 23× cheaper with the same gold-set verdict.

### Evaluate
- Each lap is scored on a held-out gold set the model never trains on, broken down by guideline clause, so a clause that slips is caught before the model ships.
- Every sampled mistake is traced to a cause: an unclear guideline (clause rewritten), an annotator miss (re-test on the clause) or a wrong gold answer (gold corrected, scores recalculated).
- Calibre measures the ceiling that contradicting labels put on any model's score, every lap, so the next lap's budget goes to more training or to cleaner data, whichever actually helps.

## Services

- **LLM training data** (core): SFT examples, preference ranking, red-teaming and evaluation sets.
- **Managed annotation**: image, text, audio and video, delivered as finished datasets with the same two-expert QA.
- **Human-in-the-loop QA**: experts review live model outputs; every correction feeds the next training batch.
- **Platform and workforce**: run projects yourself in the Calibre workspace, with vetted experts on call.
- **AI pre-labelling** (add-on): agents draft, experts correct every item, and each run reports how often they matched.
- **Model training and hosting** (in pilot): training on the finished dataset to the approved target, then running the model with expert review on what it says.

## Data security

- NDA before the first file; experts sign too.
- An isolated workspace per client, with no cross-project access.
- Access approved per person.
- Client prompts and labels are never reused to train anything else.
- Security review status, uptime and response-time commitments: to be published.

## How to start

1. Guidelines written with the customer's researchers.
2. A 100-item calibration batch, scored together.
3. Pilot volume, with agreement reported daily.
4. Steady-state weekly volume.

Pricing: per item, by task type and expert domain, quoted after the calibration batch so the price reflects real effort.

To start, send 50 items. Calibre replies within one business day with draft guidelines and a pilot timeline.

## Frequently asked questions

- **Is this RLHF?** RLHF is one way to train on human judgement: experts rank answers and the model learns to prefer the better ones. Calibre supplies that judgement, plus example answers and test sets. What it sells is the loop around it: each lap, the model's misses come back to experts and become the next batch.
- **Who is responsible for the model's performance?** Calibre is, against targets agreed before the first lap: a gold-set score per guideline clause, response time and uptime.
- **Do you use AI to label?** Only if the customer wants it. Agents pre-label, experts correct every item, and each run reports the match rate.
- **What happens when guidelines change?** The change is drafted, reviewed and signed off; gold answers move to the new version; every annotator passes a check on the new clause before labelling again. Each item records the version it was labelled under.

## Links

- [Calibre home](https://calibre.neoai.solutions/): the full loop, services, security commitments and pilot request form