LLM training data Where we lead
SFT examples, preference ranking, red-teaming and evaluation sets, written and reviewed by domain experts.
Your model’s next lap starts with better data.
We build the data, train the model, run it, and have experts review what it says. Every miss becomes the next batch, so each lap is better and cheaper to run.
Sample laps. Tap a station to jump to it.
Lap1
- Eval score
- –
- Response time
- 1.42 s
- Serving cost
- $4.10
Most vendors sell you one lap.
- The data ships, and the vendor leaves.
- The people who labelled it never see how the model did, so the same gaps get labelled the same way next time.
- Your model’s misses never come back.
- Failures in evals and production stay on a dashboard instead of becoming the next batch.
- The guidelines drift as the team grows.
- Batch 1 and batch 40 are graded by different people against a slightly different reading of the guidelines.
Two experts per item. When they disagree, a third decides.
- Expert A labels
- Expert B labels, blind
- How closely they agree
- Below your bar: adjudicator
- Gold check
0accepted first pass
0sent to a third expert
0.50How closely the two experts agreed1.00
You choose the bar. Items where the two experts closely agree are accepted. Drag it: everything below gets a third expert before it ships. 200 sample items
Specialists, vetted on paid work before they touch yours.
A doctor labels your medical data, not an anonymous gig worker. You don’t pick individuals; we match specialists to the task by domain.
- Napplied
- Npassed a paid trial task
- Nmatched a hidden gold set
- Nwork on client data today
A task they would take
The kind of specialist on your task. Sample profile.
Every row ships with its receipt.
Trace any row in your training set back to who wrote it, who checked it, where they disagreed and which version of the guidelines they used.
- Written by E-0412, MD, internal medicineRanked B over A, wrote a 2-sentence rationale. 6 min.
- Labelled blind by E-0198, PharmDAlso ranked B over A. 4 min.
- Agreement 0.71, below the 0.80 barThe two rationales cite different risks. Sent to adjudication.
- Adjudicated by E-0033, clinical leadKept B; merged both risks into the rationale.
- Gold check passed · guidelines v3.2Delivered in batch 14.
{"id": "PREF-08812", "preferred": "B",
"annotators": ["E-0412", "E-0198"], "agreement": 0.71,
"adjudicator": "E-0033", "gold_check": "pass",
"guidelines": "v3.2", "minutes": 16, "batch": 14}
Sample record
We train to your target, then stop.
You set the score the model has to reach. We add a training stage only when the last one falls short, so you never pay for stages you don’t need. The dotted line is the best score your labels allow.
- Stages run
- Final accuracy
- GPU hours
You approve the training plan. We run exactly that.
Every run starts as a plan you sign off: which data, which guidelines, the target and the compute budget. If anything changes after you approve it, the run won’t start until you approve it again.
No surprises on your bill or in your model. Try changing the plan after it’s approved. Sample plan. In pilot.
Training plan · support assistant, lap 3Plan
- Base model
- 8B instruct
- Data
- Batch 14 snapshot · 12,400 items
- Guidelines
- v3.2, gold answers v3.2
- Compute budget
- 4 × A100, 8 hours
- Target
- 0.90 on the held-out split
Approved a41f·09c2Current a41f·09c2
You see every decision we make while training.
Every choice is written down as it happens: the model size, the memory plan, each stage. When something fails, we handle it and log what we did, so you can read exactly what happened.
Two failures overnight, both handled and logged. Nothing hidden, and nothing for your team to fix. Sample run. In pilot.
- Measured label conflicts. Ceiling: 0.94.
- Picked an 8B base. Fits 4 × A100 at micro-batch 16.
- Out of memory at step 1,840. Micro-batch 16 → 8.
- Resumed from the checkpoint at step 1,800.
- Machine restarted. Resumed at step 3,200.
- SFT reached 0.86, below the 0.90 target. Adding distillation.
- Target met: 0.90 on the held-out split.
Action needed from your teamNone
Experts read what your model says in production.
We sample live outputs at a rate you choose. Experts mark each one right or wrong and write the fix, which goes straight into the next lap’s training data.
Sampled outputs · support assistantReviewing
Fixes added to next lap’s data0
Spend the review budget where the mistakes are.
Random sampling mostly finds answers that were fine. Confidence-based sampling sends experts the outputs your model was least sure of first.
Reviewed, wrongReviewed, fineNot reviewed
Every fix cites the rule it enforces.
Experts don’t just rewrite a bad answer. They point to the line of your policy it broke, so the model learns the rule, not someone’s opinion.
A fix with a citation can be checked, appealed and reused. Appeals have to cite a clause too.
Your model said
Your policy
Sample policy and outputs
Same answers, a fraction of the cost to serve.
We distil your best model into smaller students and score each against the same gold set. You set the quality floor; the smallest model that clears it ships.
The student practises. The teacher marks every word.
The smaller model answers for itself and the larger one corrects it as it goes. Experts spot-check the teacher, so the student doesn’t learn the teacher’s mistakes.
By round four the student says what the teacher would. Sample rounds. On-policy distillation is in pilot.
Is ibuprofen fine with warfarin?
Matches the teacher0%
Teacher: No. Ibuprofen raises bleeding risk with warfarin. Paracetamol is usually safer; check with your prescriber. Expert spot-check passed.
Smaller also means sooner.
A distilled model doesn’t just cost less to serve; it answers faster. Same question, same answer, side by side.
3.7× faster and 23× cheaper, with the same gold-set verdict on this item. Sample timings
Question: Is ibuprofen fine with warfarin?
Large model 70B
Full answer0.00 s
Small model 3B
Full answer0.00 s
Both correct on the gold set
A score for every clause, every lap.
Each lap is scored on a held-out gold set your model never trains on, broken down by guideline clause. A clause that slips is caught before the model ships.
The overall score can rise while one rule gets worse. This is where you see it. Sample scores
Every miss gets a cause. The cause becomes the next lap’s brief.
We sample mistakes from every batch and trace each to its cause. Disputes go to a blind arbitrator, and appeals must cite the clause they rely on.
Each cause has its own fix, so the same mistake doesn’t come back next batch. 30 sample mistakes from one audit
Is it the model or the data?
When labels contradict each other, no model can score above a certain line. We measure that ceiling every lap, so the next lap’s money goes to more training or to cleaner data, whichever actually helps.
Overall score vs. what the labels allowSample
0.701.00
- Score
- Ceiling
- 0.94
- Next lap
Calibrate on lap one. Scale without drift.
- Day 0Guidelines written with your researchers
- Day N100-item calibration batch, scored together
- Week NPilot volume, agreement reported daily
- Steady stateUp to N items a week
Sample ramp. The line that matters is the flat one.
Join the loop wherever you are.
Managed annotation
Image, text, audio and video, delivered as finished datasets with the same two-expert QA.
Your model, in productionUnopened items can be returned within 60 30 days of delivery.
Correction added to your training set
Human-in-the-loop QA
Experts review live model outputs; every correction feeds the next training batch.
- MDPREF-0912 · medicine
- JDEVAL-0388 · law
- CFASFT-1407 · finance
- SEPREF-0915 · code
Platform and workforce
Run projects yourself in our workspace, with vetted experts on call.
invoicereceiptcontractinvoiceinvoicememo
receiptcontractreceiptmemoinvoicereceipt
Agents matched the experts on 8 of 10
AI pre-labelling Add-on
Agents draft, experts correct, and every run reports how often they matched.
Model training and hosting In pilot
We train on your finished dataset to the target you approve, then run the model for you, with expert review on what it says.
Your data stays yours. Our commitments stay visible.
- NDA before the first file. Experts sign too.
- An isolated workspace per client. No cross-project access.
- Access approved per person. Every role is requested and signed off.
- Never reused. Your prompts and labels never train anything else.
- Security review: status
- Expert agreement, first pass
- 0.00
- Gold-set accuracy
- 00%
- Median turnaround, 1,000 items
- N days
- Hosted model uptime
- 00.0%
- Response time, 95th percentile
- N ms
Questions research teams ask first.
Who writes the data?
Vetted domain experts: clinicians, lawyers, analysts, engineers and mathematicians, matched to each task. Every one passes a paid trial task scored against a gold set before working on client data.
How do you measure quality?
Two experts label each item independently. We score their agreement, send anything below your bar to a third expert, and spot-check against a hidden gold set. You get those numbers per batch, not a summary at the end.
How fast can you start?
We write the guidelines with your team, run a 100-item calibration batch, review it together, then scale. Most of the time goes into the guidelines, which are the part that decides quality.
Is this RLHF?
RLHF is one way to train on human judgement: experts rank answers and the model learns to prefer the better ones. We supply that judgement, and also example answers and test sets. What we sell is the loop around it: each lap, your model’s misses come back to experts and become the next batch.
Who is responsible for the model’s performance?
We are, against targets agreed with you before the first lap: a gold-set score per guideline clause, response time and uptime. You approve every training plan, and every lap is reported against those targets.
Do you use AI to label?
Only if you want it. Our agents can pre-label, and experts correct every item. Each run reports how often the agents matched the experts, so you can see whether it saves you anything.
What happens when our guidelines change?
The change is drafted, reviewed and signed off, gold answers move to the new version, and every annotator passes a check on the new clause before labelling again. Each item records which version it was labelled under.
What does it cost?
Priced per item by task type and expert domain, quoted after the calibration batch so the price reflects real effort, not a guess.
Start lap one. Send us 50 items.
Tell us the task. We reply within one business day with draft guidelines and a timeline for the pilot batch.