Human preference optimization, explained

Research5 Sep 20267 mins read

Introduction

Supervised examples teach a model what a good answer looks like. Preference data teaches it which of two good-looking answers your experts would actually choose.

RLHF and DPO

RLHF trains a reward model on preferences and optimises the policy against it. DPO skips the reward model and learns directly from preferred and rejected pairs, which is simpler and often more stable.

Data quality decides the outcome

Who writes the pairs matters

Pairs written by domain experts capture the trade-offs that matter: caution in clinical answers, precision in financial ones. Generic raters tend to reward fluency over correctness.

The silent failure in long RL runs: trainer–inference mismatchA tiny numerical gap between rollout and training log-probs can bend a week of learning. How we detect it early.In on-policy reinforcement learning the trainer assumes the samples came from the policy it's optimising. In practice, rollouts are generated by a separately optimised inference engine, and the two rarely agree to the last decimal.Continue reading

Research21 Jul 2026

How much domain data is enough?Early results on scaling curves for domain adaptation, and why the first thousand examples matter most.9 mins read

Research16 Jun 2026

Understanding inference cost per tokenBatching, context length and model size: the three levers that decide your serving bill.6 mins read

Guides1 Oct 2026

Fine-tuning or prompting? A practical decision guidePrompting gets you started; fine-tuning gets you consistent. A checklist for choosing the right tool for each task.5 mins read

Products26 Sep 2026

Turn one-off answers into scheduled reports with CoworkerAsk once, then let Coworker deliver the answer every Monday, shared with the people who need it.3 mins read

Model training12 Sep 2026

What's new in the model catalog: 174 open-weight modelsNew MoE releases, longer contexts and per-model correctness tests: the autumn update to the catalog.4 mins read

Build frontier models and agents that learn from experience.

Model training6 Oct 2026

Introduction

General-purpose models are remarkable, but they don't know your contracts, your claims rules or the shorthand your engineers use. The gap between a demo and a dependable system is almost always domain knowledge. Learning from experience closes that gap. Instead of prompting harder, you let a model practise on your tasks, score the results against what good looks like, and keep the improvements.

6 mins readContinue Reading

// Ready to see the DataObserve in action?

Handle complex workflows at scale, from live delivery issues to compliance decisions, while maintaining over 90% resolution accuracy in production.

Book a Demo

Human preference optimization, explained

Introduction

RLHF and DPO

Data quality decides the outcome

Related articles

Build frontier models and agents that learn from experience.

Introduction

Handle complex workflows at scale, from live delivery issues to compliance decisions, while maintaining over 90% resolution accuracy in production.

We safeguard your company data and protect workspace privacy.

Security and privacy

Governance and access

Enterprise deployment

We are HiringJoin the // Leap