Compare Language Models on the Work You Actually Do

A leaderboard can narrow a shortlist. It cannot tell you whether a model will preserve a customer ID, cite the right paragraph or decline a refund outside policy. To compare two candidates, build a test around a decision your application actually needs to make.

Choose cases before seeing the answers

For a hypothetical invoice extractor, include clean invoices, scans with missing fields, two currencies on one page and documents that are not invoices at all. Define the expected output for absent information. If guessing an amount is worse than returning null, the scoring rule must reflect that cost.

Keep development examples separate from the final comparison. Repeatedly rewriting the prompt to fix the same ten cases measures familiarity with those cases. Reserve a set that is not consulted while tuning.

Score the same work under comparable conditions

Use the same input, tool access and output contract for both candidates. Record prompt, model identifier and generation settings. Repeat cases where run-to-run variation matters. Randomize answer order for human reviewers and hide the model names. Use exact checks for machine-readable fields and a written rubric for judgments such as faithful summarization.

Suppose one model passes 46 of 50 examples and the other passes 44. Inspect which four and six failed. A single invented invoice total can matter more than several harmless formatting differences. A small sample is directional evidence, not a universal ranking.

Include operational fit

Measure latency and cost on realistic input lengths, including retries and validation. Keep the losing model's strongest cases in the regression set. The final choice should explain which errors are acceptable, which trigger review and what result would justify switching later.

Further reading

Stanford CRFM: HELM

Related guide: How to Read an Ablation Study.