Most model-selection articles on this site (and elsewhere) lean on leaderboard scores: MMLU, GPQA, HumanEval, LiveCodeBench, an Artificial Analysis composite. Those numbers are useful as a coarse filter, and they are almost useless as a buyer’s recommendation for your specific workload. The site has lost a paragraph before because a 50 on one index meant something different after the index was rebased; it has lost another because a benchmark question turned out to have been in the model’s training set. This page is the recipe we use to avoid both mistakes.
Why general benchmarks drift on you
Every composite index gets revised. Artificial Analysis publishes a methodology page that lists the weights and anchors it is using at that moment; the current v4.3 weight is Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%, with a GDPval-AA v2 Elo anchored at 1000 (human expert deliverables) and an AA-Briefcase Elo anchored at 1000 (GPT-5.5 medium). The page explicitly notes that “Scores are not directly comparable with v1.0” for AA-LCR, and that Elo scores for individual models are “frozen at the time of a model’s addition.” A figure you bookmarked six months ago and a figure you read today may share a scale of zero-to-100 but no longer measure the same thing.
The Open LLM Leaderboard on Hugging Face went through the same kind of rebase when v1 was retired and replaced with v2’s harder benchmark set. Benchmark creators also retire individual datasets: a 2022 HumanEval number cannot be compared to a 2025 LiveCodeBench v6 number even when both are reported as “coding” scores on the same model card. The result is a leaderboard ecosystem where two adjacent numbers on a page can come from different epochs and mean different things, even when the column header is identical.
What the popular indexes actually measure
| Index or benchmark | What it scores | Strength | Known weakness |
|---|---|---|---|
| MMLU / MMLU-Pro | Multiple-choice knowledge across 57+ subjects | Broad coverage, easy to drop into a harness | Heavy data contamination risk; multiple-choice format favours models trained on MC patterns |
| GPQA / SuperGPQA | Graduate-level science multiple choice | Hard to game with surface knowledge | Still multiple choice; small test set, high variance per run |
| HumanEval | 164 hand-written Python problems, pass@k | Real functional tests, not MC | MIT-licensed but warns “exists to run untrusted model-generated code”; the README’s own evaluate_functional_correctness should run in a sandbox |
| LiveCodeBench v6 | Continuously updated coding problems scraped from contests | Resists contamination by design | Cuts off at a date, so older models look weaker than they were |
| Artificial Analysis Intelligence Index v4.3 | Weighted composite: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20% | Forces trade-offs to be visible | The weights are one author’s choice; freezing at “time of a model’s addition” means late additions cannot be re-scored under the same rubric |
| Chatbot Arena (Bradley-Terry Elo) | Side-by-side human preference | Captures style, helpfulness, fluency | Verbosity bias, markdown/formatting bias, position bias, uneven prompt coverage by language and domain |
The Hugging Face Open LLM Leaderboard Space, EleutherAI’s lm-evaluation-harness, and Artificial Analysis are the three places most readers will land. None of them will answer “which model is best for me.”
The constraints your real workload imposes
Three constraints narrow the field faster than any benchmark:
- Weight size and quant. A model with excellent scores that does not fit your hardware with the headroom for your KV cache is not a candidate. The site’s VRAM hub and the quantization cost page cover how to size both.
- Tokeniser and chat template. A model trained on a chat template your runner does not match will produce lower-quality outputs than the leaderboard card suggests. The site’s runners page lists the runners and which templates each applies by default.
- Latency and context length. Time-to-first-token and tokens-per-second at your expected context length are not on any leaderboard card; the card reports a single context length and a single batch size. The used-GPU page covers what each consumer card can realistically sustain.
A practical evaluation recipe
The harness interface guide documents a --log_samples flag and an --output_path JSON, which is what you need for sample-level inspection. The recipe below uses those flags.
- Pull 30 to 50 real prompts. Take them from your actual tickets, transcripts, documents, or past usage. Mix short and long, easy and hard, English and the languages you actually serve. Strip identifying information first.
- Write the reference answer or rubric. For each prompt, write what a correct or acceptable answer looks like. A 1-to-5 rubric scored by you and one other person beats a single ground-truth answer on subjective tasks (summarisation, rewriting, code review).
- Run two to four candidate models on the same prompts. Use the same runner, the same quant, the same temperature, the same system prompt. Save with
--log_samplesso you can read what each model actually produced, not just the aggregate score. - Score blind. Randomise the order, hide the model name, score each output against the rubric. Aggregate per model and per prompt-category.
- Run the same set twice, days apart. If the same model wins on both runs at the same quant, you have a signal. If the winner changes, the test set is too small and you need more prompts.
- Promote the winner only after a third run with a different seed. Three consistent wins is the minimum bar before changing production.
This recipe costs you a day or two of focused work. It produces a number that is calibrated to your workload, immune to benchmark rebase, and reproducible. The harness will handle steps 3 and the output capture if you wire it up; steps 1, 2, 4, 5, 6 are the parts only you can do.
Pitfalls you cannot engineer around
- Training contamination. Public benchmark questions can land in a model’s training data. LiveCodeBench was designed to mitigate this by scraping new contests; MMLU and GPQA have not been refreshed and contamination is unmeasurable from the outside.
- Format-only wins. A model can win on IFEval or arena votes by following instructions about formatting (markdown headers, bullet counts, length) without improving the underlying answer. Your rubric should score substance, not structure.
- Quant noise at the boundaries. A model that scores 83.0 MMLU-Pro on its BF16 reference can score within a percentage point or two of the reference at Q4_K_M, but the gap is largest on the prompts that depend most on numerics and code. The quantization cost page carries the published ladder; trust it for the typical case, and re-check on your hardest prompts.
- Sandboxing. HumanEval’s own README warns that
evaluate_functional_correctness“exists to run untrusted model-generated code” and recommends a robust sandbox. The harness can generate and execute code against your test cases. Run it in a container that cannot reach the network and cannot write outside a tmpfs. - Licence and gating. A score on a gated model you cannot pull (gated repos on Hugging Face, custom-licence families) is not actionable; the site’s licence page covers which open-weight families you can run locally without paperwork.
When to trust a benchmark
A benchmark is worth reading when it (a) tests something close to what you do, (b) is recent enough that the model’s training corpus almost certainly included the cutoff or postdates it, and (c) is one of several signals you are combining. LiveCodeBench v6 plus a 30-prompt in-house test plus a runner-level throughput check is a defensible decision. A single MMLU-Pro number is not.
Bottom line
Benchmark indexes exist to compare models at scale across many users. They do not exist to pick the right model for one user. Build the in-house test, run it on the candidates, score it blind, and treat the leaderboards as a shortlist generator, not an answer. The work pays back the first time an index you trusted gets rebased and your real test does not.