Guides by Editorial Team

How to Read Open-Source LLM Leaderboards Without Lying

The five major open-source LLM leaderboards and what they actually measure, the methodology traps that shift the numbers, and a recipe for picking a model.

Most open-weight model reviews end with a paragraph of numbers and a screenshot of a leaderboard column. The numbers are real. The order is rarely what a buyer needs.

Open-source LLM leaderboards are useful when they save you from a bad bet and dangerous when they take the place of one. They are also five different projects with five different methodologies, and they index each other in ways that confuse. This page is the recipe for reading them.

TL;DR

  • Five leaderboards do most of the work for an open-weight buyer: LMSYS Chatbot Arena (pairwise human preferences), Hugging Face’s Open LLM Leaderboard (lm-evaluation-harness tasks), Artificial Analysis Intelligence Index (weighted composite), HELM (Stanford CRFM, multi-metric), and the model card itself (vendor- or lab-published).
  • Each measures a different thing. Pairwise human votes measure style, helpfulness, and fluency under a prompt distribution the leaderboard picked. Static benchmarks measure accuracy on a frozen test set. Composites weight them by one author’s choice. None of them measure your workload.
  • Four traps bite every cycle: the index rebase (v3.0’s “50” is not v4.3’s “50”), benchmark contamination (a public test question can sit in pre-training data), format-only wins (markdown headers and bullet counts inflate IFEval without helping substance), and licensing drift (a model that scored 80% may not be one you can run or ship).
  • The recipe: read the methodology page, treat the leaderboard as a shortlist, run two or three candidates on your own prompts, and only then promote the winner.

What each leaderboard actually measures

LMSYS Chatbot Arena (lmarena.ai / arena.ai)

Crowdsourced side-by-side. A user types a prompt, two anonymous models reply, the user picks the better one, the result feeds a global rating. The methodology page is explicit: “Your votes directly shape the model rankings through the Bradley-Terry rating system, a statistical model originally developed for paired comparison experiments. It is similar to the Elo rating system developed for ranking players in competitive games like chess.” The FastChat project that powers it has collected “over 1.5M human votes from side-by-side LLM battles” with “70+ LLMs” and “over 10 million chat requests” served, all under the project’s Apache-2.0 licence.

What it captures: style, helpfulness, fluency, response length, markdown formatting, refusal behavior. What it ignores: factual accuracy, latency, cost, and any task that does not fit the prompt template the Arena uses. The Arena’s own FAQ notes: “Only votes made while the models are anonymous count toward official rankings; any votes cast after model identities are revealed will not impact leaderboard standings.” A separate paper on the same system (arXiv:2306.05685) shows the LLM-as-judge variant can match crowdsourced human preferences “achieving over 80% agreement,” but is exposed to “position, verbosity, and self-enhancement biases.” Useful for ordering on style; biased for models that format like chat output.

Hugging Face Open LLM Leaderboard

Static benchmark tasks run through EleutherAI’s lm-evaluation-harness, which describes itself as “a framework for few-shot evaluation of language models” and lists “over 60 standard academic benchmarks” with “hundreds of subtasks and variants.” The Hugging Face Open LLM Leaderboard is the canonical “open-weight capability floor” reference: the v1 task set was retired and rebuilt into v2’s harder variant set, and the harness’s interface guide is what publishes the per-task scores you read on the Space.

The catch: any benchmark question that appears in a model’s pre-training corpus stops measuring generalisation and starts measuring memorisation. LiveCodeBench was designed to mitigate this by scraping new contests; static knowledge benchmarks like MMLU and GPQA have not been refreshed on the same cadence, and the harness’s --log_samples flag is how you can inspect which prompts a model actually saw.

Artificial Analysis Intelligence Index (v4.3.2)

A weighted composite. The methodology page is direct: “Intelligence Index is calculated as a weighted average across four categories: Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%).” Anchors pin DeepSeek V4.1 Flash at 1600 (GDPval-AA Elo) and GPT-5.5 (medium) at 1000 (AA-Briefcase). The same page is explicit that “scores are not directly comparable with v1.0” for AA-LCR and that “the task set is shared, but the document input and judge differ, so Artificial Analysis and Surge scores are not directly comparable.” Useful for cross-vendor ordering; less useful if your workload sits outside the weighted categories. The methodology page also notes that “we benchmark models for image inputs, speech inputs and multilingual performance separately” and that the Index is “primarily text-based, English-language.”

HELM (Stanford CRFM)

Holistic. The HELM announcement page lists “accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency” as the seven metric categories, with “16 core scenarios” and “26 finer-grained scenarios” (42 in total) over “30 prominent language models.” The same page notes that “HELM by design foregrounds its incompleteness” and explicitly lists missing scenarios including “languages beyond English, applications beyond traditional NLP tasks such as copywriting, and metrics that capture human-LM interaction.” The HELM GitHub repository confirms the project “entered maintenance mode on June 1, 2026,” which is also a signal that newer entrants may not be added on the same timeline as older labs.

The model card

The card on Hugging Face or the vendor’s own page is a sixth source, not a leaderboard. It is what the lab chose to publish, with the methodology and seeds the lab chose, on the prompts the lab curated. Read it last, not first.

The four traps that quietly shift every number

  1. The index rebase. Artificial Analysis explicitly notes that “scores are not directly comparable with v1.0.” Hugging Face retired the v1 task set and rebuilt with harder benchmarks. A 60 on one index and a 60 on another may be the same integer and entirely different capabilities.
  2. Benchmark contamination. Public test questions can leak into pre-training corpora. The most charitable assumption is “uncertain”; the realistic one is “happens often enough to be a known unknown.”
  3. Format-only wins. A model that scores higher on IFEval because it uses better markdown headers has not improved on substance. Your rubric should score substance.
  4. Licence drift. A model with a high score may be gated on Hugging Face, may require a commercial licence, or may have a MAU cap that disqualifies your use case. The leaderboard does not check this; you do.

A recipe for reading any new leaderboard

  1. Read the methodology page. Skip the leaderboard table. The methodology page is the contract.
  2. Check the version number and the date. A v3 result from 2025 may share a scale with a v4 result from 2026 and mean different things.
  3. Note the anchor and the weighting. A composite whose author gave 30% weight to a category you don’t care about is a noisy signal for you.
  4. Cross-reference at least one primary benchmark (a published model card or paper) for any model you intend to actually use.
  5. Run your own prompts through two or three candidates. Treat the leaderboard as a shortlist, not an answer.

The bottom line

Leaderboards exist to compare models at scale across many users. They do not exist to pick the right model for one user. Read the methodology page, treat the leaderboard table as a shortlist, and run your own evaluation before promoting any candidate to production. The work pays back the first time an index you trusted gets rebased and your real test does not.