Most open-weight model reviews end with a paragraph of numbers and a screenshot of a leaderboard column. The numbers are real. The order is rarely what a buyer needs.
Open-source LLM leaderboards are useful when they save you from a bad bet and dangerous when they take the place of one. They are also separate projects with different methodologies, and they index each other in ways that confuse. This page is the recipe for reading them.
TL;DR
- Five sources do most of the work for an open-weight buyer: LMSYS Chatbot Arena (pairwise human preferences), Hugging Face’s Open LLM Leaderboard (lm-evaluation-harness tasks; retired in March 2025, so a historical snapshot), Artificial Analysis Intelligence Index (weighted composite), HELM (Stanford CRFM, multi-metric; in maintenance mode since June 2026), and the model card itself (vendor- or lab-published, not a leaderboard).
- Each measures a different thing. Pairwise human votes measure style, helpfulness, and fluency under a prompt distribution the leaderboard picked. Static benchmarks measure accuracy on a frozen test set. Composites weight them by one author’s choice. None of them measure your workload.
- Four traps bite every cycle: the index rebase (a composite that drops or adds evaluations between versions changes what a “50” means), benchmark contamination (a public test question can sit in pre-training data), format-only wins (judges, human or LLM, can reward length and formatting over substance), and licensing drift (a model that scored 80% may not be one you can run or ship).
- The recipe: read the methodology page, treat the leaderboard as a shortlist, run two or three candidates on your own prompts, and only then promote the winner.
What each leaderboard actually measures
LMSYS Chatbot Arena (lmarena.ai / arena.ai)
Crowdsourced side-by-side. A user types a prompt, two anonymous models reply, the user picks the better one, the result feeds a global rating. The methodology page is explicit: “Your votes directly shape the model rankings through the Bradley-Terry rating system, a statistical model originally developed for paired comparison experiments. It is similar to the Elo rating system developed for ranking players in competitive games like chess.” The FastChat project that powers it has collected “over 1.5M human votes from side-by-side LLM battles” with “70+ LLMs” and “over 10 million chat requests” served, all under the project’s Apache-2.0 licence.
What it captures: style, helpfulness, fluency, response length, markdown formatting, refusal behavior. What it ignores: factual accuracy, latency, cost, and any task that does not fit the prompt template the Arena uses. The Arena’s own FAQ notes: “Only votes made while the models are anonymous count toward official rankings; any votes cast after model identities are revealed will not impact leaderboard standings.” A separate paper on the same system (arXiv:2306.05685) shows the LLM-as-judge variant can match crowdsourced human preferences “achieving over 80% agreement,” but is exposed to “position, verbosity, and self-enhancement biases.” Useful for ordering on style; biased for models that format like chat output.
Hugging Face Open LLM Leaderboard
Static benchmark tasks run through EleutherAI’s lm-evaluation-harness, which describes itself as “a framework for few-shot evaluation of language models” and lists “over 60 standard academic benchmarks” with “hundreds of subtasks and variants.” The harness README calls itself the backend for Hugging Face’s Open LLM Leaderboard. That leaderboard archived its v1 task set in June 2024 and replaced it with a harder v2, then announced its retirement in March 2025 after evaluating “over 13K models”, saying “The leaderboard is slowly becoming obsolete”. Its scores remain a useful historical snapshot, not a current ranking. The harness’s interface guide documents how to run the same tasks yourself.
The catch: any benchmark question that appears in a model’s pre-training corpus stops measuring generalisation and starts measuring memorisation. Benchmarks built from newly published problems try to mitigate this; static knowledge benchmarks like MMLU and GPQA are fixed sets. When you run the harness yourself, its --log_samples flag will “Save all model inputs/outputs for post-hoc analysis”, so you can inspect exactly which prompts were scored and how.
Artificial Analysis Intelligence Index (v4.3.2)
A weighted composite. The methodology page is direct: “Intelligence Index is calculated as a weighted average across four categories: Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%).” Anchors pin DeepSeek V4.1 Flash (max) at 1600 (GDPval-AA Elo) and GPT-5.5 (medium) at 1000 (AA-Briefcase). The same page is explicit that AA-LCR v1.1 “Scores are not directly comparable with v1.0” and that “the task set is shared, but the document input and judge differ, so Artificial Analysis and Surge scores are not directly comparable.” Useful for cross-vendor ordering; less useful if your workload sits outside the weighted categories. The methodology page also notes that “we benchmark models for image inputs, speech inputs and multilingual performance separately” and that the Index is “primarily text-based, English-language.”
HELM (Stanford CRFM)
Holistic. The HELM announcement page lists “accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency” as the seven metric categories, with “16 core scenarios” and “26 finer-grained scenarios” (42 in total) over “30 prominent language models.” The same page notes that “HELM by design foregrounds its incompleteness” and explicitly lists missing scenarios including “languages beyond English, applications beyond traditional NLP tasks such as copywriting, and metrics that capture human-LM interaction.” The HELM GitHub repository confirms the project “entered maintenance mode on June 1, 2026,” which is also a signal that newer models may not be added on the old timeline.
The model card
The card on Hugging Face or the vendor’s own page is the fifth source, and it is not a leaderboard. It is what the lab chose to publish, with the methodology and seeds the lab chose, on the prompts the lab curated. Read it last, not first.
The four traps that quietly shift every number
- The index rebase. The Artificial Analysis version history notes evaluations such as MMLU-Pro and LiveCodeBench were “Removed from the Intelligence Index in v4.0”, and its AA-LCR v1.1 note says “Scores are not directly comparable with v1.0.” Hugging Face archived its v1 task set and rebuilt with harder benchmarks. A 60 on one index and a 60 on another may be the same integer and entirely different capabilities.
- Benchmark contamination. Public test questions can leak into pre-training corpora. The most charitable assumption is “uncertain”; the realistic one is “happens often enough to be a known unknown.”
- Format-only wins. The MT-Bench and Chatbot Arena paper (arXiv:2306.05685) documents “position, verbosity, and self-enhancement biases” in LLM judges; a longer or better-formatted answer can win a vote without being more correct. Your rubric should score substance.
- Licence drift. A model with a high score may be gated on Hugging Face, may require a commercial licence, or may have a MAU cap that disqualifies your use case. The leaderboard does not check this; you do.
A recipe for reading any new leaderboard
- Read the methodology page. Skip the leaderboard table. The methodology page is the contract.
- Check the version number and the date. A v3 result from 2025 may share a scale with a v4 result from 2026 and mean different things.
- Note the anchor and the weighting. A composite whose author gave 30% weight to a category you don’t care about is a noisy signal for you.
- Cross-reference at least one primary benchmark (a published model card or paper) for any model you intend to actually use.
- Run your own prompts through two or three candidates. Treat the leaderboard as a shortlist, not an answer.
The bottom line
Leaderboards exist to compare models at scale across many users. They do not exist to pick the right model for one user. Read the methodology page, treat the leaderboard table as a shortlist, and run your own evaluation before promoting any candidate to production. The work pays back the first time an index you trusted gets rebased and your real test does not.