LLM chess benchmark: the public leaderboards compared

An LLM chess benchmark makes language models play chess, against each other, against engines or against a random mover, and turns the results into a rating. As of October 26, 2026, we could verify four public ones with their methods and numbers. Gemini models lead three of them, the best models from OpenAI and Anthropic sit in the 1500s to 1600s on Dubesor's and Saplin's boards, and no model comes close to a real chess engine.

The ratings can't be swapped between boards. Each one has its own opponents, its own rules for illegal moves and its own zero point. So this page puts the method next to every number.

The four LLM chess leaderboards at a glance

Checked on October 26, 2026. "Last data" is the date the source itself shows.

Leaderboard	Who runs it	How ratings are made	Opponents	Illegal moves	Last data
AI Chess Leaderboard	Dubesor, independent developer	Elo updated after each game; starting value from Stockfish 17.1 accuracy	Other AI models; maia5-1500 and a random mover as references	Tracked as a "Legal %" column	Oct 26, 2026 (357 models, 5,163 matches)
LLM Chess Leaderboard	Maxim Saplin; paper with co-authors (arXiv 2512.01992)	Maximum-likelihood Elo with a 95% interval	A random player and the Komodo Dragon engine at levels rated on Chess.com; the model always plays Black	Loss after 3 mistakes in one move, or 10 turns of dialogue	Data file of Oct 20, 2026 (GitHub)
LLM Chess Benchmark	Open-source project (GitHub)	Glicko-2, calibrated to an estimated Lichess Classical rating	Fixed-rating anchors (Maia networks, the Eubos engine, a random mover set to 400) and other models	One retry, then the game is forfeited	No date on the page; checked Oct 26, 2026
Game Arena paper	Google DeepMind and Kaggle, 62 authors	Bradley–Terry Elo; the weakest model is set to 0	Frontier models from 5 labs, 40 games per pair	Loss after four failed attempts	Submitted Sept 25, 2026; models chosen as of Feb 2026

Kaggle also runs a live chess leaderboard for Game Arena. Its numbers load in the browser and we couldn't read them reliably, so we left them out.

Top ratings on each board

Every row names its source. "±" and ranges are each board's own uncertainty. We only copied numbers we could see on the source on October 26, 2026.

Board	Model	Rating	Games	Source
Dubesor	Gemini 3.1 Pro Preview	1860 ±85	81	dubesor.de, Oct 26
Dubesor	GPT-6 Astra	1637 ±86	45	dubesor.de, Oct 26
Dubesor	Claude Opus 5.5	1571 ±98	35	dubesor.de, Oct 26
Dubesor	maia5-1500 (human-like network, not an LLM)	1554 ±90	46	dubesor.de, Oct 26
Dubesor	DeepSeek V4 Flash (0731)	1438 ±99	44	dubesor.de, Oct 26
Dubesor	Grok 4.20 Reasoning	1362 ±89	47	dubesor.de, Oct 26
Dubesor	Random mover (baseline)	419	200	dubesor.de, Oct 26
Saplin	GPT-6 Astra (high)	1614 ±110	67	Saplin, data of Oct 20
Saplin	GPT-6.1 Sol (xhigh)	1612 ±162	30	Saplin, data of Oct 20
Saplin	Gemini 4 Argon (high)	1609 ±131	40	Saplin, data of Oct 20
Saplin	Claude Opus 5.5 (high)	1575 ±113	64	Saplin, data of Oct 20
LLM Chess Benchmark	Gemini 3.1 Pro Preview (high)	2144 (1963–2325)	18	chessbenchllm, Oct 26
LLM Chess Benchmark	GPT-6 Astra (high)	1975 (1870–2081)	50	chessbenchllm, Oct 26
LLM Chess Benchmark	maia-1900 (anchor, not an LLM)	1816	256	chessbenchllm, Oct 26
LLM Chess Benchmark	Claude Opus 5.5 (high)	1197 (517–1876)	2	chessbenchllm, Oct 26
Game Arena paper	Gemini 3 Pro Preview	1325	40 per pair	arXiv 2609.31473
Game Arena paper	o3	1009	40 per pair	arXiv 2609.31473
Game Arena paper	Grok 4	773	40 per pair	arXiv 2609.31473
Game Arena paper	Claude Opus 4.5	236	40 per pair	arXiv 2609.31473

Look at the game counts before the ratings. Claude Opus 5.5 has two games on the LLM Chess Benchmark, so its 95% range there runs from 517 to 1876: it tells you almost nothing yet. The Game Arena paper tested an older generation of models, which is why its numbers look so different.

How each LLM chess benchmark works

Dubesor's AI Chess Leaderboard is the biggest pool. Each model gets a starting rating from the accuracy of its first ten games, as judged by Stockfish 17.1, and then normal Elo updates after every AI-vs-AI game. It also publishes Stockfish accuracy and a blunder rate for every model, plus a separate puzzle rating against official Lichess puzzle ratings. On that puzzle table, Gemini 4 Argon (high) leads at 2193 ±85, GPT-6 Astra scores 1921 ±71 and Claude Opus 5.5 1631 ±72. Dubesor's page warns that Elo here is relative to its own pool.

Saplin's LLM Chess Leaderboard asks models to play through a short dialogue: they can ask for the board, ask for the legal moves, or submit a move. Breaking the format three times in one move, or talking for ten turns without moving, loses the game. That makes it partly a test of following instructions. The paper behind it (December 2025) found that many leading models struggled to finish games or win consistently, even against a random opponent, and that reasoning models clearly beat non-reasoning ones.

The LLM Chess Benchmark tries hardest to speak the language of human ratings. Models face anchors with fixed Lichess Classical ratings, and the results are mapped onto that scale with Glicko-2, a rating system that also tracks how uncertain each rating is. There's no rating floor, so weak models can go below zero. Its methodology page adds a caveat we agree with: the author expects models would do worse against humans, who are better at exploiting systematic weaknesses.

The Game Arena paper is the most controlled. Every pair of models plays 40 games, 20 with each colour, through public APIs with default settings and no tools. Ratings are relative, so the weakest model is set to 0. The authors used Stockfish to track who was winning during each game and found that the models start out balanced and drift apart in the middlegame.

What these benchmarks measure: playing, not explaining

All four boards score the same thing: can the model win a game and keep its moves legal. None of them checks whether the model explains a position correctly, which is what most people actually ask a chatbot to do after a game.

Those are different skills. A separate research benchmark, ChessQA (October 2025), tested understanding instead of play, across five areas from basic rules to tactics, judging positions and chess concepts. Its authors report persistent weaknesses in every area for the models they tested. So a model with a decent playing rating can still give a confident, wrong explanation.

That gap is the honest takeaway from every LLM chess leaderboard. Even on the board that maps to Lichess Classical ratings, where the best model is estimated at 2144, the models still blunder, and nothing inside the model checks its claims. For Claude in particular, see What is Claude's chess Elo? and Claude vs ChatGPT at chess. For what happens when a model meets a real engine, see ChatGPT vs Stockfish.

Why Chess Coach uses Stockfish for the facts

These leaderboards are the reason Chess Coach doesn't let the language model decide what's true about a position. Chess Coach is a connector for Claude and ChatGPT. It reviews your finished Chess.com and Lichess games with Stockfish on our own server, and every evaluation, best move and line comes from the engine. Your assistant's job is the part language models are good at: turning those facts into plain words for your level, on an interactive board with the key moments marked. That way, the assistant's own chess rating doesn't limit the quality of your review. The free plan includes one in-depth review a day. Connect Chess Coach, read the docs, or see how chess tools plug into Claude in our Claude chess skill guide.