Every model gets the same puzzles from the Lichess puzzle database (opens in new tab), the same prompt, and one try at each. It plays one move at a time, as a person would on Lichess, and every move has to be the solution move or checkmate.
lichess-089f2c83df37
generated Nov 10, 2026, 11:49 PM UTC
01
Four steps, the same for every model and every puzzle.
Drawn from the public Lichess puzzle database. A fixed seed takes 10 puzzles per tactic per rating band, up to three plies long, and only puzzles played 1,000+ times that players rate well.
The model sees the position after the opponent's move: FEN, a text board, its side and the tactic. It answers with one move in UCI, like e2e4.
A bare move is used as-is. Otherwise the move on the last line is taken, and algebraic moves like Qxf7# are converted.
Each move must be the Lichess move or give checkmate. If it is and the puzzle goes on, Lichess's reply is played and the model moves again.
What the model sees
mate in 18 . . . . . . k . 7 . . . . . p p p 6 . . . . . . . . 5 . . . . . . . . 4 . . . . . . . . 3 . . . . . . . . 2 . . . . . P P P 1 . . . . R . K . a b c d e f g h
Task
You play White, mate in 1
Expected
e1e8
How a reply is read
02
10 puzzles in every cell, so each rating band weighs the same. Cells show the share of models that solved them.
A ply is one side's move. Mate in two is three plies: your move, the opponent's reply (played for the model from the Lichess solution), then mate.
03
One system prompt for every model, then one message per move. Nothing that identifies the puzzle, like its Lichess id or rating, is sent.
System prompt
You are solving chess puzzles taken from real games. In each puzzle one side has a winning tactic, and you play that side. You answer with one move at a time. After each correct move you are shown the opponent's reply and the new position, and you play your next move, until the puzzle is complete. Answer format: - Reply with exactly one move in lowercase UCI notation: the source square followed by the destination square, for example e2e4 or g1f3. - Castling is written as the king's move: e1g1, e1c1, e8g8 or e8c8. - Promotions add the piece letter: e7e8q, a2a1n. - Do not use SAN or algebraic notation (Nf3, Qxe5, O-O). No move numbers, check symbols, punctuation, markdown or explanation. - You may think before answering, but your final response must be only the move.
First message of a puzzle
examplePuzzle type: Mate in 1 You play White. Find the move that checkmates immediately. Opponent's last move: f8g8 (Kg8) Position (FEN): 6k1/5ppp/8/8/8/8/5PPP/4R1K1 w - - 0 1 Board (White's side at the bottom; uppercase = White, lowercase = Black): 8 . . . . . . k . 7 . . . . . p p p 6 . . . . . . . . 5 . . . . . . . . 4 . . . . . . . . 3 . . . . . . . . 2 . . . . . P P P 1 . . . . R . K . a b c d e f g h White to move. Reply with your move in UCI.
After a correct move, the next message in the same conversation starts “Correct. The opponent replied …” and shows the new position the same way. Earlier thinking isn't sent back, so each message carries the full position.
04
Some answers ignore the format. If the move is still clear, it's read and scored on the board; if not, the puzzle is wrong. 95.2% of puzzle attempts across these runs could be read. Each attempt is counted by its first problem, or by why it ended.
Scored
4,998 · 95.2%
Marked wrong
252 · 4.8%
05
Each of the model's moves must be the Lichess move, or any move that gives checkmate. The opponent's replies come from Lichess; the model never has to guess them.
One answer per move, no second chances. A wrong, illegal, unreadable or blank answer ends the puzzle. API failures are retried, never scored.
Solved puzzles divided by 150. Every tactic has the same number of puzzles, so this is also the average of the tactic scores.
Tokens as OpenRouter reported them, priced at the model's standard API rate and summed over every move of the run.
06
Read the leaderboard with these in mind.
One right move
Apart from checkmate, only the Lichess move counts, as on Lichess. Another move that also wins is marked wrong.
Small gaps are noise
One puzzle is worth about 0.7 points, and with 150 puzzles a score of 70% is only accurate to about ±7 points. Treat close scores as ties.
Single run
Each model was run once, with its provider’s default sampling. A second run can score differently.
Tactic is named
The prompt says which tactic to look for (mate in 2, fork…). Lichess’s rated puzzles don’t, so this is easier.
Rating range
Puzzles are rated 1200–2400. Strong models may get close to the ceiling.
Text only
The board is sent as text. Models never see a picture of the position.