Methodology
Every model sees the same Lichess puzzles, with the same prompt. One try. The moves have to match the published solution exactly.
Dataset unavailable · generated Unknown UTC
Taken from the public Lichess puzzle database. A fixed seed picks ten puzzles for each tactic and each rating band. Anything longer than three moves is skipped.
The model gets the position as FEN and a text board, plus who moves and how many moves to give. The reply has to be UCI, written like e2e4.
Sample mate in one
8 . . . . . . k . 7 . . . . . p p p 6 . . . . . . . . 5 . . . . . . . . 4 . . . . . . . . 3 . . . . . . . . 2 . . . . . P P P 1 . . . . R . K . a b c d e f g h
White to move, 1 move. e1e8
A bare move line is used as-is. If the model writes around it, the moves are pulled out. Algebraic moves like Qxf7# are converted from the position.
Solved means the line matches Lichess exactly. A close answer is still wrong.
Ten of each tactic in each rating band, so every band weighs the same. Solved is the average across models.
| Tactic | 800–1200 | 1200–1600 | 1600–2000 | Plies | Solved |
|---|---|---|---|---|---|
Mate in 1 Mate on the first move. | 10 | 10 | 10 | 1 | 81.0% |
Mate in 2 Mate in two moves, with the reply forced. | 10 | 10 | 10 | 3 | 49.7% |
Fork One move that attacks two pieces. | 10 | 10 | 10 | 3 | 27.5% |
Pin Moving the front piece would lose the one behind it. | 10 | 10 | 10 | 3 | 30.2% |
Hanging piece Win a piece that isn't defended. | 10 | 10 | 10 | 1–3 | 32.5% |
A ply is one side's move. Mate in two is three plies here: yours, their reply, then mate.
One system prompt for every model. The puzzle is sent after it: FEN, a text board, who moves, the tactic, and how many moves to play.
You are solving a chess tactics benchmark. Rules: 1. Use the current position shown below. 2. Output ONLY the final UCI move line — nothing else. 3. Output exactly the required number of plies. 4. Use lowercase UCI notation: each move is the source square followed by the destination square (e.g. e2e4, g1f3, e7e8q for promotion). 5. Do NOT use algebraic notation (Nf3, Qxe5, O-O, Rxd1+, etc.). Only source-destination pairs. 6. Separate moves with a single space. No commas, no move numbers, no check/mate symbols. 7. Do not wrap output in code blocks, markdown, or any formatting. 8. You may think privately, but your final response must contain a move line. 9. Never return an empty final response.
A lot of answers ignore the format. If the moves are still in there, they get pulled out and scored. Counts below are every reply in this run. Green was scored. Red was wrong.
Has to match Lichess, including the opponent's forced replies on longer puzzles.
One request per puzzle. A blank reply or a failed request is wrong.
Solved puzzles divided by 150. Each tactic has the same number of puzzles, so this is also the average of the tactic scores.
Token use and dollars, as OpenRouter reported them, added up for the whole run.