It is shockingly difficult to use LLMs in chess study. I don't need it to be a better (or worse) Stockfish; an LLM should be great at taking a FEN or multiple lines from Stockfish via MCP or tool call and explain why positions are evaluated the way they are, typical plans in the position (drawing from pretraining knowledge of a vast archive of games), and how to explain to a human to study these positions.

I suspect that the large amount of chess pretraining data is not well synchronized with positions, because in books and articles the text is typically accompanied by pictures of the positions, NOT FENs / PGNs. So the training on the text is decoupled from the representation of the position.

Regarding your tool call thing with stockfish/maia, I made a tool like this for myself called Blunder Sniper which iteratively feeds positions - that I'm likely to get given my openings - and recursively calls the lichess DB and finds the first time in the top 80% played moves in each chain where the opponents in the rating range blunder as the most common move in the chain.

It was a fun way to use an alternative to engine-based preparation that many strong players use, which is something like Nibbler + lc0 and using contempt high values to find higher variance lines rather than game theory optimal ones.

Some day I'll expand on the gpt-chess articles [0] that I found super interesting, fine-tune models... well, I keep telling myself that, anyway...

[0]: https://dynomight.net/more-chess/