It will come as no surprise that I was a member of the chess club and that I’ve played “The Game of Kings” for almost six decades. Digital Chess has been a part of my life for almost as long, starting with Boris, the Chess computer that I played to death, and continuing in 1986 or so with Kasparov Pocket Chess from SciSys.
In chess circles, that name has long carried significant weight. Gary Kasparov became a world chess champion in 1985 and memorably lost to IBM’s Deep Blue more than a decade later. It was, at the time, considered a turning point for computational power and the nascent field of artificial intelligence.
Fast forward almost 30 years, and AI chatbots like Gemini, Copilot, Claude, and ChatGPT put seemingly Deep Blue-level power in the palms of our hands. OpenAI’s ChatGPT seems particularly adept at the most complex questions, or at least never hesitates to answer one. Surely, it could handle a 1,500-year-old board game.
I’d read that ChatGPT had not fared particularly well against some computer Chess systems. Skeptical, I decided to dust off my Pocket Chess game and pit it against ChatGPT’s 4o model.
The aptly named Pocket Chess is a small, battery-operated chess game with tiny physical pieces and a pressure-sensitive board. There are red LEDs along two axes that light up to show you which piece Pocket Chess wants to move, and when you press the indicated piece down on the board, the new coordinates light up. This is also how you move your own pieces: a press of the piece you want to move and another press in the new position.
For a time, I traveled the world with this digital game and used it for hours on flights from New York to California and even a few to Europe and South Korea. The advent of smartphone chess apps marked the end of my days playing Pocket Chess.
Opening gambit
Before I could start my little AI Chess experiment, I had to pull some old AAA batteries and clear a few corroded contacts. Soon, I was watching the LEDs serially light to telegraph the launch sequence and hearing those familiar “ready to play” beeps.
This game, though, would not be for me. I opened ChatGPT and asked in a prompt if it wanted to play Chess. Ever the eager bunny, ChatGPT said, “Absolutely, let’s play,” and instantly presented a text-based chessboard.
I took a photo of Pocket Chess and wrote, “I have a better idea.” I wrote, “I want you to play against this Pocket Chess Board Game. You’re white. So what’s your opening move?”
This game did not last. I had wanted ChatGPT to face Pocket Chess at its best, so I’d set it to Level 8 (out of a possible 8), but quickly found that the thinking time for each move was minutes long. I put Pocket Chess on Level 1 and restarted, thinking that if ChatGPT won too easily, I could level up on the next match.
The plan was for me to take photos of the board to show ChatGPT each of Pocket Chess’ moves. ChatGPT would stand in for me and play white.
ChatGPT opened with the classic e2 to e4, which is a straight, two-position move of the central pawn. Pocket Chess instantly responded by moving a pawn two positions forward, so it was opposite mine.
ChatGPT misinterpreted the move and thought that Pocket Chess had employed the Scandinavian Defense, which would have put its pawn diagonally from mine. That’s a much more aggressive play.
I informed ChatGPT of its error, explaining that there was no pawn in that position. ChatGPT apologized, reset a bit, and gave me a fresh move.
After a few moves in which I tried repeatedly to share clear images of the game board with ChatGPT, and it made numerous errors, I switched to telling it chessboard coordinates. This improved play, but as I gave up piece after piece, it became clear that we were not exactly winning.
Perhaps sensing my confusion and frustration, ChatGPT began giving me regular updates on the “Current Position Overview.” This helped because I could verify that it knew where all our pieces were.
ChatGPT still struggled to keep track of the game properly. At one point, I had to remind it that we had not lost a knight. I took another photo of the board and shared it. Incapable of embarrassment, ChatGPT replied, “You’re absolutely right — I stand corrected, and thank you for the clear image.”
In another situation, it tried to slide a Castle through the Knight sitting right next to it. We’d never moved that piece, and there was no reason for ChatGPT to lose track of it.
We continued on this way with me losing more pieces than Pocket Chess, which, I can say, never made a mistake
Things did not improve from there. I had to correct it on piece positions every other move. It got so bad that ChatGPT didn’t even know we were in check until I told it:
Me: “You know we’re in check, right?”
ChatGPT: “You’re absolutely right – and I missed a critical detail. let’s fix that immediately.”
It’s no wonder that I gave up so many pieces and eventually lost in a checkmate. ChatGPT 4o seemed incapable of keeping perfect track of the board and our pieces.
If you know anything about chess, then you know that our loss was inevitable. Playing chess is not just about next move strategy; it’s about the next three moves (and beyond). If you don’t know what pieces you have or where they are on the board, that’s impossible.
ChatGPT’s inventory of pieces was usually correct, but when it delivered the next move, it was as if it forgot the board.
I get why AI programmers have long focused on games like Chess and Go as tests for their artificial brains. The number of pieces and positions puts the possibilities into the trillions. It’s a lot for humans to manage, but perhaps not that easy for computers, either.
AI is excellent at digging through linear data to find the right information and present it in a way that seems almost human. However, it’s still not that great at what I like to call non-linear logic, seeing around corners, and sifting through millions of possibilities. Perhaps that’s what tripped up ChatGPT.
Image 1 of 2
I would not necessarily credit the Pocket Chess win with Kasparov’s game brain. There is no evidence that Kapsorov’s moves were programmed into the digital set. Even so, in the instruction manual’s foreword, Kasparov wrote for the game, the chess champ sounds like an oracle.
“When computers were first invented just four decades ago, few people realized that mankind was witness to the most important single development of our time. Today, computers have become freely available, and in a few years, there will be a computer in almost every household.”
And now we have them in every pocket, and yet this ubiquitous AI still can’t beat a chess computer from a bygone era. Guess those old systems were just built differently.