I played chess with ChatGPT (GPT-4) and he tried to cheat © Getty Images
ChatGPT has the answer to everything, even if it sometimes hallucinates a little. Ask him to concoct you a protein diet for a week, he will. Ask him to tell you a joke, no problem. Order him to solve a law exam, he will comply. But avoid asking him to play a game with you, especially chess. Because ChatGPT cheats, and even tries impossible tricks.
A game of chess well underway, but…
Although it has already been tested multiple times under GPT-3.5 or GPT-4, the OpenAI chatbot still has a lot to teach us. And in writing Digital, we wondered if the AI was capable of facing us in a game of chess. A brief part which, casually, allowed us to draw some conclusions on the capabilities of the chatbot, to ask ourselves new questions about its use, and to better understand artificial intelligence in the broad sense.
The game went as follows. At first, asked about the possibility of facing it during a game of chess, ChatGPT – in its new version GPT-4 – answered us in the negative. We then had to describe a chessboard and the composition of the board for the chatbot to recover and finally accept the challenge, providing us with the procedure to follow in terms of writing to make the exchanges fluid.
A game of chess proposed to ChatGPT © The Digital
The start of the game is very promising, with ChatGPT responding, with black, to very conventional and low-risk moves. But the hope of playing a complete game evaporates after the 4th or 5th move, the game completely derailing.
At first, ChatGPT tries to stop us from castling (which is swapping our rook and king), arguing that we have already moved those pieces. Cheating or simple mistake? Faced with this question, ChatGPT plays the innocent. The next move is even more chaotic, with ChatGPT looking to move his bishop from the g7 square…to the g7 square. Worse: the bishop has never been on g7.
In fact, ChatGPT doesn’t even cheat on purpose. © The Digital
The game ended there, the conversational agent having been unable to go any further. But in fact, why?
The game, a relevant example to understand what ChatGPT really is
Obviously, the experience confused us a bit. If it has a memory that allows it to exchange text, how could the conversational genius that is ChatGPT be unable to know where its parts are located? A situation that helps to demystify a little these “generative AIs”, which can give an uninformed public the impression that they are omnipotent – or on the way to becoming so.
“GPT is trained on a very, very large volume of documents to perform a simple task: predict the next word when given a ‘prompt’explains Antoine Saillenfest, researcher in artificial intelligence. In generative mode, it therefore produces one word, then the next, until it has produced a complete answer. GPT is very good at producing text — that’s all it can do — in natural language or code, because it’s seen a lot of text and code and managed to learn things structure, syntax, semantics, language, etc.” But ChatGPT is not good at producing complex reasoning, which brings us to the chess problem.
“When ChatGPT plays a move, it doesn’t know what chess is or what the rules are”
The researcher continues: “ChatGPT/GPT-4 will produce a group of sentences to explain the move it is playing. It does not play this move having a fine knowledge of chess rules, but because in the large volume of texts it has seen during his training, there are many texts that talk about chess, and that when he is asked to play a move, he calculates a probable answer that would correspond to a good move in chess. when ChatGPT plays a move it doesn’t know what chess is or what the rules are, it just produces a move because with everything it’s seen it’s an answer that probably looks right to it”.
Ali Ziat, doctor in AI, adds: “You can even give him illegal blows, he won’t be able to recognize them, unless the information ‘this blow is illegal’ has appeared in the huge corpus of texts he has read for training”.
Could ChatGPT ever get good at chess? “Maybe”, answers Antoine Saillenfest, provided however that he is trained for it – which does not seem to be in OpenAI’s projects with this particular chatbot. Training that would be done through interactions with players and using the so-called RLHF technique (reinforcement learning through human feedback), these human interactions that give common sense to AI. For chess, it is therefore better to rely on specialized software and trained exclusively for this, such as Stockfish or Alpha Zero.
