There’s a lot of buzz around AI, or artificial intelligence, and how it may make our jobs easier -- or even replace them altogether! But for all of the potential benefits, AI still has a long way to go in many regards. Large language models (LLMs) like ChatGPT, Gemini, Llama, etc. are great at memorization and recollection of facts and details, but they are not so great at reasoning or analytical tasks. Take a simple question like, “How many Rs are in the word strawberry?” as an example. Our human brains look at the word and see three. I asked three AI models - ChatGPT, Gemini, and Meta AI - this question and got some interesting answers.
ChatGPT and Gemini both got the correct answer -- interesting, since in my prior research they both got this wrong in different ways, but more on that later. Meta AI, on the other hand, tells us:
There are 2 Rs in the word strawberry.
Are you sure about that? Let’s try again, but this time, show your work…
Let me recheck!
Here's the correct count:
S-T-R-A-W-B-E-R-R-Y
-
S (no R)
-
T (no R)
-
R (1 R)
-
A (no R)
-
W (no R)
-
B (no R)
-
E (no R)
-
R (1 R)
-
R (1 R)
-
Y (no R)
There are actually 3 Rs in the word strawberry! Thank you for correcting me!
Okay, got it on the second try by breaking it down letter-by-letter, but why did it think there were only two the first time?
The problem is in tokenization of data, how each model stores those pieces of information. Tokens can be full or even partial words; you can use OpenAI’s Tokenizer tool to see how it breaks down each question. It really boils down to statistical relationships from one token to another, which LLMs generally only analyze one at a time, and the model deciding which token frequently comes next. But considering two of the three got it right this time around, these models learn rapidly and create new tokens as words become more statistically significant.
Try asking, “How many Ws are in the word Mississippi?” The answer you get may not be the one you’d expect!
Nate McCoy