I think we all know about the problems that large language models (LLMs) have with counting the number of occurrences of the letter r in strawberry. Modern LLMs have that fixed.
Now, how about a different test? Aspartyltryptophan is a word I did not make up. It’s the name for a compound, and you can find out more about it here. Why do I bring it up? Well, it has a lot of consecutive consonants: 9 of them.
aspartyltryptophan –> a sp a rtyltrypt o ph a n
You will note that the longest block of consonants is 9 letters in length. (Is y a vowel here? Don’t think about it too much; that’s not the point.)
Let’s ask ChatGPT about this.
Okay, ChatGPT responds with the wrong number and the wrong block. Let’s see if we can help it figure out its mistake.
Well, the code is correct and I love the confidence, but the answer is wrong. Time to call ChatGPT on its mistake, because running the code gives the correct answer: 9.
That’s not at all what the code does. Let’s ask for more detail.
(Skipping…)
This resulted in getting the correct output! The number of consonants is, in fact, 9! Getting the correct result from tracing is quite interesting. Let’s see if ChatGPT has learned anything.
This feels like more of an accomplshment than it probably is. Just don’t look too closely at the sequence of letters ChatGPT found.