Occasionally we host guest posts on this blog. They are often topical, provocative or simply informative. This guest post by Dr. Iain Strachan is all three. As always, your comments and questions are welcome.
Can AI pass the Turing Test today? by Iain Strachan
In a Invisible Poets Road Show in Derby, Steve Wheeler asked me if I thought AI had passed the Turing test. I replied "Yes ... and No."
I had just read a poem "A chatBot named Christopher" about Alan Turing, where I had claimed that the answer is "No".
However, AI-generated poetry continues to fool us again and again. I have been fooled by it. I once praised someone's Villanelle on Invisible Poets that on closer examination turned out to be AI generated.
Why does this happen? I think it's to do with the way we interact with pop songs. I asked a member of my family "When you listen to a pop song, do you think about the words?" He replied "Not really. If I know the words, I'll sing along to the tune, but I don't think about what they mean."
I expect most people are like that. Pop songs have to be singable, so the lyrics fit the tune, and so we are only engaging with the words on a superficial level. So they need to flow smoothly, have simple rhyme schemes etc.
For thou'rt slave to fate, chance, kings and desperate men (John Donne), or
The soil/is bare now, nor can foot feel, being shod (Gerard Manley Hopkins).
See how the strong syllables pile up with no light syllables between. The Hopkins also has enjambment, where the sense carries on over the line break. Song lyrics don't do this - normally each line stands alone.
It's the same with AI generated poetry. It is polished, and flows nicely; each line is self-contained. But, whereas with a pop song, you can find depth and meaning in the lyrics: a story told, or a telling metaphor, if you examine an AI poem carefully, you won't find any depth; it falls apart as a sequence of poetic sounding phrases and clichés strung together with no clear overall message.
So if you find a poem that seems super smooth and polished, take a closer look before you enthuse about it. Don't give the AI fakers their serotonin boost! If it's AI, it will fall apart and you'll find the words of my chatBot poem to be still true:
Chatbots today can't pass the Turing Test
Their show of understanding's fake, at best.
Their show of understanding's fake, at best.
Iain Strachan
Image used under a Creative Commons License
Interesting. I hope it’s ok to say that I find AI to be perceptive in its analysis of poetry already written. I have sometimes asked it to look at something and it’s mentioned strengths and weaknesses. It suggests “improvements”. Now, I don’t accept those on principle but they have occasionally suggested ways forward and led me to more reflection. I don’t think think this has happened often enough to call it a practice but I think we all need to know what AI can offer positively rather than reject it wholesale - little straw man, there! This holds good in quite a few areas, I think. I put in, “Is this poem any good?” It was confused, so I said, “That’s the poem” and I got some interesting remarks about what constitutes art and how I could explore my “theme” further in a series.
ReplyDeleteExcellent insight, Iain. Thanks for sharing with us.
ReplyDeleteI have invented a term for a new feature I'm noticing in AI generated poetry.
ReplyDeleteThe term is "metaphor-soup".
chatGPT in analysing an obvious AI poem suggested that there are too many "stacked metaphors" just metaphors on top of each other that sound poetic but don't really mean anything.
I think "metaphor soup" is a term that describes this happening.
You might say the term is a meta-metaphor ie a metaphor about metaphors!
There was something I wanted to add about the depth of meaning of pop song lyrics, but was working within a word limit.
ReplyDeleteConsider the following line from "Eleanor Rigby":
Wearing a face which she keeps in a jar by the door.
I wonder how many people just sing along that line without thinking about it? We often see the phrase "putting on my face" ie putting on makeup. But _Wearing_ a face implies (to me) a mask that hides your true self, perhaps through shame or embarrassment, and that this blends with the overall message of the song about loneliness, perhaps the way we cut ourselves off from each other by hiding behind a mask or a face.
So I think the line is pretty deep. I think the lyrics of the song deserve more attention than just singing them along with the tune.
But you won't find that sort of depth in a poem written by AI. Pick out one of the croutons in the metaphor soup and you'll find it doesn't really harmonise with the rest.
Thank you for the blog Iain!! I am still rather old school in my approach… this AI creating poetry era reminds me of something out of a Kurt Vonnegut paperback. I’m disheartened by the entire affair… but am positive that humanity shall overcome the bland banality of it all in the end.
ReplyDeleteForever learning. Thanks for this Iain. X
ReplyDelete