A few weeks ago, I was preparing for an event where I had to talk about the history of butter in India. Normally, my routine is to first Google it and get a broad sense of the subject from the first few pages of search results. But, having decades of experience dealing with dubious blogs and content farms that search engine optimise their citation-free content, I use some of the search engine’s advanced tools to filter it down to sources I trust. These tend to be academic journals or actual excerpts from books. This is, very approximately, the workflow of anyone using the Internet for secondary research. Except, this time around, I got lazy and did what at least 100 million people are doing nowadays — I asked ChatGPT for a “crisp set of memorable facts about the history of butter in India as bullet points”.
And one of those bullet points was: “Butter was so valuable in ancient India that it was used as currency.” It doesn’t take an economics expert to realise that currencies don’t tend to be things that disintegrate at room temperature. Ancient Indians may have been financially liquid, but I’m sure they didn’t take it literally.
Artificial intelligence (AI) researchers, usually the ilk that will use incomprehensible terms such as “backpropagation” and “convolutional neural networks”, surprisingly termed this phenomenon with a memorable word: “hallucinations”. To understand AI hallucinations, we need to understand Large Language Models (LLMs), the underlying technology that powers AI bots such as ChatGPT. These are sophisticated pattern recognisers, trained on a vast ocean of text data, capable of generating human-like text based on the patterns they’ve learned.
Convincing, not accurate
First, it’s important to realise that the original design goal of an LLM is to be able to generate convincing human language, not factually accurate human language. That it is mostly able to do the job is down to the quality of the training data. As Ganesh Bagler, associate professor at the Infosys Centre for Artificial Intelligence at Indraprastha Institute of Information Technology, Delhi, points out, “While large language models benefit from patterns mined from an ocean of data, these statistical parrots can occasionally churn out nonsense.”
And in our butter example, the statistical parrot named ChatGPT, which has no deep, contextual understanding of cows, dairy, and monetary economics, made a connection that an adult human with a college degree would have filtered out for not making sense. Nothing in its training data explicitly stated that butter was not used as currency. Cows were indeed used as currency in many societies, and currency is valuable, like butter. The next logical leap makes no sense to us, but makes sense to how LLMs work.
While this example is mildly amusing, imagine a scenario where someone asks for help in diagnosing an illness or uses it to do legal research for a court case. And unsurprisingly, that is exactly what happened in New York where a law firm decided to use ChatGPT to do case research and the bot ended up fabricating most of it — an error that was rather painfully caught live in court.
So, while it might seem like its ability to rapidly provide responses to most day-to-day queries is impressive, the unpredictability of when it might fabricate answers can make it tricky. Author and historian Sidin Vadukut told me his favourite hallucination was when he used ChatGPT to recommend YouTube videos. “It used to pick actual videos, sometimes the right summaries, but then entirely fabricate hyperlinks,” he said.
Why does this happen?
When generating responses, an LLM uses probabilities based on patterns it has learned from millions of books and Internet articles, but it doesn’t understand context as we do. When we speak to each other and someone says, “Ramesh told Aravind that he failed”, our brains will seek additional clarification on who the pronoun is referring to — Ramesh or Aravind? We further attempt to use any existing knowledge we might have about them and guess which of the two is more likely to fail. Even if we don’t do all of that, our ears can still catch intonation differences in how someone says “he” and figure out who the pronoun points to. But an LLM’s job is to simply calculate probabilities and wing it.
Context is also often rooted in specific cultures. As we use AI tools more and more, it’s important to realise that a lot of the training data has a significant first-world bias. AI tools will vastly amplify and exacerbate existing biases.
When Madhu Menon, photographer and chef, asked Google Bard, another generative AI chatbot, for an authentic recipe from a Thai chef, he was quite surprised. “I asked for a Thai stir-fry recipe from a Thai person and it made up a completely fake name, a list of books they’d written [which didn’t exist], and even a bio for the chef from bits and pieces of other real people’s bios.”
Hallucinations will regularly lead to creative, but potentially dangerously inaccurate content generation. A rather interesting irony here is that the Indian education system largely rewards students who are able to generate content efficiently based on the patterns they’ve learned without testing to see if they have actually understood the subject.
ChatGPT is the absolute epitome of every student who cracks engineering entrance tests without truly understanding the underlying science.
Feeding biases
Sometimes, hallucinations can take on a life of their own if they feed existing confirmation biases in an already polarised populace. As Vimoh, YouTuber and writer, points out, “I recently asked ChatGPT about whether there have been beings in Hindu mythology that may be compared to robots and it made up entire stories claiming that certain characters were artificial constructs. When I pointed out that it was not so, it apologised and withdrew everything. I have found it to be useful as an aid, but it is less than reliable for research purposes.”
But to be fair, it is also a spectacular leap in computing technology. That we are able to converse in natural language with a bot that is pretty accurate most of the time is stunning. For all its faults, it is the greatest unpaid research assistant and intern you will ever have. The situation is a bit like how the occasional electric vehicle battery catching fire is bigger news than the millions that work perfectly fine. It’s not as if college students don’t make stuff up in their answer papers, but when a bot trained on the net sum of all human language hallucinates in consequential situations like healthcare or citizen services, it can be a problem.
So, the knee-jerk fear that this technology will result in large-scale job loss might be jumping the gun. The human in the loop is going to be far more crucial than breathless techno-utopian news articles might have you believe. A more pragmatic estimate is that it will make existing job roles significantly more productive.
How should we deal with hallucinations? Just like how we learned heuristics (not too well, to be fair) to deal with misinformation, it’s important to pick up a set of habits that will help us deal with this problem. For starters, AI, irrespective of its sophistication, doesn’t “comprehend” as humans do. Always assume that you need to bring additional context to AI-generated information. I often start with a question and once I get a response, I provide additional context and then ask it to regenerate. This helps address a fair amount of hallucination problems because the machine doesn’t hallucinate twice in the same way.
Cross-verification is key. Anyone researching anything must verify responses from these bots with citations in actual books or journals. Don’t blindly trust the sources the bot generates because it can occasionally hallucinate citations, too. Nowadays, when I’m lazy, I simply ask the same question to both Bard and ChatGPT (many more LLMs will be available in the near future) and see if their answers match.
Another important habit is, if you come across hallucinated or incorrect information, reporting it helps developers improve the model, so always use the like and dislike buttons liberally to help the AI get better over time.
As with everything in AI, improvements are also coming at a rapid clip. Every update to these bots is improving their ability to provide clearer data contexts, refining the AI’s self fact-checking ability, and also introducing new ways for users to guide and improve AI interactions. In fact, I won’t be surprised if this article itself will look hilariously dated in six months as the LLMs improve exponentially.
At this point, while we marvel at its ability to improve our creative productivity, understanding AI’s constantly evolving limitations is crucial. To hark back to our butter example, the Hindi expression ‘makhan lagaana’ means to praise someone shamelessly, but with AI, take the advice of the Buddha instead: ‘Question everything.’
The writer is a software professional and author.
