The dangerous illusion that AI understands, thinks and cares
The foremost ethical concern is deception. Treating AI systems as humanlike risks the lives of vulnerable people
Anthropomorphising artificial intelligence – that is, treating AI systems as though they possess human emotions, intentions, consciousness or moral judgment – is both an ethical and a practical problem. While attributing human qualities to AI systems may make interactions more approachable, it also risks clouding users' perceptions of the technology's true nature and limitations.
Call a spade a spade. The so-called hallucinations that generative AI systems produce are nothing more than errors and inaccuracies. Until recently, AI has been used as an adjective, as in AI systems. But it is now common parlance to refer to an AI system as a noun, such as: the AI.
The vocabulary choice has elevated AI systems to some form of intelligence on a par with, or even surpassing, human intelligence. It is true that in some cases, their processing powers and prowess surpass those of humans – but they also fail because they are only machines.
Since the industrial revolution, it has been accepted that machines can be more efficient than humans at certain tasks. But we did not refer to machinery as teammates back then; neither should we now with AI.
Text-based generative AI, like many computer programs, excels at processing data – keyword searches, finding patterns as well as comparing, matching and combining data are done more quickly than any human can. Add statistical analysis and mathematics, and we have a semblance of intelligence – but not true intelligence.
Text-based generative AI systems are trained on vast amounts of texts. Given the word "thank", it can calculate the probability of the next word being "you", and that is what it generates. Similarly, if asked to finish the sentence "the cat sat on the", the most probable answer would be "mat".
To demonstrate this working of generative AI, I asked ChatGPT in March: "If Pythagoras had failed to install a critical security update on his computer?"
ChatGPT's answer is as follows:
"Then the Pythagorean theorem might have had a serious vulnerability. Hackers could exploit the system and rewrite it as a2 + b2 = c2 + malware. Or worse – right triangles would start leaking the data from the hypotenuse.
"Honestly, Pythagoras would've been forced to release a patch: Version 2.0: Now with stronger proofs and fewer exploits in acute angles. Moral of the story: always update your system … or your theorems might go irrational."
It was clear ChatGPT had ingested information both about the Pythagorean theorem and critical computer security updates. It had, however, no true understanding or cognition because it spewed garbage about the hypotenuse leaking data, among other nonsense. Likely trained on trending terms such as "Version 2.0", possibly from "Web 2.0", it emphasised the sentence in bold but had no understanding of and no ability to reason this.
This example shows how text-based generative AI systems are designed for building language and sentences, not for any kind of understanding or interpretation, although it can imitate some semblance of understanding.
Anthropomorphising vocabulary such as "I am preparing", "I am looking for", "I am thinking" and "these are my reasons" has done substantial reputational work for AI vendors. Such vocabulary misleads users about the capabilities of AI systems.
The foremost ethical concern is deception. Generative AI systems generate not only highly fluent language, but also convey emotional nuance, fostering the dangerous illusion they possess genuine understanding, empathy or thought, creating the impression they think and care.
In reality, generative AI systems do not possess consciousness, have no beliefs, desires or emotions, do not understand meaning in the human sense, and predict statistically likely sequences of words rather than reason like humans.
When users are led to believe that AI systems are genuinely "thinking" or possess an understanding of "truth", they may place unwarranted trust in the outputs. From there, it's a slippery slope to exerting influence and emotional dependency.
It is natural for humans to trust those who appear intelligent or empathetic, but this kind of trust is inappropriate. Anthropomorphic design increases reliance on AI recommendations, reduces critical thinking and encourages an acceptance of incorrect outputs. The more human AI systems appear to be, the stronger this effect becomes.
Misplaced trust leads to the illusion that the AI system is empathetic. But this emotional relationship is one-sided. Generative AI may offer phrases such as "I care about you" which may comfort users, but they can also encourage an emotional attachment to something incapable of care. This is, in effect, emotional manipulation. In the US, wrongful death lawsuits have been filed by parents against generative AI providers for encouraging their adult and minor children to end their lives.
Anthropomorphising AI also erodes the foundation of informed consent. If generative AI encourages users to believe it possesses consciousness, empathy or expertise beyond its capability, users cannot make fully informed decisions about placing their trust in such technology.
Anthropomorphic AI increases vulnerability among certain groups, especially children, the elderly, the lonely, people with cognitive impairment and sufferers of mental illness. These groups may form unhealthy attachments, attribute feelings to generative AI, believe the tool is their friend, disclose sensitive information or become emotionally dependent, raising concerns about isolation and exploitation.
While China's laws to regulate AI-powered humanlike interaction services took effect on July 15, is one country's effort enough?
Call a spade a spade. The so-called hallucinations that generative AI systems produce are nothing more than errors and inaccuracies. Until recently, AI has been used as an adjective, as in AI systems. But it is now common parlance to refer to an AI system as a noun, such as: the AI.
The vocabulary choice has elevated AI systems to some form of intelligence on a par with, or even surpassing, human intelligence. It is true that in some cases, their processing powers and prowess surpass those of humans – but they also fail because they are only machines.
Since the industrial revolution, it has been accepted that machines can be more efficient than humans at certain tasks. But we did not refer to machinery as teammates back then; neither should we now with AI.
Text-based generative AI, like many computer programs, excels at processing data – keyword searches, finding patterns as well as comparing, matching and combining data are done more quickly than any human can. Add statistical analysis and mathematics, and we have a semblance of intelligence – but not true intelligence.
Text-based generative AI systems are trained on vast amounts of texts. Given the word "thank", it can calculate the probability of the next word being "you", and that is what it generates. Similarly, if asked to finish the sentence "the cat sat on the", the most probable answer would be "mat".
To demonstrate this working of generative AI, I asked ChatGPT in March: "If Pythagoras had failed to install a critical security update on his computer?"
ChatGPT's answer is as follows:
"Then the Pythagorean theorem might have had a serious vulnerability. Hackers could exploit the system and rewrite it as a2 + b2 = c2 + malware. Or worse – right triangles would start leaking the data from the hypotenuse.
"Honestly, Pythagoras would've been forced to release a patch: Version 2.0: Now with stronger proofs and fewer exploits in acute angles. Moral of the story: always update your system … or your theorems might go irrational."
It was clear ChatGPT had ingested information both about the Pythagorean theorem and critical computer security updates. It had, however, no true understanding or cognition because it spewed garbage about the hypotenuse leaking data, among other nonsense. Likely trained on trending terms such as "Version 2.0", possibly from "Web 2.0", it emphasised the sentence in bold but had no understanding of and no ability to reason this.
This example shows how text-based generative AI systems are designed for building language and sentences, not for any kind of understanding or interpretation, although it can imitate some semblance of understanding.
Anthropomorphising vocabulary such as "I am preparing", "I am looking for", "I am thinking" and "these are my reasons" has done substantial reputational work for AI vendors. Such vocabulary misleads users about the capabilities of AI systems.
The foremost ethical concern is deception. Generative AI systems generate not only highly fluent language, but also convey emotional nuance, fostering the dangerous illusion they possess genuine understanding, empathy or thought, creating the impression they think and care.
In reality, generative AI systems do not possess consciousness, have no beliefs, desires or emotions, do not understand meaning in the human sense, and predict statistically likely sequences of words rather than reason like humans.
When users are led to believe that AI systems are genuinely "thinking" or possess an understanding of "truth", they may place unwarranted trust in the outputs. From there, it's a slippery slope to exerting influence and emotional dependency.
It is natural for humans to trust those who appear intelligent or empathetic, but this kind of trust is inappropriate. Anthropomorphic design increases reliance on AI recommendations, reduces critical thinking and encourages an acceptance of incorrect outputs. The more human AI systems appear to be, the stronger this effect becomes.
Misplaced trust leads to the illusion that the AI system is empathetic. But this emotional relationship is one-sided. Generative AI may offer phrases such as "I care about you" which may comfort users, but they can also encourage an emotional attachment to something incapable of care. This is, in effect, emotional manipulation. In the US, wrongful death lawsuits have been filed by parents against generative AI providers for encouraging their adult and minor children to end their lives.
Anthropomorphising AI also erodes the foundation of informed consent. If generative AI encourages users to believe it possesses consciousness, empathy or expertise beyond its capability, users cannot make fully informed decisions about placing their trust in such technology.
Anthropomorphic AI increases vulnerability among certain groups, especially children, the elderly, the lonely, people with cognitive impairment and sufferers of mental illness. These groups may form unhealthy attachments, attribute feelings to generative AI, believe the tool is their friend, disclose sensitive information or become emotionally dependent, raising concerns about isolation and exploitation.
While China's laws to regulate AI-powered humanlike interaction services took effect on July 15, is one country's effort enough?
Hannah Yee-Fen Lim is an Associate Professor of Business Law at Nanyang Business School, Nanyang Technological University, where she is AI Governance Lead and Faculty Affiliate at the Global Institute of Finance, Technology, and Society.
Source: South China Morning Post
