What if AI is not all that safe? | Associate Professor Hannah Yee-Fen Lim

In Episode 5 of Nanyang Business School’s “What If” video series, we delve into the complex risks associated with artificial intelligence. With expertise spanning both computer science and law, Associate Professor Hannah Yee-Fen Lim – who’s also the AI Governance Lead at NTU’s Global Institute of Finance, Technology, and Society (GIFTS) and a member of the IMDA Working Group on the IMDA Discussion Paper Legal Responsibility for AI Agents – unpacks why we have to take AI Safety seriously and start understanding the profound legal and business risks of autonomous systems. 

AI is Not a Being with Intelligence: The Danger of Vocabulary

AI systems pose a myriad of risks to organisations: financial, legal, reputational, cybersecurity, and even bodily risks, so they are not 100% safe. Much depends on what kind of AI system is used, how it’s deployed, what kind of use cases are involved, and who the vendors of the AI systems are.

Managing these risks must start with understanding AI at a conceptual level. However, a fundamental problem lies in our vocabulary. In recent years, it has become common to refer to AI as a noun, “the AI”, as if it’s a “thing”.

“Treating it as a noun and as a thing has elevated AI systems to some sort of intelligence on par with, or even better than, human intelligence,” Assoc Prof Hannah Lim notes. “This really is a problem, because it obscures all the risks that AI systems pose.”

In reality, AI is an adjective because it refers to various techniques and methods used in computer science and is rooted in mathematics, statistics, and probability. Using terms like “hallucinations” to describe the errors and mistakes that generative AI commonly produces humanises the system and is a quiet and subtle mechanism to play down the unsafe nature of generative AI systems. For any other computer system, we would simply call this a “programming bug”. If an ATM told you that your $1,000 salary was actually $10, you would not say the machine hallucinated; you would call it a bug and chase after your bank.

The Illusion of Cognition and the Risks of Probability

In truth, AI systems have no cognition. They operate on lots of mathematics and probability. Here are some examples:

• Autonomous vehicles: If an autonomous vehicle is trained on 10 million datasets of speeding through red traffic lights, and only 100 datasets of braking and stopping, it’s going to learn to speed through red traffic lights.
• Text-based generative AI: Text-based AI takes the same concept and applies it to the construction of language and sentences. If you input the word “thank”, it statistically calculates that the most likely next word should be “you” and that is what it will generate – “thank you”.

To demonstrate how generative AI systems are just predicting statistically likely sequences of words rather than reasoning like humans, Assoc Prof Lim asked ChatGPT: “If Pythagoras failed to install a critical security update on his computer?”

The answer from ChatGPT included nonsense like “right triangles would start leaking data from the hypotenuse” and that “Pythagoras would have been forced to release a patch”. The system recognised keywords (Pythagoras, security update) and strung together statistically likely sequences of words around these two topics, proving it possessed no actual comprehension of the concepts.

Real-World Consequences: Corporate Liability

When business leaders understand that generative AI is just predicting sequences without reasoning, they should become aware of the significant risks of deploying it for high-stakes client workflows. These are not theoretical residual risks. Organisations are being held liable for the false statements given by their AI chatbots.

• The Air Canada Case (2024): Air Canada’s generative AI chatbot gave a passenger incorrect information regarding discounts. The passenger sued. Air Canada argued that the chatbot was a separate legal entity and the airline was not responsible for its outputs. The tribunal rejected this, ruling that the airline was fully liable and ordering it to pay damages, resulting in front-page news and reputational damage.
• The German Court cases (2026): In May, German courts held two separate companies liable for the words of their AI systems, including a medical clinic that misrepresented its doctors' qualifications and Google for false AI overviews.

Organisations are liable for false statements given by their AI chatbots, even when the model was supposedly trained exclusively on accurate internal data, and even in the presence of disclaimers.

Realising the Risks and Business Implications

There are many different types of AI systems, so it would be impossible to list all the risks for all businesses. Case in point: Microsoft’s AI assistant, Copilot, read the confidential emails of its users by mistake earlier this year, when it was not supposed to. This is already a breach of contract law and data protection law. Because AI agents can now take actions, such as browsing, executing code, and using tools, the question of risk has moved directly into the realm of corporate governance. There needs to be a fundamental re-evaluation of how AI tools are integrated into organisations, especially in high-stakes client workflows.

“The board of directors will be held accountable regardless of whether they understand their AI tools,” Assoc Prof Lim emphasises. A proper governance structure needs to be in place to control the AI systems. To do this, board members must understand the AI technologies that are being deployed, how they function, and their limitations so that they can ascertain the risks.

Looking Ahead and the Interdisciplinary Solution

For policymakers, guardrails will need to be introduced so that tools such as Mythos, which are not safe for society if released to the general public, do not fall into the wrong hands.

For business leaders, developing AI strategies needs to start with understanding the different types of AI technologies, understanding where and why AI systems perform well, and where they do not. And the best way to mitigate these risks is to ensure that corporate teams include members with interdisciplinary backgrounds. NTU’s Global Institute of Finance, Technology, and Society (GIFTS) champions this very transdisciplinary collaboration to ensure the future digital economy and society remains safe, trustworthy, resilient, and aligned with human values.