Advancing Embodied AI: Prof Yang’s Research on Intelligent Robotics

NTU research team receives NVIDIA Academic Grant to develop multimodal AI models that enable robots to perceive and interact with the physical world.

A research team led by Assistant Professor Yang Jianfei at Nanyang Technological University has been awarded a prestigious robotics grant from NVIDIA through the NVIDIA Academic Grant Program. This competitive initiative supports groundbreaking academic research by providing universities with world-class computing resources.

As part of the award, Prof Yang’s team will receive four NVIDIA RTX PRO 6000 GPUs and four Jetson AGX Orin platforms. These resources will support the training of robotic foundation models, robot simulation, and data generation for real-world robotic learning.

Teaching Robots to Perceive and Act in the Real World

Prof Yang’s project, titled “Multimodal Language Action Model for Robotic Manipulation with Reinforcement Learning,” addresses a central challenge in robotics: enabling robots to function reliably in everyday environments such as homes. Traditional visual systems often struggle in low-light conditions or privacy-sensitive spaces, yet robots increasingly need to operate in these settings.

To tackle this, the team is developing a Multimodal Language Action (MLA) model that integrates information from multiple sensory inputs, including vision, tactile sensors, audio, infrared, and mmWave radar, combined with reinforcement learning. This approach allows robots to perceive, reason, and act with greater reliability across unpredictable real-world scenarios.

Advancing Physical AI for human-centred applications

Beyond this project, Prof Yang’s broader research focuses on Embodied AI, where intelligent algorithms enable robots and embodied agents to perceive, reason, and act in the physical world. A major emphasis is human-centred embodied AI, allowing robots to understand human behaviour, adapt to individual users, and operate safely alongside people.

The team also explores deploying advanced AI on edge devices using model compression and quantisation techniques. This ensures that foundation models can run efficiently on robotic hardware with low latency and high autonomy.

The multimodal learning framework under development has potential applications in healthcare assistance, household support, and security systems, including environments that are privacy-sensitive or low-light.

“The grant will help the team enable robots with cutting-edge AI models so that robots can learn to perceive, understand, and interact with the physical world in real time,” said Assistant Professor Yang Jianfei.

Learn more about Prof Yang’s research and projects:
https://marsyang.site/research/, https://marslab.tech/