Of cities, cadavers and cells

Forging new paths in interactive media, scientists at Fraunhofer Singapore in NTU are changing the way we see the world.

Just as the arrival of the printing press transformed the way information was transmitted around the world, emerging technologies like augmented and virtual reality are changing the way we see the future. In fact, our mobile phones are now so advanced that they are essentially supercomputers in our pockets, says Prof Wolfgang Müller-Wittig, Executive Director of Fraunhofer Singapore, an applied research centre at NTU.

These and other technological advancements mean that media need not be restricted to the printed page and can now be more interactive, personalised and omnipresent. “With things like social media and augmented reality, we have the possibility of realising truly interactive media, and much more importantly, linking media to the physical world,” says Prof Müller-Wittig.

Building on foundations laid by the Fraunhofer Project Centre for Interactive Digital Media (Fraunhofer IDM@NTU), set up in 2010, which in turn evolved from the Centre for Advanced Media Technology that was opened in 1998, Fraunhofer Singapore is at the forefront of developing immersive, real-time visual environments that will make waves in industry and society.

Seeing cities in a new light

The challenge of developing such interactive media is twofold, Prof Müller-Wittig explains. On the one hand, there is the technical challenge of accurately representing a large volume of complex data, and on the other hand, there is a need to create a digital interface that would allow users to easily access and understand that data. This is particularly challenging in the case of cities, which are both highly complex and dynamic, and yet must be understood by city planners in a real-time manner.

“The trend in this area is to create a digital replica, which Singapore has done with its Virtual Singapore project,” Prof Müller-Wittig says. “Virtual Singapore allows planners and policymakers to integrate information such as energy consumption, transport and infrastructure into a dynamic three-dimensional map of Singapore, helping not only the public but users such as construction companies, researchers and so on.”

Led by the National Research Foundation Singapore, the Singapore Land Authority and the Government Technology Agency of Singapore, Virtual Singapore has produced promising initial results and is moving into its final phase. “This stage is very exciting. Fraunhofer Singapore is now working with the Building and Construction Authority and stakeholders like the Housing and Development Board to roll out a showroom where potential customers can use our platform to explore not only the geometry and aesthetics of a building, but also operational information such as what materials are used for the foundation,” Prof Müller-Wittig shares.

Mixed reality to interactively explore operational information of residential units. Credit: Greatearth/Fraunhofer Singapore.

“Very soon, we will be handing over our platform to the respective construction companies for them to load data of their own buildings for exploration and further feedback. In the next innovation stage, we hope to bring our technology from the showroom to the construction site, using augmented reality and mobile platforms as tools for quality management.”

In the long run, Prof Müller-Wittig envisions that interactive media will be integrated into the entire life cycle of a building to enhance planning and facility management. “Being able to dynamically understand energy consumption patterns, for example, would help Singapore achieve its Smart Nation objectives,” he says.

No cadaver, no problem

One other area where interactive media will have a profound impact is in education and outreach, according to Prof Müller-Wittig. In close collaboration with NTU’s Lee Kong Chian School of Medicine,  Fraunhofer Singapore has been using augmented reality to bring plastinated specimens to life, allowing students to learn more about anatomy without the need for cadavers.

“Being able to overlay digital and visual information on top of a user’s view of real objects brings a new dimension to learning and the anatomical structures much closer to the students,” he says. “Furthermore, the threshold for adopting these technologies is very low; our medical students now all use iPads as their personalised learning environment to access teaching materials.”

But digital cadavers are just one of the many ways interactive media can be used to train medical students. Augmented reality can also help would-be doctors gain a better understanding of complex machinery such as computed tomography scanners, Prof Müller-Wittig says. Furthermore, there is great potential in using interactive media outside of medical education for outreach to the wider public.

In collaboration with Prof Gerhard Grüber of NTU’s School of Biological Sciences, Fraunhofer Singapore developed an interactive platform that makes it possible to investigate the dynamics of protein structures, with games conveying the idea of proteins as molecular machines.

“Biomolecule visualisation is a traditional use of visual computing, but we developed a tool that provides interactivity, converting complex information into an easily digestible visual story,” says Prof Müller-Wittig. “This gives the public insight into how protein structures can have an impact on their health. Importantly, it also helps people understand what is being done with research funding, and how research outcomes can affect society and their personal lives.”

The brain and beyond

As exciting as all these current applications of interactive media may be, they are just the beginning of what we can expect in the future, Prof Müller-Wittig says. Researchers at Fraunhofer Singapore are currently designing brain-computer interfaces and wearable devices that can shed light on the mental status of, say, drivers for fatigue detection or consumers for neuromarketing. Sensors embedded into the environment or even our own bodies are also likely to be active areas of research, he says.

“While there are still technical challenges like accuracy and robustness, we should not be hindered by the limitations we currently face. Technology will only get better, perhaps in unimaginable ways,” Prof Müller-Wittig adds. “It doesn’t matter if it will be used by a consumer or an expert; interactive media will help provide a tailored experience.”

The article appeared first in NTU's research & innovation magazine  Pushing Frontiers (issue #15, June 2019).