A Dance with Virtual Reality

After years of anticipation, the field of immersive media has come to life in the form of multi-channel projections and spherical domes or curved display screens to include our peripheral vision, all for shared user experiences, and virtual reality (VR) headsets that allow users to experience unique worlds.

Although full-sphere camera arrays and software are already available, gathering footage for these new forms of storytelling remains a challenge. At NTU’s School of Art, Design and Media (ADM), an interdisciplinary collective of specialists from different areas—ranging from conception and visualisation to special effects and movie sound and score—are using immersive media to capture, recreate or invent alternative realities.

Immersing in a tropical garden
In our research project, titled “Exploring the possibilities of capturing multi-sensory impressions to create immersive Virtual Reality environments, exemplified with the forking path structure of the Yunnan Garden at NTU”, I am working with ADM’s Assoc Prof Benjamin Seide and Asst Prof Ross Williams to investigate if the experience of being inside a tropical garden can be recreated with omnidirectional video or remodelled for VR environments. With its forking-path structure and surrounding vistas, NTU’s Yunnan Garden provides a good location for in-depth research in the field.

We began to capture the garden digitally with 360° video, for a piece called “Secret Detours”. For this, we engaged choreographer Susan Sentler and dancers from Singapore’s LASALLE College of the Arts to develop choreography that highlights the four cardinal directions of the garden, following the original layout of the landscape architecture.

This part of the ongoing project was captured with a multi-camera array, which did not provide any form of preview. It was a challenging experience as—instead of framing and directing the shoot—the team had to wait until the footage from all individual cameras was stitched together to view the resulting video (Figure 1).

Figure 1: Dressed in vermillion, azure, black and white (symbolising the south, east, north and west cardinal directions in Chinese mythology), dancers meet at forking paths in the tropical Yunnan Garden. Planar display of the video as equirectangular projection allows presentation on a sphere or cylinder. (Bottom) A different atmosphere of the garden is created through a winterly colour scheme.

The spherical high-resolution video footage was edited and exported to fit a range of different immersive environments such as cylindrical panoramas or hemispherical dome environments. The hemispherical dome at ADM invited people to sit down and enjoy the dancers’ performance, while the cylindrical panorama screen at ZKM (Centre for Art and Media) Karlsruhe, Germany, provided high-resolution details and engaged the audience to walk around and share the experience (Figure 2).

Figure 2: Immersive experience of “Secret Detours” in a seven-metre fulldome at ADM during the Multiversal Experiences symposium, organised by Elke Reinhuber and Benjamin Seide in February 2018 (top); Display in the cylindrical panorama at ZKM (Centre for Art and Media) Karlsruhe, Germany (bottom).

Another version of the video footage was optimised for the Media Art Nexus video wall on the NTU campus, where passers-by could view life-size representations of the dancers amidst the lush environment of the garden (Figure 3).

Figure 3: Dancers performing in the Yunnan Garden displayed life-size on the “Media Art Nexus” panoramic video wall at NTU. Credit: Ina Conradi.

A virtual walk in the park
As we want to reach different audiences with our research but need to consider constraints such as budget and space, we are also investigating presentation methodologies for single viewers, including the use of VR. Made possible through VR headsets, which only recently became widely accessible and are easily transportable, VR technologies provide distinct perspectives and experiences.

Using photogrammetry, we are currently creating a 3D model of the garden for a walk-through VR experience (Figure 4). This technique allows us to generate point clouds for the creation of objects such as stones, lamps or bridges based on photographs, which are then mapped with different textures. Once completed, the VR environment may be used in various settings: for instance as movie sets, an environment for games, a place to share photographic memories of different eras, or as an educational guide to the diverse flora and fauna found in the garden.

Figure 4: Developing a walk-through VR experience of the Yunnan Garden, using photogrammetry (top), point cloud processing (middle) and VR programming (bottom). Credit: (top, bottom) Benjamin Seide and Xu Peisen, (middle) Benjamin Seide.

Author: Elke Reinhuber

An industrial photographer by training and specialist in media arts, Asst Prof Elke Reinhuber of NTU’s School of Art, Design and Media explores alternative layers of reality using immersive environments, augmented reality and expanded photography. Read about her recent research in  Kultur und Informatik: Augmented Reality (2016), Verlag Werner Hülsbusch, Germany, ISBN: 386488103X; and Proceedings of Electronic Visualisation and the Arts, London (2018), BCS, The Chartered Institute for IT, ISBN: 978-1-78017-454-9.

This article appeared first in NTU’s research & innovation magazine Pushing Frontiers (issue #13, June 2018).