SCSE research team creates new research technology – Pluralistic Image Completion

An SCSE research team, comprising PhD student Zheng Chuanxia and professors Cham Tat Jen and Cai Jianfei, has created a new research technology, called Pluralistic Image Completion. This technology enables partially erased images to be automatically completed, or “inpainted”, with not only one but multiple different plausible solutions. These new images appear real and visually seamless, but are in fact synthesised from scratch and do not exist in reality.

In the software implementation, a user selects or provides an input image, and further erases various parts of this image. The system then automatically synthesises new content to fill in the erased parts, in a manner such that the remaining visible parts and the newly generated imagery blend seamlessly. Most significantly, by repeated sampling the system is able to create a wide range of very different, yet visually plausible, imagery for the same partially erased image, which contrasts to previous methods where only one realistic image can be generated, or with very limited variation. In the team’s case, the greater the area of the image that has been erased, the greater the variety in the visually realistic content that can be generated.

The potential uses for such a system include removing unwanted large artefacts in a photo, or changing facial expressions or identities of people in an image, or varying the landscapes of both urban and natural scenes. The large variety in realistic synthesised images further allows users to choose the most appropriate image to be used for their purposes, rather than be restricted to a single output from the system.

The technology is based on important and mathematically principled innovations in deep learning-based methods, and will be presented at the prestigious Conference on Computer Vision and Pattern Recognition (CVPR) in June 2019. This work is supported in part by the NRF-funded BeingTogether Centre in NTU.

Watch the demo video at https://youtu.be/tplhXSYX9us, or try out a limited web version of the interactive demo itself at http://www.chuanxiaz.com/project/pluralistic/.