AI Transforms RGB-D Images Into an Impressive 3D Format

This is an updated version.

In 2018, Facebook introduced a machine learning-based 3D photo feature which enabled users to generate an immersive 3D image from any ordinary photo. This was an “almost perfect” 3D image generator — yes it would grab your friends’ attention, but the background renderings were pretty blurry. Now, a research group from Virginia Tech, National Tsing Hua University and Facebook has introduced a game-changing algorithm that generates impressive 3D photos from a single RGB-D (colour and depth) image.

Unlike depth-based warping techniques that produce gaps or stretch existing image content or the Facebook 3D photo approach which can produce unrealistic surface textures, the proposed method leads to much more photorealistic results. Taking an RGB-D image as input, researchers use a Layered Depth Image (LDI) technique with explicit pixel connectivity as the underlying representation. The learning-based inpainting model synthesizes new colour or depth textures and structures into the occluded regions of the image in a spatial context-aware manner. The generated 3D photos can be rendered with motion parallax using standard graphics engines.

Researchers compared the new approach with MPI based methods on the RealEstate10K dataset and quantified performance using the common SSIM and PSNR image similarity tests on the synthesized target views and the ground truth. The LPIPS (Learned Perceptual Image Patch Similarity) metric was also included to quantify the performance of the generated view compared to human perception. The proposed method showed similar performance on SSIM and PSNR metrics, while LPIS scores indicated the synthesis views exhibit better perceptual quality. Researchers validated the method on a wide variety of everyday scenes, where it produces considerably fewer visual artifacts compared with state-of-the-art novel view synthesis techniques.

The paper 3D Photography using Context-aware Layered Depth Inpainting is on arXiv. This research’s GitHub page is here. The research group has also introduced a Chrome extension that can add depth parallax on images from Instagram profile pages.

---

Author: Yuqing Li | Editor: Michael Sarazen