Facebook Open-Sources PyTorch3D CV Library

Although 3D computer vision is an increasingly important part of the computer vision research field, studies on 3D in deep learning have been limited by a lack of available tools and resources that can deal with the complexities involved in applying rich 3D data on neural networks. In a bid to simplify 3D deep learning and improve processing performance and efficiency, Facebook recently introduced an open-source framework for 3D computer vision. PyTorch3D is an efficient and reusable 3D computer vision library based on PyTorch that outperforms existing tools in many aspects of 3D modeling, rendering and other processing operations.

The Facebook researchers also propose a new 3D data structure, Meshes, for batching heterogeneous meshes in deep learning applications with better efficiency and flexibility. Meshes enable researchers to store and transform triangle meshes into different representation views easily and quickly.

The project includes an efficient new modular differential renderer for converting 3D scene properties into the pixel information of a 2D image. Traditional rendering methods are generally non-differentiable and so cannot be combined with deep learning. To address this issue, researchers developed an efficient, modular and differentiable renderer that is easy to customize, enabling users for example to change light and shadow effects during rendering. Like other PyTorch3D operators, the renderer can also handle batches of heterogeneous data.

The researchers performed optimization work to support heterogeneous batches of data inputs for 3D data processing. Users can import operators directly in PyTorch3D to speed up their experiment processes without re-creating or re-implementing the operators.

The Facebook team has produced four tutorials to help users to get started with PyTorch3D:

• Deform a sphere mesh to dolphin
• Bundle adjustment
• Render textured meshes
• Camera position optimization

Facebook says it hopes PyTorch3D can promote the integrated development of deep learning and 3D. In their future work, the team intends to expand the operators provided in PyTorch3D, and has invited the broader computer vision and deep learning research communities to contribute to this process.

For more information about PyTorch3D, please check out the project page.

---

Author: Herin Zhao | Editor: Michael Sarazen