In the field of computer science, accurately reconstructing 3D models from 2D images—a problem known as pose inference—presents complex challenges. For instance, the task can be vital in producing 3D models for e-commerce or assisting in autonomous vehicle navigation. Existing methods rely on gathering the camera poses prior, or harnessing generative adversarial networks (GANs), but neither technique provides a satisfactory solution in terms of accuracy and efficiency.
Addressing this problem, researchers from Google and Stanford University have created MELON, a system designed to reconstruct 3D objects from 2D images without knowledge of the camera poses. MELON adopts a simpler but more effective approach than previous solutions, such as BARF and SAMURAI, which utilized initial pose estimates or complicated training techniques involving GANs.
The MELON solution combines two key techniques. Firstly, it employs a dynamically trained lightweight CNN encoder to derive camera poses from training images, eliminating the need for pre-training. This process regularizes optimization by matching similar-looking images with similar poses. Secondly, MELON introduces a modulo loss, accounting for an object’s pseudo symmetries. Here, an object is rendered from a fixed set of viewpoints, and the loss is backpropagated through the view that most accurately matches each training image. This eliminates the need for complex training or pre-training on labelled data, focusing instead on simplicity and effectiveness. MELON tackles the task directly, integrating these techniques into the standard NeRF training process.
To test its capabilities, MELON was evaluated on the NeRF Synthetic dataset. The results revealed an impressive ability to swiftly and accurately determine poses and generate high-fidelity novel views, even from extremely noisy, unposed images.
In conclusion, the MELON technique shows promise in resolving the challenges of 3D reconstruction from 2D images without knowing camera poses, pushing the envelope for both robustness and simplicity. Its methodology involving lightweight CNN encoders and the application of a modulo loss fully addresses a task that previously necessitated cumbersome training schemes or initial pose approximations.
Final credit goes to the researchers developing this project at Google and Stanford University. For further information, their research can be found in the linked paper.
