Researchers from MIT and Meta have developed a computational vision technique, named PlatoNeRF, that allows for creating vivid, accurate 3D models of a scene from a single camera view. The innovative technology uses the shadowing in a scene to determine what could lie within obstructed areas. By combining machine learning with LIDAR (Light Detection and Ranging) technology, PlatoNeRF generates more accurate 3D geometries than similar existing technologies. The software could be used to improve the functionality of autonomous vehicles by allowing them to see and react to obstacles ahead of them regardless of their visibility. This technique contributes significantly to the safety of autonomous vehicles.
PlatoNeRF, aptly named after Plato’s allegory of the cave, uses multibounce LIDAR and deep learning to produce its detailed 3D models, capturing information about secondary rays of light creating shadows, and adding depth to its 3D models. The method combines these technologies to determine the points that reside within shadow due to the absence of light, effectively allowing it to infer the geometries of hidden objects.
The superb fidelity and interpolation potential of the Neural Radiance Field (NeRF) are also incorporated into PlatoNeRF. The NeRF ability to accurately estimate scenescapes, when combined with multibounce LIDAR, could result in highly accurate reconstructions.
PlatoNeRF matches more than the combination of multibounce lidar and machine learning technologies, demonstrating its capability by outpacing alternative techniques specifically in scenarios when the lidar sensor resolution was subpar – a common occurrence in modern commercial devices.
The innovation of PlatoNeRF extends beyond autonomous vehicles. The technology could also enhance AR/VR headsets, allowing users to model a room’s geometry without physically walking around to take measurements. Warehouse robots could also employ the technology to locate items in cluttered environments more quickly. The researchers aspire to refine the technique by tracking beyond two light bounces for potential improvements in scene reconstructions. They are also interested in leveraging more deep learning methods and incorporating PlatoNeRF with colour image measurements for more comprehensive texture data.
David Lindell, an assistant professor in the Department of Computer Science at the University of Toronto, shared his admiration for the research, highlighting how the marriage of smart algorithms and conventional sensors can lead to extraordinary capabilities.
