In Short:
Researchers from MIT and Meta have developed PlatoNeRF, a computer vision technique that creates 3D models of scenes, including obstructed areas using shadows, which can help improve autonomous vehicles and AR/VR headsets. The technique combines multibounce lidar with neural radiance fields for accurate reconstructions. The researchers aim to track more bounces of light in the future for better scene reconstructions and possibly add deep learning techniques.
Imagine driving through a tunnel in an autonomous vehicle, but unbeknownst to you, a crash has stopped traffic up ahead. Normally, you’d need to rely on the car in front of you to know you should start braking. But what if your vehicle could see around the car ahead and apply the brakes even sooner?
New Computer Vision Technique for Autonomous Vehicles Developed by MIT and Meta
Researchers from MIT and Meta have developed a computer vision technique that could someday enable an autonomous vehicle to see around obstacles and apply brakes sooner in the event of a traffic delay.
PlatoNeRF: A Breakthrough Approach
They have introduced a method called PlatoNeRF that creates physically accurate, 3D models of an entire scene, including obstructed areas, using images from a single camera position. This technique uses shadows to determine what lies in blocked parts of the scene.
PlatoNeRF combines lidar technology with machine learning to generate more precise 3D reconstructions compared to existing AI methods. It excels at reconstructing scenes with difficult-to-see shadows, like those with high ambient light or dark backgrounds.
Potential Applications of PlatoNeRF
Besides enhancing the safety of autonomous vehicles, PlatoNeRF could improve AR/VR headsets’ efficiency and assist warehouse robots in navigating cluttered environments.
According to Tzofi Klinghoffer, an MIT graduate student and lead author of a paper on PlatoNeRF, the approach combines multibounce lidar and machine learning to unlock new possibilities.
Unique Methodology of PlatoNeRF
The MIT researchers leverage single-photon lidar to illuminate points in a scene, capturing shadows and hidden geometry. This innovative approach outperforms traditional methods, especially with lower-resolution lidar sensors.
“Our method combines multibounce lidar with neural radiance fields to achieve highly accurate scene reconstructions,” says Klinghoffer.
Future Research and Applications
The researchers plan to explore tracking more than two bounces of light for improved reconstructions. They also aim to integrate deep learning techniques and color image measurements into PlatoNeRF for enhanced texture information capture.
David Lindell, an assistant professor at the University of Toronto, commends the innovative use of ordinary sensors in PlatoNeRF, highlighting the significant advancements in reconstructed hidden geometry.
