Real-Time View Synthesis using Deep Learning

Ghosh, Sushobhan

doi:https://doi.org/10.21985/n2-6dkk-pa71

Work

Real-Time View Synthesis using Deep Learning

Public

Download PDF

Download All Files (.zip)

Recent developments in deep learning have led to breakthroughs in rendering novel views from sparse input views of a scene.While the accuracy of these algorithms has improved dramatically, it has come at a huge computational cost. While developments in graphics hardware have ameliorated some of the computational burdens, deep learning-based algorithms are far from being used for real-time view synthesis on commodity hardware and further from being used in head-mounted displays for AR/VR applications. We analyze different deep learning methods employed for view synthesis and test their performance in terms of accuracy as well as runtime. We identify the accuracy as well as speed bottlenecks in different approaches and propose various solutions to solve them. Specifically, we show that depth/disparity warping-based deep learning methods are fundamentally limited in the accuracy they can achieve when the input views have disparities larger than a few pixels.We propose an alternative baseline reduction approach to improve the reconstruction from such methods. We also show that current methods to generate MPIs from sparse input views is computationally inefficient and offers very little flexibility in adapting to change in scene parameters such as depth range and object density. We propose an MPI generation method that is flexible with the number and position of planes post-training and hence offers the ability to adapt to scenes with different scene parameters. We show the advantage of such flexibility on the task of live view synthesis and video view synthesis. Finally, we also present some preliminary work on neural light field representation as an alternative to neural radiance fields. We show that neural light fields potentially offer three orders of magnitude improvement in speed while offering comparable accuracy to NeRF representations. Overall, the key contribution of this thesis is the study and development of view synthesis algorithms with a focus on the runtime efficiency of such algorithms.

Creator