FWD: Real-time Novel View Synthesis with Forward Warping and Depth

Novel view synthesis (NVS) is a challenging task requir- ing systems to generate photorealistic images of scenes from new viewpoints, where both quality and speed are important for applications. Previous image-based rendering (IBR) methods are fast, but have poor quality when input views are sparse. Recent Neural Radiance Fields (NeRF) and generalizable variants give impressive results but are not real-time. In our paper, we propose a generalizable NVS method with sparse inputs, called FWD , which gives high- quality synthesis in real-time. With explicit depth and dif- ferentiable rendering, it achieves competitive results to the SOTA methods with 130-1000× speedup and better percep- tual quality. If available, we can seamlessly integrate sen- sor depth during either training or inference to improve im- age quality while retaining real-time speed. With the grow- ing prevalence of depths sensors, we hope that methods making use of depth will become increasingly useful.