NeX: Real-time View Synthesis with Neural Basis Expansion

Abstract

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce NeXt-level view-dependent effects---in real time. Unlike traditional MPI that uses a set of simple RGBα planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as the rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art.

Rendering speed

This table shows the average FPS for the scenes in Shiny and Real Forward-Facing datasets. The rendering was done in real time using Google Chrome’s WebGL on a machine with an NVIDIA RTX3090. (Profiling done without Vertical Synchronization (Vsync) to go beyond the monitor’s frame rate)