HexPlane

Abstract

HexPlane is a fast and explicit representation for dynamic 3D scenes.

Modeling and re-rendering dynamic 3D scenes is a challenging task in 3D vision. Many current approaches, building on NeRF, rely on implicit representations in this task, which are relatively slow because of tremendous MLP evaluations, constraining real-world applications. Surprisingly, we show that dynamic 3D scenes could be explicitly represented by six feature planes, leading to an elegant solution called HexPlane. As an explicit representation, HexPlane computes features of spacetime points by fusing vectors extracted from each plane, which is effective and highly efficient. With a tiny MLP, it provides impressive results for dynamic novel view synthesis, matching the image quality of prior work but improving training time by more than 100x. We conduct extensive ablations to investigate this representation and reveal its intriguing properties. Designed as a general representation, we hope HexPlane can broadly contribute to spacetime tasks and dynamic 3D applications.

Approach

HexPlane is an elegant solution to explicitly represent dynamic 3D scenes, decomposing a 4D spacetime grid into six feature planes spanning each pair of coordinate axes (e.g., XY, ZT). It computes a feature vector for a 4D point in spacetime by projecting the point onto each feature plane. then aggregating the six resulting feature vectors. The fused feature vector is then passed to a tiny MLP which predicts the color of the point; novel views can then be rendered via volume rendering.

Synthesis Results for Dynamic Scenes

Here we show synthesis results using HexPlane as the representation in Plenoptic Video Dataset using both test view and virtual camera trajectories, which dataset contains high-resolution videos with challenging content and visual appearance.

Current MLP-based method requires over 1400 GPU hours of training for a single view, while our method finishes it within 10 hours with the same quality, which is over 100x accelerations.

Mesh of Dynamic Scenes

We extract a set of coarse meshes for dynamic scenes and colorize them by querying vertices' colors from the MLP. Please note that colors are not the same as those in rendered images since we don't do volumetric rendering here. Feel free to drag your mouse to see different views.

Time

BibTeX

@article{Cao2023HEXPLANE,
    author    = {Cao, Ang and Johnson, Justin},
    title     = {HexPlane: A Fast Representation for Dynamic Scenes},
    journal   = {CVPR},
    year      = {2023},
    }

There is another exciting work with similar idea released after HexPlane on arXiv. Please consider checking out that paper as well: https://sarafridov.github.io/K-Planes/

Acknowledgement

Toyota Research Institute provided funds to support this work but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity. We thank Shengyi Qian for the title suggestion, David Fouhey, Mohamed El Banani, Ziyang Chen, Linyi Jin and for helpful discussions and feedbacks.

HexPlane: A Fast Representation for Dynamic Scenes