ReFiNe: Recursive Field Networks for Cross-Modal Multi-Scene Representation (2024)

1. Introduction

Neural fields that encode scene properties at arbitrary resolutions using neural networks have reached unprecedented levels of detail. Typically using fully-connected multi-layer perceptrons (MLPs) to predict continuous field values, they have been used to represent geometry and appearance with applications in computer vision(Tancik etal., 2022), robotics(Rashid etal., 2023), and computer graphics(Mitra etal., 2019). However, most high-fidelity methods are limited to single scenes(Müller etal., 2022; Takikawa etal., 2021, 2022a) and overfit to the target geometry or appearance(Müller etal., 2022), while methods that capture multiple shapes typically sacrifice high-frequency details(Jang and Agapito, 2021; Park etal., 2019; Mescheder etal., 2019), limiting utility for applications such as streaming and representation learning. We would like to enable the compression of multiple complex shapes into single vectors with a single neural network and while maintaining the ability reconstruct high frequency geometric and textural information.

Global conditioning methods(sit, 2019; Jang and Agapito, 2021; Park etal., 2019) (i.e. one latent vector per shape) are capable of learning latent spaces over large numbers of shapes but require ground truth 3D supervision and suffer when representing high frequency details. Conversely, locally-conditioned methods partition the implicit function by leveraging hybrid discrete-continuous neural scene representations, effectively blurring the line between classical data structures and neural representations and allowing for more precise reconstructions by handling scenes as collections of localized primitives. These methods typically encode single scenes and leverage a secondary data structure(Takikawa etal., 2021; Zakharov etal., 2022; Müller etal., 2022), trading off additional memory for a less complex neural function mapping feature vectors to the target signal. Recently,(Zakharov etal., 2022) proposed to take advantage of both global and local conditioning via a recursive octree formulation, but the approach only captures geometry and outputs oriented point clouds that do not allow for continuous querying of the underlying implicit function, precluding the application of techniques such as ray-tracing.

In this work, we propose to encode many scenes represented as fields in a single network, where each scene is denoted by a single latent vector in a high dimensional space. We show how entire datasets of colored shapes can be encoded into a single neural network without sacrificing high frequency details (color or geometry) and without incurring a high memory cost. Key to our approach is a recursive formulation that allows us to effectively combine local and global conditioning. Our main motivation for a recursive structure comes from the observation that natural objects are self-similar(Shechtman and Irani, 2007), that is they are similar to a part of themselves at different scales. This property is famously used in the Fractal compression methods(Jacquin, 1990). Our method effectively extends prior work to the continuous setting, which allows us to recover geometry and color information with a higher degree of fidelity than previously possible. Our novel formulation allows us to learn from direct 3D supervision (SDF plus optionally RGB), as well as from continuous valued fields (NeRFs). We also investigate the properties of the resulting latent space and our results suggest the emergence of structure based on shape and appearance similarity. We address the limitations of related methods for representing multiple 3D shapes through ReFiNe: Recursive Field Networks and our contributions are:

2. Related Work

3. Methodology

We would like to learn to represent a set of objects $\mathcal{O}=\{O_{1},\dots,O_{K}\}$. In particular, we are interested in representing objects as fields, where each object is a mapping from a 3D coordinate in space to a value of dimension $F$, i.e., $O_{k}:\mathbb{R}^{3}\to\mathbb{R}^{F}$. Examples of common fields are Signed Distance Fields (where $F=1$ and the value of the field indicates the distance to the nearest surface) and radiance fields (where $F=4$, representing RGB and density values). For each object, we assume supervision in the form of $N_{k}$ coordinate and field value tuples of $\{\bm{x}_{j},f_{j}\}_{j=0}^{N_{k}}$, where $\bm{x}\in\mathbb{R}^{3}$ and $f\in\mathbb{R}^{F}$ is the field value.

3.1. ReFiNe

Our method represents each shape $O_{k}$ with a $D$-dimensional latent vector $\bm{z}^{0}$ that is recursively expanded into an octree with a maximum Level-of-Detail (LoD) $M$. Each level of the octree corresponds to a feature volume. We then perform both spatial and hierarchical feature aggregations before decoding into field values. Crucially, the expansion of each latent vector into an Octree-based neural field is achieved via the same simple MLP for each LoD and decoders are shared across all objects in $\mathcal{O}$. Once optimized, ReFiNe represents all $K$ objects in a set of $K$ latent vectors, a recursive autodecoder for octree expansion, an occupancy prediction network, and field-specific decoders (i.e., for RGB, SDF, etc). Figure1 illustrates how after training, our method can extract neural fields given different optimized LoD 0 latents, where we have dropped the superscipt for readability. Figure2 shows a more detailed overview of a reconstruction given a single input latent.

3.1.1. Recursive Subdivision & Pruning

Given a latent vector $\bm{z}^{m}\in\mathbb{R}^{D}$ from LoD $m$, our recursive autodecoder subdivision network $\phi:\mathbb{R}^{D}\to\mathbb{R}^{8D}$ traverses an octree by latent subdivision:

(1)

$$\phi(\bm{z}^{m})\to\{\bm{z}^{m+1}_{i}\}_{i=0}^{7}$$

Thus, a latent is divided into 8 cells, each with an associated child latent that is positioned at the cell’s center. Cell locations are defined by the Morton space-filling curve(Morton, 1966).

Each child latent is then further decoded to occupancy values $o$ using occupancy network $\omega:\mathbb{R}^{D}\to\mathbb{R}^{1}$. Rather than continuing to expand the tree for all child latents, $ReFiNe$ selects a subset based on the predicted occupancy value:

(2)

$$\mathcal{Z}^{m+1}=\{\bm{z}^{m+1}\in\phi(\bm{z}^{m})\mid\omega(\bm{z}^{m+1})>0.$$

where $\mathcal{Z}^{m+1}$ is the set of children latents from a particular parent latent $\bm{z}^{m}$ having predicted occupancies above a threshold of $0.5$, from which the next set of children will be recursed. This process can be seen in the left inset of Fig.2.To supervise occupancy predictions, we further assume access to the structure of the ground-truth octree during training, i.e., annotations of which voxels at each LoD are occupied. If a voxel is predicted to be more likely unoccupied during reconstruction, we prune it from the octree structure.

To build the set of latents at a particular LoD, the latent expansion process described by Equations 1 and 2 for a single latent is applied to all unpruned children latents from the previous LoD. In this way, $ReFiNe$ recursively expands a latent octree from a single root latent $\bm{z}^{0}$ to a set of latents at the desired LoD.

3.1.2. Multiscale Feature Fusion

Once an octree is constructed, it can be decoded to various outputs depending on the desired field parametrization. As mentioned, we use $\omega$ and decode each recursively extracted latent vector to occupancy. However, to model more complex signals with high-frequency details (e.g. SDF or RGB) we found that directly decoding latents positioned at voxel centers results in coarse approximations at low octree LoDs and is directly tied to the voxel size, presenting challenges in scaling to high resolutions and/or complex scenes.Instead, we approximate latents at sampled locations by performing trilinear interpolation given spatially surrounding latents at the same LoD. We repeat this at every LoD except the first and then fuse resulting intermediate latents as shown in Fig.2 into a new latent $\bm{\bar{z}}\in\mathbb{R}^{\bar{D}}$, where the dimension ${\bar{D}}$ of the fused latent varies based on whether a concatenation or summation scheme is used. In the summation scheme, the latent size remains unchanged, i.e., $\bar{D}=D$, whereas in the concatenation scheme, it is equal to the original latent size $D$ multiplied by the maximum LoD $M$.

3.1.3. Geometry Extraction and Rendering

Similar to (Zakharov etal., 2022), once the feature octree has been extracted for a given object we can decode the voxel centers into field values. However, our resulting representation can also be used to differentiably render images via volumetric rendering. We first estimate AABB intersections with voxels at the highest LoD. Given enter and exit points for each voxel, we then sample points within the voxel volume, enabling rendering via methods such as sphere ray tracing and volumetric compositing.

3.2. Field Specific Details

To demonstrate the utility and flexibility of ReFiNe, we focus in this work on two popular choices of object fields: Signed Distance Fields (SDF)(Park etal., 2019) for representing surfaces and Neural Radiance Fields (NeRF)(Mildenhall etal., 2020) for volumetric rendering and view synthesis. ReFiNe regresses field specific signals via neural mappings that map regressed latents, and optionally viewing direction to the desired output (e.g. SDF, SDF and RGB or density and RGB). We denote the neural mapping responsible for geometry as $\psi$ and the neural mapping responsible for appearance as $\xi$ and discuss specific instantiations below.

3.2.1. SDF

Each fused latent $\bm{\bar{z}}$ regressed via spatial interpolation over the octree and fused over multiple LoDs is given to network $\psi:\mathbb{R}^{\bar{D}}\to\mathbb{R}^{1}$ to estimate an SDF value $s$ corresponding to a distance to the closest surface, with positive and negative values representing exterior and interior areas respectively. When dealing with colored objects, we introduce network $\xi:\mathbb{R}^{\bar{D}}\to\mathbb{R}^{3}$ to estimate a 3D vector $\textbf{c}=(r,g,b)$ that represents RGB colors.

To quickly extract points on the surface of the object, we can simply decode $s$ for the coordinate of each occupied voxel at the highest LoD and calculate the normal of the point by taking the derivative w.r.t to the spatial coordinates. If more points are desired, we can additionally sample within occupied voxels to obtain more surface points. Given further computation time, we may also render the encoded scene via sphere ray tracing, i.e. at each step querying a SDF value within voxels that defines a sphere radius for the next step. We repeat the process until we reach the surface. The latents at the surface points are then used to estimate color values. Figures 3 and 4 show qualitative examples of iso-surface projection and sphere ray tracing, respectively.

3.2.2. NeRF

When representing neural radiance fields each fused multiscale feature is given to networks $\xi:\mathbb{R}^{{\bar{D}}+3}\to\mathbb{R}^{3}$ and $\psi:\mathbb{R}^{\bar{D}}\to\mathbb{R}^{1}$ to estimate a 4D vector $(\textbf{c},\sigma)$, where $\textbf{c}=(r,g,b)$ are RGB colors and $\sigma$ are densities per point. When trained on NeRF, our color network additionally takes a 3-channel view direction vector $d$ and the corresponding annotation $\mathcal{D}$ is augmented accordingly.

To render an image, each pixel value in the desired image frame is generated by compositing $K$ color predictions along the viewing ray via:

(3)

$$\small\hat{\textbf{c}}_{ij}=\sum_{k=1}^{K}w_{k}\hat{\textbf{c}}_{k},$$

where weights $w_{k}$ and accumulated densities $T_{k}$, provided intervals $\delta_{k}=t_{k+1}-t_{k}$, are defined as follows:

(4)		$\displaystyle\small w_{k}$	$\displaystyle=T_{k}\Big{(}1-\exp(-\sigma_{k}\delta_{k})\Big{)}$
(5)		$\displaystyle T_{k}$	$\displaystyle=\exp\Big{(}-\sum_{k^{\prime}=1}^{K}\sigma_{k^{\prime}}\delta_{k^$

and $\{t_{k}\}_{k=0}^{K-1}$ are sampled adaptive depth values. Example visualizations of NeRF-based volumetric rendering can be seen in Fig.5.

4. Experiments

To demonstrate the utility of our method, we perform experiments across a variety of datasets (Thingi32, ShapeNet150, SRN Cars, GSO, and RTMV) and field representations (SDF, SDF+RGB, and NeRF). We highlight that our method encodes entire datasets within a single neural network, and thus we aim to compare with baselines that focus on the same task and require the same kind of supervision, as opposed to methods that overfit to single shapes or scenes.

4.3. Reconstruction Benchmarks

4.3.1. Thingi32 / ShapeNet150 (SDF)

In the first benchmark we evaluate our method’s ability to represent and reconstruct object surfaces in the form of a SDF. We follow the experimental setup of(Takikawa etal., 2021; Zakharov etal., 2022) and train two networks: one on a subset of 32 objects from Thingi10K(Zhou and Jacobson, 2016) denoted Thingi32, and another on a subset of 150 objects from ShapeNet(Chang etal., 2015) denoted ShapeNet150. We use a latent dimension of 64 for Thingi32 and a latent dimension of 80 for ShapeNet150. We compute the commonly used Chamfer (CD), gIoU, and normal consistency (NC) metrics to evaluate surface reconstruction and we also record a memory footprint and inference time for each baseline. To extract a pointcloud from ReFiNe, we utilize the zero isosurface projection discussed in Section 3. Following ROAD’s(Zakharov etal., 2022) setup, gIoU is computed by recovering the object mesh using Poisson surface reconstruction(Kazhdan etal., 2006). We compare to DeepSDF(Park etal., 2019), Curriculum DeepSDF(Duan etal., 2020), using both methods’ open-sourced implementations for data generation and training with some minor hyper-parameter tuning to improve performance. Further details can be found in the supplemental material.

See Also

DF05M Price, Distributor, StockReFiNe: Recursive Field Networks for Cross-Modal Multi-Scene Representation

Method	View Synthesis			Runtime	Size
	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	s$\downarrow$	MB$\downarrow$
SRN	28.02	0.95	0.06	0.03	198
CodeNeRF	27.87	0.95	0.08	0.17	2.8
ReFiNe / LoD4	28.19	0.95	0.08	0.03	2.6
ReFiNe / LoD5	29.80	0.96	0.06	0.04	2.6
ReFiNe / LoD6	30.19	0.96	0.06	0.04	2.6

Our results are summarized in Table1 and we note that our method outperforms other SDF-based baselines with respect to Chamfer and gIoU, while having the smallest storage requirements. Figure3 qualitatively shows that DeepSDF(Park etal., 2019) and Curriculum DeepSDF(Duan etal., 2020) have difficulties reconstructing high frequency details. ROAD(Zakharov etal., 2022), on the other hand, can recover high frequency details but is discrete and outputs oriented point clouds with a fixed number of points at each level of detail. While ReFiNe has an analogous recursive backbone, it also performs multi-scale spatial feature interpolation, and instead models the object as a continuous field. ReFiNe outperforms ROAD on the ShapeNet150 dataset and performs on par on Thingi32 while only needing to traverse the octree to LoD6 as opposed to the expensive traversal to LoD9 for ROAD.Additionally, we compare the average surface extraction times for all baselines. Notably, both us and ROAD are significantly faster than DeepSDF and Curriculum DeepSDF. While ROAD demonstrates faster runtimes per the same LoD, it can’t sample values continuously limiting it to the extracted discrete cell centers. ReFiNe shows competitive extraction times with ROAD and already at LoD6 it outperforms ROAD’s LoD9 thanks to the ability to sample values continuously.

Method	View Synthesis			Runtime	Size
	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	s$\downarrow$	MB$\downarrow$
ReFiNe / Lat 32	24.18	0.83	0.23	1.19	8.4
ReFiNe / Lat 64	25.29	0.85	0.21	1.57	13.7
ReFiNe / Lat 128	25.96	0.86	0.20	2.34	24.3
ReFiNe / Lat 256	26.72	0.87	0.19	3.89	45.6

4.3.2. SRN Cars (NeRF)

In the next benchmark, we evaluate ReFiNe on another popular representation - Neural Radiance Fields (NeRFs). We use a feature dimension of 64 and compare our method against CodeNeRF(Jang and Agapito, 2021) and SRN(sit, 2019) on a subset of the SRN dataset consisting of 32 cars. We use 45 images for training and 5 non-overlapping images for testing on the task of novel view synthesis. As seen in Table2, our representation outperforms both SRN and CodeNeRF baselines. Fig.6 shows that ReFiNe does better when it comes to reconstructing high-frequency details.To compare inference time for NeRF-based baselines, we compute the average rendering time over the test images of the SRN benchmark. Our method demonstrates runtimes similar to those of SRN, with both significantly faster than CodeNeRF.

5. Limitations and Future Work

Our representation is currently limited to bounded scenes. This limitation can potentially be resolved by introducing an inverted sphere scene model for backgrounds from (Zhang etal., 2020). We would also like to leverage diffusion-based generative models to explore the task of 3D synthesis conditioned on various modalities such as text, images, and depth maps.

References

• (1)
• sit (2019)2019.Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations, In Sitzmann, Vincent and Zollhoefer, Michael and Wetzstein, Gordon.NeurIPS.
• Adamkiewicz etal. (2022)Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. 2022.Vision-only robot navigation in a neural radiance world.RA-L (2022).
• Barron etal. (2021)JonathanT Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and PratulP Srinivasan. 2021.Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV.
• Bi etal. (2020)Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020.Neural reflectance fields for appearance acquisition.arXiv (2020).
• Breyer etal. (2021)Michel Breyer, JenJen Chung, Lionel Ott, Roland Siegwart, and Juan Nieto. 2021.Volumetric grasping network: Real-time 6 dof grasp detection in clutter. In CoRL.
• Cai etal. (2020)Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. 2020.Learning gradient fields for shape generation. In ECCV.
• Chang etal. (2015)AngelX Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, etal. 2015.ShapeNet: An information-rich 3D model repository.arXiv (2015).
• Chen and Zhang (2019)Zhiqin Chen and Hao Zhang. 2019.Learning implicit fields for generative shape modeling. In CVPR.
• Davies etal. (2020)Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2020.Overfit neural networks as a compact shape representation.arXiv (2020).
• Deng etal. (2020)Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. 2020.Neural Articulated Shape Approximation. In ECCV.
• Deng etal. (2022)Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. 2022.Depth-supervised nerf: Fewer views and faster training for free. In CVPR.
• Deng etal. (2021)Yu Deng, Jiaolong Yang, and Xin Tong. 2021.Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In CVPR.
• Downs etal. (2022)Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, ThomasB McHugh, and Vincent Vanhoucke. 2022.Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items.arXiv (2022).
• Duan etal. (2020)Yueqi Duan, Haidong Zhu, He Wang, Li Yi, Ram Nevatia, and LeonidasJ Guibas. 2020.Curriculum deepsdf. In ECCV.
• FujiTsang etal. (2022)Clement FujiTsang, Maria Shugrina, JeanFrancois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, KrishnaMurthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. 2022.Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research.https://github.com/NVIDIAGameWorks/kaolin.
• Guizilini etal. (2022)Vitor Guizilini, Igor Vasiljevic, Jiading Fang, Rares Ambrus, Greg Shakhnarovich, MatthewR Walter, and Adrien Gaidon. 2022.Depth field networks for generalizable multi-view scene representation. In ECCV.
• Hodan etal. (2018)Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, etal. 2018.Bop: Benchmark for 6d object pose estimation. In ECCV.
• Ichnowski etal. (2022)Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg. 2022.Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects. In CoRL.
• Irshad etal. (2022)MuhammadZubair Irshad, Sergey Zakharov, Rares Ambrus, Thomas Kollar, Zsolt Kira, and Adrien Gaidon. 2022.ShAPO: Implicit Representations for Multi-Object Shape Appearance and Pose Optimization. In ECCV.
• Jacquin (1990)ArnaudE Jacquin. 1990.Fractal image coding based on a theory of iterated contractive image transformations. In VCIP.
• Jang and Agapito (2021)Wonbong Jang and Lourdes Agapito. 2021.Codenerf: Disentangled neural radiance fields for object categories. In ICCV.
• Kaskman etal. (2019)Roman Kaskman, Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. 2019.Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects. In ICCV Workshops.
• Kato etal. (2020)Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. 2020.Differentiable rendering: A survey.arXiv (2020).
• Kazhdan etal. (2006)Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006.Poisson surface reconstruction. In SGP.
• Kim etal. (2024)Doyub Kim, Minjae Lee, and Ken Museth. 2024.Neuralvdb: High-resolution sparse volume representation using hierarchical neural networks.TOG (2024).
• Kingma and Ba (2014)DiederikP Kingma and Jimmy Ba. 2014.Adam: A method for stochastic optimization.arXiv (2014).
• Lindell etal. (2021)DavidB. Lindell, JulienN.P. Martel, and Gordon Wetzstein. 2021.AutoInt: Automatic Integration for Fast Neural Volume Rendering. In CVPR.
• Liu etal. (2020a)Lingjie Liu, Jiatao Gu, KyawZaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020a.Neural Sparse Voxel Fields. In NeurIPS.
• Liu etal. (2020b)Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. 2020b.Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR.
• Locatello etal. (2020)Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020.Object-centric learning with slot attention. In NeurIPS.
• Lombardi etal. (2019)Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019.Neural volumes: learning dynamic renderable volumes from images.TOG (2019).
• Martin-Brualla etal. (2021)Ricardo Martin-Brualla, Noha Radwan, Mehdi S.M. Sajjadi, JonathanT. Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021.NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR.
• Mescheder etal. (2019)Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019.Occupancy networks: Learning 3d reconstruction in function space. In CVPR.
• Mildenhall etal. (2020)Ben Mildenhall, PratulP Srinivasan, Matthew Tancik, JonathanT Barron, Ravi Ramamoorthi, and Ren Ng. 2020.Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV.
• Mitra etal. (2019)NiloyJ. Mitra, Iasonas Kokkinos, Paul Guerrero, Nils Thuerey, Vladimir Kim, and Leonidas Guibas. 2019.CreativeAI: Deep Learning for Graphics. In SIGGRAPH 2019 Courses.
• Morton (1966)GuyM Morton. 1966.A computer oriented geodetic data base and a new technique in file sequencing.(1966).
• Mu etal. (2021)Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. 2021.A-sdf: Learning disentangled signed distance functions for articulated shape representation. In ICCV.
• Müller etal. (2022)Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.TOG (2022).
• Neff etal. (2021)Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, JoergH Mueller, Chakravarty RAlla Chaitanya, Anton Kaplanyan, and Markus Steinberger. 2021.DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. In Computer Graphics Forum.
• Niemeyer and Geiger (2021)Michael Niemeyer and Andreas Geiger. 2021.Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR.
• Niemeyer etal. (2020)Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020.Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR.
• Ortiz etal. (2022)Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. 2022.iSDF: Real-Time Neural Signed Distance Fields for Robot Perception. In RSS.
• Ost etal. (2021)Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. 2021.Neural scene graphs for dynamic scenes. In CVPR.
• Palafox etal. (2021)Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, and Angela Dai. 2021.Npms: Neural parametric models for 3d deformable shapes. In ICCV.
• Park etal. (2019)JeongJoon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019.DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR.
• Park etal. (2021)Keunhong Park, Utkarsh Sinha, JonathanT. Barron, Sofien Bouaziz, DanB Goldman, StevenM. Seitz, and Ricardo Martin-Brualla. 2021.Nerfies: Deformable Neural Radiance Fields. In ICCV.
• Peng etal. (2020)Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020.Convolutional occupancy networks. In ECCV.
• Pumarola etal. (2021)Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021.D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR.
• Rashid etal. (2023)Adam Rashid, Satvik Sharma, ChungMin Kim, Justin Kerr, LawrenceYunliang Chen, Angjoo Kanazawa, and Ken Goldberg. 2023.Language embedded radiance fields for zero-shot task-oriented grasping. In CoRL.
• Rebain etal. (2021)Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, KwangMoo Yi, and Andrea Tagliasacchi. 2021.Derf: Decomposed radiance fields. In CVPR.
• Sajjadi etal. (2022a)MehdiSM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd VanSteenkiste, Filip Pavetic, Mario Lucic, LeonidasJ Guibas, Klaus Greff, and Thomas Kipf. 2022a.Object scene representation transformer. In NeurIPS.
• Sajjadi etal. (2022b)MehdiSM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, etal. 2022b.Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In CVPR.
• Shechtman and Irani (2007)Eli Shechtman and Michal Irani. 2007.Matching local self-similarities across images and videos. In CVPR.
• Sitzmann etal. (2020a)Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. 2020a.Metasdf: Meta-learning signed distance functions. In NeurIPS.
• Sitzmann etal. (2020b)Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020b.Implicit neural representations with periodic activation functions. In NeurIPS.
• Srinivasan etal. (2021)PratulP. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and JonathanT. Barron. 2021.NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In CVPR.
• Stelzner etal. (2021)Karl Stelzner, Kristian Kersting, and AdamR Kosiorek. 2021.Decomposing 3d scenes into objects via unsupervised volume segmentation.arXiv (2021).
• Sucar etal. (2021)Edgar Sucar, Shikun Liu, Joseph Ortiz, and AndrewJ Davison. 2021.iMAP: Implicit mapping and positioning in real-time. In ICCV.
• Takikawa etal. (2022a)Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. 2022a.Variable bitrate neural fields. In SIGGRAPH.
• Takikawa etal. (2021)Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2021.Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In CVPR.
• Takikawa etal. (2022b)Towaki Takikawa, Or Perel, ClementFuji Tsang, Charles Loop, Joey Litalien, Jonathan Tremblay, Sanja Fidler, and Maria Shugrina. 2022b.Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research.https://github.com/NVIDIAGameWorks/kaolin-wisp.
• Tancik etal. (2022)Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, PratulP Srinivasan, JonathanT Barron, and Henrik Kretzschmar. 2022.Block-nerf: Scalable large scene neural view synthesis. In CVPR.
• Tancik etal. (2021)Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, PratulP Srinivasan, JonathanT Barron, and Ren Ng. 2021.Learned initializations for optimizing coordinate-based neural representations. In CVPR.
• Tang etal. (2021)Jia-Heng Tang, Weikai Chen, Jie Yang, Bo Wang, Songrun Liu, Bo Yang, and Lin Gao. 2021.OctField: Hierarchical Implicit Functions for 3D Modeling. In NeurIPS.
• Tewari etal. (2021)Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, etal. 2021.Advances in neural rendering.arXiv (2021).
• Tremblay etal. (2022)Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, and Stan Birchfield. 2022.RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis.ECCV Workshops.
• Wang etal. (2022)Yifan Wang, Lukas Rahmann, and Olga Sorkine-Hornung. 2022.Geometry-consistent neural shape representation with implicit displacement fields. In ICLR.
• Wei etal. (2021)Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. 2021.Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV.
• Williams etal. (2022)Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. 2022.Neural fields as learnable kernels for 3d reconstruction. In CVPR.
• Xian etal. (2021)Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. 2021.Space-time neural irradiance fields for free-viewpoint video. In CVPR.
• Xie etal. (2021)Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021.Neural Fields in Visual Computing and Beyond.arXiv (2021).
• Yang etal. (2019)Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019.Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV.
• Yang etal. (2022)Guo-Wei Yang, Wen-Yang Zhou, Hao-Yang Peng, Dun Liang, Tai-Jiang Mu, and Shi-Min Hu. 2022.Recursive-nerf: An efficient and dynamically growing nerf.TVCG (2022).
• Yi etal. (2023)Brent Yi, Weijia Zeng, Sam Buchanan, and Yi Ma. 2023.Canonical factors for hybrid neural fields. In ICCV.
• Yu etal. (2021)Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021.pixelnerf: Neural radiance fields from one or few images. In CVPR.
• Yu etal. (2022)Hong-Xing Yu, LeonidasJ. Guibas, and Jiajun Wu. 2022.Unsupervised Discovery of Object Radiance Fields. In ICLR.
• Yuan etal. (2021)Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. 2021.STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering. In CVPR.
• Zakharov etal. (2022)Sergey Zakharov, Rares Ambrus, Katherine Liu, and Adrien Gaidon. 2022.ROAD: Learning an Implicit Recursive Octree Auto-Decoder to Efficiently Encode 3D Shapes. In CoRL.
• Zakharov etal. (2021)Sergey Zakharov, RaresAndrei Ambrus, VitorCampagnolo Guizilini, Dennis Park, Wadim Kehl, Fredo Durand, JoshuaB Tenenbaum, Vincent Sitzmann, Jiajun Wu, and Adrien Gaidon. 2021.Single-Shot Scene Reconstruction. In CoRL.
• Zakharov etal. (2020)Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. 2020.Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors. In CVPR.
• Zeng etal. (2022)Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. 2022.LION: Latent Point Diffusion Models for 3D Shape Generation. In NeurIPS.
• Zhang etal. (2020)Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020.Nerf++: Analyzing and improving neural radiance fields.arXiv (2020).
• Zhou etal. (2021)Linqi Zhou, Yilun Du, and Jiajun Wu. 2021.3d shape generation and completion through point-voxel diffusion. In ICCV.
• Zhou and Jacobson (2016)Qingnan Zhou and Alec Jacobson. 2016.Thingi10k: A dataset of 10,000 3d-printing models.arXiv (2016).

Appendix A Training Details

A.2. Baseline Method Details

DeepSDF

We use the open-source implementation of DeepSDF (Park etal., 2019). To generate training data, we preprocess models from Thingi32 and ShapeNet150 via the provided code and parameters, which aims to generate approximately 500k training points. We improve results on the overfitting scenario by setting the dropout rate to zero and removing the latent regularization. We use a learning rate of $0.001$ for the decoder network parameters and $0.002$ for latents as well as a decay factor of $0.75$ every 500 steps, training the methods until convergence (about 20k epochs). For the experiments on Thingi32 we use a batch size of 32 objects and for ShapeNet150 we use a batch size of 64 objects. All other parameters we leave as provided by the example implementations (i.e., we used a code length of 256 and keep the neural network architecture unchanged).

Curriculum DeepSDF

We also use the open-source implementation of Curriculum-DeepSDF (Duan etal., 2020). We duplicate the parameter changes made to DeepSDF for consistency, and use the same training data input. We do not modify the curriculum proposed in (Duan etal., 2020) other than lengthening the last stage of training. We observe that the proposed curriculum provided quantitative reconstruction gains for ShapeNet150 and not Thingi32, suggesting that a different curriculum may improve results for the latter dataset. However, searching for the optimal curriculum is expensive and we choose to report results based on the baseline curriculum given in the open-source implementation.

SRN & CodeNeRF

We use the open-source implementations with default configurations for SRN(sit, 2019) and CodeNeRF(Jang and Agapito, 2021) and train both methods on our subset of the SRN dataset as described in Section 4.3 of the main paper. Both baselines use a default latent code size of 256, whereas CodeNeRF uses 2 latent codes of 256 to represent an object - one for geometry, another for appearance. In Table 2 and Fig. 6 of the main paper we demonstrate that our method outperforms both baselines, while using a more lightweight architecture and a latent code size of 64.

Fusion	Reconstruction		Runtime	Size
	CD$\downarrow$	3D PSNR$\uparrow$	s$\downarrow$	MB$\downarrow$
Sum	0.046	33.61	0.11	3.2
Concatenate	0.046	34.89	0.12	3.8

Appendix B Evaluation Details

To calculate the Chamfer distance for DeepSDF and Curriculum DeepSDF we first extract surface points following the protocol of(Irshad etal., 2022). In particular, we define a coarse voxel grid of LoD 2 and estimate SDF values for eachof the points using a pretrained SDF network. The voxels whose SDF values are larger than their size are pruned and the remaining voxels are propagated to the next level via subdivision. When the desired LoD is reached, we use zero isosurface projection to extract surface points using predicted SDF values and estimated surface normals. Finally, we use the Chamfer distance implementation from(Takikawa etal., 2022b) to compare our prediction against a ground truth point cloud of $2^{17}$ points sampled from the original mesh. When reconstructing SDF + Color, we additionally use PSNR to evaluate RGB values regressed from the same $2^{17}$ points.To compute gIoU, we first reconstruct a mesh using Poisson surface reconstruction from(Kazhdan etal., 2006) and then compare against $2^{17}$ ground truth values randomly computed using the original mesh.

Appendix C Additional Results

Latent Space Interpolation and Clustering

We present a qualitative analysis of our latent space conducted on the ShapeNet150 and SRN Cars datasets. As our method outputs a continuous feature field, it can be used for interpolation in the latent space between objects of similar geometry. Figure10 shows an example of such interpolation between two objects of different classes.In addition, we plot latent spaces of Thingi32, ShapeNet150, and Google Scanned Objects represented by respective networks using the principal component analysis (see Fig.8). Projected latent spaces suggest that the structure of ReFiNe’s latent space clusters similar objects defined either by geometry (Thingi32, ShapeNet150) or geometry and color (Google Scanned Objects), pointing to potential classification utility.

Network Size vs Reconstruction Quality

Similar to our latent size experiments in Table 3 of the main paper, in this ablation we study how changing the hidden dimension of our recursive subdivision network $\phi$ affects reconstruction quality. We train four baselines with different sizes for the hidden dimension of the recursive subdivision network, $\phi$: 128, 256, 512, and 1024. The remaining parameters are consistent across all four networks: a latent size of 64, and each of the decoding networks $\omega$, $\psi$, and $\xi$ utilizes two layers of 256 fully connected units. All the baselines are trained on the HB dataset (SDF+RGB). As shown in Fig.11, we observe a graceful degradation of quality with decreasing network capacity.

Appendix D Applications

In recent years neural fields have found its use in various domains including robotics and graphics. In recent years, neural fields have found utility in various domains, including robotics and graphics. In robotics, neural fields are actively employed to represent 3D geometry and appearance, with applications in object pose estimation and refinement(Zakharov etal., 2020; Irshad etal., 2022), grasping(Breyer etal., 2021; Ichnowski etal., 2022), and trajectory planning(Adamkiewicz etal., 2022). In graphics, they have been successfully utilized for object reconstruction from sparse and noisy data(Williams etal., 2022) and for representing high-quality 3D assets(Takikawa etal., 2022a).

ReFiNe employs a recursive hierarchical formulation that leverages object self-similarity, resulting in a highly compressed and efficient shape latent space. We demonstrate that our method achieves impressive results in SDF-based reconstruction (Table 1 of the main paper) and NeRF-based novel-view synthesis (Tables 2 and 3 of the main paper), and features well-clustered latent spaces allowing for smooth interpolation (Figs.8 and 10). We believe that these properties will accelerate the applicability of neural fields in real-world tasks, particularly those involving compression.