Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and Manipulation

4.1 Compact Wavelet Representation

Preparing a compact wavelet representation of a given 3D shape (see Figure 2(a)) involves the following two steps: (i) implicitly represent the shape using a signed distance field (SDF); and (ii) decompose the implicit representation via wavelet transform into coefficient volumes, each encoding a specific scale of the shape.

In the first step, we scale each shape to fit \([-0.9,+0.9]^3\) and sample an SDF of resolution \(256^3\) to implicitly represent the shape. Importantly, we truncate the distance p in the SDF to \([-0.1,+0.1]\), so regions not close to the object surface are clipped to a constant. We denote the truncated signed distance field (TSDF) for the i-th shape in training set as \(S_i\). Using \(S_i\), we can significantly reduce the shape representation redundancy, such that the shape learning process can focus better on the shape structures and fine details.

The second step is a multi-scale wavelet decomposition [Mallat 1989; Daubechies 1990; Velho et al. 1994] on the TSDF. Here, we decompose \(S_i\) into a high-frequency detail coefficient volume and a low-frequency coarse coefficient volume, which is roughly a compressed version of \(S_i\). We repeat this process J times on the coarse coefficient volume of each scale, decomposing \(S_i\) into a series of multi-level coefficient volumes. We denote the coarse and detail coefficient volumes at the j-th step (scale) as \(C^j_i\) and \(D^j_i\), respectively, where \(j = \lbrace 1,\ldots ,J\rbrace\). The representation is lossless, meaning that the extracted coefficient volumes together can faithfully reconstruct the original TSDF via a series of inverse wavelet transforms.

There are three important considerations in the data preparation. First, multi-scale decomposition can effectively separate rich structures, fine details, and noise in the TSDF. Empirically, we evaluate the reconstruction error on the TSDF by masking out all higher-scale detail coefficients and reconstructing \(S_i\) only from the coefficients at scale \(J=3\), i.e., \(C^3_i\) and \(D^3_i\). We found that the reconstructed TSDF values have relatively small changes from the originals (only 2.8% in magnitude), even without 97% of the coefficients for the Chair category in ShapeNet [Chang et al. 2015]. Comparing Figures 3(a) vs. (b), we can see that reconstructing only from the coarse scale of \(J=3\) already well retains the chair’s structure. Motivated by this observation, we propose to construct the compact wavelet representation at a coarse scale (\(J=3\)) and drop other detail coefficient volumes, i.e., \(D^1_i\) and \(D^2_i\), for efficient shape learning. See supplementary material Section I for more details on the wavelet decomposition.

Fig. 3.

View Figure

Second, we need a suitable wavelet filter. While Haar wavelet is a popular choice due to its simplicity, using it to encode smooth and continuous signals such as the SDF may introduce serious voxelization artifacts, see, e.g., Figure 3(d). In this work, we propose to adopt the biorthogonal wavelets [Cohen 1992], since it enables a more smooth decomposition of the TSDF. Specifically, we tried different settings in the biorthogonal wavelets and chose to use high vanishing moments: six for the synthesis filter and eight for the analysis filter; see Figures 3(b) vs. (c). Also, instead of storing the detail coefficient volumes with seven channels, as in traditional wavelet decomposition, we follow [Velho et al. 1994] to efficiently compute it as the difference between the inverse transformed \(C^{j}_i\) and \(C^{j-1}_i\) in a Laplacian pyramid style. Hence, the detail coefficient volume has a higher resolution than the coarser one, but both have much lower resolution than the original TSDF volume (\(256^3\)).

In this work, we adopt ShapeNet [Chang et al. 2015] to prepare the wavelet representation. Note that the shapes in this dataset are canonically aligned, thus potentially simplifying the learning process. Also, it is important to truncate the SDF before constructing the wavelet representation for shape learning. By truncating the SDF, regions not close to the shape surface would be cast to a constant function to make efficient the wavelet decomposition and shape learning. Otherwise, we found that the shape learning process will collapse and the training loss cannot be reduced.

4.2 Shape Learning

Next, to learn the 3D shape distribution in a given shape set, we gather coefficient volumes \(\lbrace C_i^J , D_i^J\rbrace\) of the shapes in the set for training (i) the generator network to learn to iteratively remove noise from a random Gaussian noise sample to generate \(C_i^J\); and (ii) the detail predictor network to learn to predict \(D_i^J\) from \(C_i^J\) to enhance the details in the generated shapes. Further, to enable shape inversion, we may additionally train (iii) the encoder network (jointly optimized with the generator network) to learn a latent space for mapping the coarse coefficient volume \(C_i^J\) to a latent code \(z_{i}\).

Network structure. To start, we formulate a simple but efficient neural network structure for both the generator and detail predictor networks. The two networks have the same structure, as both take a 3D volume as input and output a 3D volume of the same resolution as the input. Specifically, we adopt a modified 3D version of the U-Net architecture [Nichol and Dhariwal 2021]. First, we apply three 3D residual blocks to progressively compose and downsample the input into a set of multi-scale features and a bottleneck feature volume. Then, we apply a single self-attention layer to aggregate features in the bottleneck volume, so that we can efficiently incorporate non-local information into the features. Further, we upsample and concatenate features in the same scale and progressively perform an inverse convolution with three residual blocks to produce the output. For all convolution layers in the network structure, we use a filter size of three with a stride of one.

For the encoder network, we design a five-layer 3D convolutional neural network with kernel size \(k=4\) and stride \(s=1\), each followed by an instance normalization. Also, we adopt a single linear transform to produce the output latent code z.

In the following, we will first introduce the modeling of the shape generation process, followed by the adaptation for the shape inversion process. Lastly, we will introduce the detail predictor network for enhancing the details of the generated shapes.

Modeling the shape generation process. We formulate the 3D shape generation process based on the denoising diffusion probabilistic model [Sohl-Dickstein et al. 2015]. For simplicity, we drop the subscript and superscript in \(C_i^J\) , and denote \(\lbrace C_{0}, \ldots , C_{T} \rbrace\) as the shape generation sequence, where \(C_0\) is the target, which is \(C_i^J\); \(C_T\) is a random noise volume from the Gaussian prior; and T is the total number of time steps. As shown on top of Figure 2(b), we have (i) a forward process (denoted as \(q(C_{0:T})\)) that progressively adds noise based on a Gaussian distribution to corrupt \(C_0\) into a random noise volume; and (ii) a backward process (denoted as \(p_{\theta }(C_{0:T})\)) that employs the generator network (with network parameter \(\theta\)) to iteratively remove noise from \(C_T\) to generate the target. Note that all 3D shapes \(\lbrace C_{0}, \ldots , C_{T} \rbrace\) are represented as 3D volumes and each voxel value is a wavelet coefficient at its spatial location.

Both the forward and backward processes are modeled as the Markov processes. The generator network is optimized to maximize the generation probability of the target, i.e., \(p_{\theta }(C_0)\). Also, as suggested in Ho et al. [2020], this training procedure can be further simplified to use the generator network to predict the noise volume \(\epsilon _{\theta }\). So, we adopt a mean-squares loss to train our framework: (1) \(\begin{equation} L_2 = E_{t,C_0,\epsilon }[{\parallel } \epsilon - \epsilon _{\theta }(C_t, t) {\parallel }^2], \epsilon \sim \mathcal {N}(0,\mathbf {I}), \end{equation}\) where t is the time steps; \(\epsilon\) is a noise volume; and \(\mathcal {N}(0,\mathbf {I})\) denotes a unit Gaussian distribution. In particular, we first sample noise volume \(\epsilon\) from a unit Gaussian distribution \(\mathcal {N}(0,\mathbf {I})\) and time step \(t \in [1,\ldots ,T]\) to corrupt \(C_0\) into \(C_t\). Then, our generator network learns to predict noise \(\epsilon\) based on the corrupted coefficient volume \(C_t\). Further, as the network takes time step t as input, we convert value t into an embedding via two MLP layers. Using this embedding, we can condition all the convolution modules in the prediction and enable the generator to be more aware of the amount of noise contaminated in \(C_t\). For more details on the derivation of the training objectives, please refer to supplementary material Section J.

Modeling the shape inversion process. The shape inversion process aims to embed a given 3D shape as a compact code z, such that we can faithfully reconstruct the coarse wavelet volume \(C_0\) of the shape from z and further reconstruct the shape. Motivated by Preechakul et al. [2022], we introduce an optional encoder \(Enc_{\phi }\) to derive the latent code z of coarse wavelet volume \(C_0\), i.e., \(z = Enc_{\phi }(C_0)\). Then, we inject z into the generator during the backward process as an additional shape condition; see the bottom part of Figure 2(b). During the training, we jointly optimize the encoder and generator networks to maximize the conditional generation probability, i.e., \(p_\theta (C_0|z)\). This can be achieved by modifying the objective in Equation (1) into (2) \(\begin{equation} L_2 = E_{t,C_0,\epsilon }[{\parallel } \epsilon - \epsilon _{\theta }(C_t, z, t) {\parallel }^2], \epsilon \sim \mathcal {N}(0,\mathbf {I}). \end{equation}\) Note that since we need the generator to be aware of the current shape condition, we follow [Preechakul et al. 2022] to use the group normalization layers to jointly take the time embedding and the latent code as the input to the generator.

Detail predictor network. Next, we train the detail predictor network to produce the detail coefficient volume \(D_0\) from coarse coefficient volume \(C_0\) (see the top part of Figure 2(b)), so that we can further enhance the details in our generated (or inverted) shapes.

To train the detail predictor network, we leverage the paired coefficient volumes \(\lbrace C_i^J, D_i^J \rbrace\) from the data preparation. Importantly, each detail coefficient volume \(D_0\) should be highly correlated to its associated coarse coefficient volume \(C_0\). Hence, we pose detail prediction as a conditional regression on the detail coefficient volume, aiming at learning neural network function \(f: C_0 \rightarrow D_0\); hence, we optimize f via a mean squared error loss. Overall, the detail predictor has the same network structure as the generator, but we include more convolution layers to accommodate the cubic growth in the number of nonzero values in the detail coefficient volume.

4.3 Shape Generation

Now, we are ready to generate 3D shapes. Figure 2(c) illustrates the shape generation procedure. First, we randomize a 3D noise volume as \(C_T\) from the standard Gaussian distribution. Then, we can employ the trained generator for T time steps to produce \(C_0\) from \(C_T\). This process is iterative and inter-dependent. We cannot parallelize the operations in different time steps, so leading to a very long computing time. To speed up the inference process, we adopt an approach in Song et al. [2020] to sub-sample a set of time steps from \([1,\ldots , T]\) during the inference; in practice, we evenly sample \(1/10\) of the total time steps in all our experiments.

After we obtain the coarse coefficient volume \(C_0\), we can then use the detail predictor network to predict the detail coefficient volume \(D_0\) from \(C_0\), followed by a series of inverse wavelet transforms from \(\lbrace C_0 , D_0 \rbrace\) at scale J=3 to reconstruct the original TSDF. By then, we can extract an explicit 3D mesh from the reconstructed TSDF using the marching cube algorithm [Lorensen and Cline 1987].

4.4 Shape Inversion

Thanks to the encoder network trained during the shape learning, our approach can leverage it for shape inversion. Having said that, our goal is to invert a given unseen shape into a latent code and then reconstruct it faithfully from the latent code.

Figure 2(d) illustrates the overall shape inversion procedure. First, we produce coarse coefficient volume \(C_0\) from the input shape, following the procedure in Section 4.1. Then, we feed coefficient volume \(C_0\) into the encoder network to obtain the latent code \(z=Enc_{\phi }(C_0)\). After that, we feed latent code z together with a sampled noise volume \(\epsilon\) into the generator network to directly produce a coarse coefficient volume for reproducing the original shape, following the procedure in Section 4.3.

However, directly generating the shape from latent code z would lead to loss in topological structures and fine geometric details originally in the shape; compare Figure 4(a) and (b). To enhance the quality of the inverted shape, we further propose a shape-guided refinement scheme to search for latent code \(z^{\prime }\) around z in the latent space to better fit the input shape \(C_0\). In detail, we initialize latent code \(z^{\prime }\) as z and adapt it by gradient descent using the inversion objective in Equation (2) (see Section 4.2) for 400 iterations on the input. Using the initial latent code z, we can already obtain a plausible shape similar to the input. By using this shape-guided refinement, we can further obtain fine details and structures in local regions missed in the initial code z. Also, note that during the refinement, both the encoder and generator networks are fixed. As Figure 4(c) shows, our refined latent code helps encourage more faithful shape inversion with precise topological structures and fine geometric details, more similar to the original inputs. In Section 5.2, we will present quantitative evaluation of our results, showing that our approach can produce inverted shapes of much higher fidelity, compared with the state-of-the-art methods.

Fig. 4.

View Figure

4.5 Shape Manipulation

Shape inversion enables us to faithfully encode shapes into the learned latent space, in which our latent codes can faithfully represent their associated 3D shapes. Further, we design the shape generation process with the latent code as a condition. Hence, by manipulating the latent code and regenerating shape from the manipulated code, we can produce new shapes from the existing ones, e.g., by simply interpolating latent codes of different shapes; see our high-quality shape interpolation results shown in Figure 5(a).

Fig. 5.

View Figure

Further than that, our method enables a rich variety of region-aware manipulations, in which we can manipulate a particular region of a shape, while leaving other regions untouched:

To support these region-aware shape manipulations, we utilize the property that our generated coarse coefficient volume \(C_0\) can maintain a good spatial correspondence with the original TSDF volume. Hence, one can select a region in a shape and locally manipulate the associated portion in the coarse coefficient volume. However, naively changing the coefficient values in the selected region will introduce noise around the connecting boundary, since the new coefficient values may not be consistent with the original coefficients in the other regions of the shape; see, e.g., Figure 7(a).

Fig. 6.

View Figure

Fig. 7.

View Figure

To address this issue, we propose the region-aware manipulation procedure shown in Figure 6 by extending the shape inversion pipeline. Overall, the diffusion process takes T steps to produce the manipulated coarse coefficient volume \(C^{A}_{0}\) from the random noise volume \(C^{A}_T\). See Figure 6 (top left), given input shape A and its refined latent code \(z_A\) (from Section 4.4), we first run the shape inversion process for \(\Delta T\) steps to obtain the partially-denoised coefficient volume \(C^A_{T-\Delta T}\). In parallel, see Figure 6 (bottom left), we do the same on input shape B (which can be the donor/target shape, depending on the type of the manipulation operation) and its refined code \(z_B\) to produce another coefficient volume \(C^{B}_{T-\Delta T}\).

Importantly, after every \(\Delta T\) steps in the diffusion process (where \(T = M \cdot \Delta T\) for some positive integer M), we replace the coefficient values in the selected region of \(C^{A}_{T-\Delta T}\) by those in \(C^{B}_{T-\Delta T}\) to produce the mixed coefficient volume \(C^{\text{mix}}_{T-\Delta T}\). As the coefficient values near the mixed boundary in \(C^{\text{mix}}_{T-\Delta T}\) may not be consistent from \(C^{A}_{T-\Delta T}\) and \(C^{B}_{T-\Delta T}\), the generated shapes may contain some small artifacts; see, e.g., Figure 7(b). To smooth the coefficient mixing, we propose to adopt the harmonizing process in Lugmayr et al. [2022] to produce the harmonized coefficient volume \(C^{\text{h}}_{T-\Delta T}\) (details in the next paragraph). By harmonizing \(C^{\text{mix}}_{T-\Delta T}\) after combining \(C^A_{T-\Delta T}\) and \(C^B_{T-\Delta T}\), we can obtain a smooth transition of coefficient values near the boundary; see the improved result in Figure 7(c). After that, the harmonized coefficient volume can guide the subsequent steps of the two processes, so we use a combine-and-harmonize process in every \(\Delta T\) steps to obtain the final manipulated coefficient volume; see again Figure 6. We empirically set \(\Delta T = 10\) for region-aware manipulation experiments. Also, we can replace the inversion process guided by the refined latent code \(z_B\) with an unconditional shape generation for achieving part-wise re-generation. For the details on how we select the manipulation region in the input shape and how we compute the corresponding region in the wavelet domain; please refer to supplementary material Section F.

Details on the harmonizing process. Given the mixed coefficient volume \(C^{\text{mix}}_t\), we follow the forward process of the diffusion model to add noise to it by sampling \(C^{\text{mix}}_{t+1} \sim \mathcal {N}(\sqrt {1 - \beta _t} {C}_{t}, \beta _t \mathbf {I})\). Then, we apply the two inversion processes mentioned above, guided by the refined latent codes \(z_A\) and \(z_B\), respectively, separately on \(C^{\text{mix}}_{t+1}\), for a single step to obtain two denoised coefficient volumes. We then combine the volumes according to the selected region again and repeat the above adding noise and denoising procedure ten times to obtain the harmonized coefficient volume. By doing so, the generator can better account for the coefficient value changes in the manipulated region and adapt the values near the boundary. In the case of part-wise re-generation, we use the unconditional shape generation for harmonizing the mixed coefficient volume instead of the inversion process guided by the refined latent code \(z_B\).

Discussions on shape manipulation. While our approach is able to create high-fidelity shapes with fine details, there is still no guarantee that the results always meet the user expectations. For example, the quality of the manipulated result can be sensitive to hyper-parameters, e.g., \(\Delta T\). Furthermore, the alignment between the input shape and donor shape can impact the performance of the region-aware manipulations, since it is hard to expect shapes with a significant misalignment can produce a reasonable result.

4.6 Implementation Details

We employed ShapeNet [Chang et al. 2015] to prepare the training dataset used in all our experiments. For shape generation, we follow the data split in Chen and Zhang [2019] and use only the training split to supervise our network training. For shape inversion, we follow the data split in Park et al. [2019] for ease of comparison. Also, similar to Hertz et al. [2022], Li et al. [2021], and Luo and Hu [2021], we train one model of each category in the ShapeNet dataset [Chang et al. 2015] for both shape generation and inversion.

We implement our networks using PyTorch and run all experiments on a GPU cluster with four RTX3090 GPUs. For both shape generation and inversion, we follow [Ho et al. 2020] to set \(\lbrace \beta _t\rbrace\) to increase linearly from \(1e^{-4}\) to 0.02 for 1,000 steps and set \(\sigma _t = \frac{1-\bar{\alpha }_{t-1}}{1 - \bar{\alpha }_t} \beta _t\). We train the generator, optionally with the encoder, for 800,000 iterations and the detail predictor for 60,000 iterations, both using the Adam optimizer [Kingma and Ba 2014] with a learning rate of \(1e^{-4}\). Training the generator and detail predictor takes around three days and 12 hours, respectively. For shape-guided refinement, we also adopt Adam optimizer [Kingma and Ba 2014] with a learning rate of \(5e^{-2}\) for 400 iterations. For shape generation, the inference takes around six seconds per shape on an RTX 3090 GPU. For shape inversion, the refinement procedure and inference totally take around two minutes on an RTX 3090 GPU using 1,000 diffusion steps. As for shape manipulation, our region-aware manipulation procedure takes four minutes on an RTX 3090 GPU for running 1,000 diffusion steps. We adapt [Cotter 2020] to implement the 3D wavelet decomposition.

5.1 Shape Generation

Galleries of our generated shapes. We present Figure 1 (left) and Figure 8 to showcase the compelling capability of our method for generating shapes of various categories. Our generated shapes exhibit diverse topology, fine details, and also clean surfaces without obvious artifacts, covering a rich variety of small, thin, and complex structures that are typically very challenging for the existing approaches to produce. More 3D shape generation results produced by our method are provided in supplementary material Section A.

Fig. 8.

View Figure

Baselines for comparison. We compare the shape generation capability of our method with six state-of-the-art methods: IM-GAN [Chen and Zhang 2019], Voxel-GAN [Kleineberg et al. 2020], Point-Diff [Luo and Hu 2021], SPAGHETTI [Hertz et al. 2022], SDF-StyleGAN [Zheng et al. 2022], and GET3D [Gao et al. 2022]. To our best knowledge, our method is the first work that generates implicit shape representations in frequency domain and considers coarse and detail coefficients to enhance the generation of structures and fine details.

Our experiments follow the same setting as the above works. Specifically, we leverage our trained model on the Chair and Airplane categories in ShapeNet [Chang et al. 2015] to randomly generate 2,000 shapes for each category. Then, we uniformly sample 2,048 points on each generated shape and evaluate the shapes using the same set of metrics as in the previous methods (details to be presented later). For GET3D, we use the official pre-trained model for the Chair category and adopt their code to train a model for the Airplane category, which is not provided in their released repository. For the other comparison methods, we employ publicly-released trained network models to generate shapes.

Evaluation metrics for shape generation. Following [Luo and Hu 2021; Hertz et al. 2022], we evaluate the generation quality using (i) minimum matching distance (MMD) measures the fidelity of the generated shapes; (ii) coverage (COV) indicates how well the generated shapes cover the given 3D repository; and (iii) 1-NN classifier accuracy (1-NNA) measures how well a classifier differentiates the generated shapes from those in the repository. Overall, a low MMD, a high COV, and an 1-NNA close to 50% indicate good generation quality; see supplementary material Section K for the details.

Quantitative evaluation for shape generation. Table 1 reports the quantitative comparison results, showing that our method surpasses all others for almost all the evaluation cases over the three metrics for both the Chair and Airplane categories. We employ the Chair category, due to its large variations in structure and topology, and the Airplane category, due to the fine details in its shapes. As discussed in Luo and Hu [2021] and Yang et al. [2019], the COV and MMD metrics have limited capabilities to account for details, so they are not suitable for measuring the fine quality of the generation results, e.g., the generated shapes sometimes show a better performance even when compared with the ground-truth training shapes on these metrics. In contrast, 1-NNA is more robust and can better correlate with the generation quality. In this metric, our approach outperforms all others, while having a significant margin in the Airplane category, manifesting the diversity and fidelity of our generated results.

Qualitative evaluation for shape generation. Figure 9 shows some visual comparisons. For each random shape generated by our method, we find a similar shape (with similar structures and topology) generated by each of the other methods to make the visual comparison easier. See supplementary material Sections B and D for more visual comparisons. Further, as different methods likely have different statistical modes in the generation distribution, we also take random shapes generated by IM-GAN and find similar shapes generated by our method for comparison; see supplementary material Section C for the results. From all these results, we can see that the 3D shapes generated by our method clearly exhibit finer details, higher fidelity structures, and cleaner surfaces, without obvious artifacts.

Fig. 9.

View Figure

5.2 Shape Inversion

Baselines for comparison. We compare our shape inversion results with those from four state-of-the-art methods: IM-NET [Chen and Zhang 2019], DeepSDF [Park et al. 2019], DualSDF [Hao et al. 2020], and SPAGHETTI [Hertz et al. 2022]. We employ their official code to train their models, following the same train-test split as Park et al. [2019] for a fair comparison. Note that we do not evaluate the results of SPAGHETTI in the Lamp category, as its official pre-trained model is unavailable and the training code has not been officially released by the time of the submission. Further, we notice a very recent work, NeuForm [Lin et al. 2022], which overfits the given shape for achieving the shape inversion. Even though the code and pre-trained models of this work have not been released, we provide a visual comparison on the inversion results in supplementary material Section E using the example results given in their paper.

Evaluation metrics for shape inversion. Following prior works, we evaluate the inversion quality by measuring the similarity between the inverted shapes and the original inputs using Chamfer Distance (CD), Earth Mover’s Distance (EMD), and Light Field Distance (LFD) [Chen et al. 2003]. For CD and EMD, they evaluate point-wise distances between two point clouds sampled on the shape surfaces. Here, we uniformly sample 2,048 points on each shape (inverted and original) and evaluate the metrics on the sampled point clouds. For LFD, it measures the difference between two shapes in the rendered image domain. First, we uniformly sample 20 viewpoints to render images for each of the inverted shapes and input shapes. Then, we compute a 45-dimension feature vector for each rendered image and obtain the final metric by summing up all pairwise L1 distances between the image feature vectors from the same viewpoint. Note that a low value indicates a better performance for all three metrics. Also, there could be noise in the sampled TSDF grid, so we post-process the inverted shapes. For the details on the metrics and post-processing, please see supplementary material Section L.

Quantitative evaluation for shape inversion. Table 2 shows the quantitative comparison results. Without refinement, our method already performs better in all metrics than IM-NET [Chen and Zhang 2019], which also does not require additional refinement. By further refining the latent code, our method can significantly outperform all the other methods in all metrics. Particularly, for the LFD metric, our method manifests more than \(30\%\) performance gain over the second-best method in all categories.

Qualitative evaluation for shape inversion. Figure 10 shows some visual comparisons on shape inversion. We can observe that our method can already produce plausible results without refinement; see Figure 10(f). By further introducing the shape-guided refinement, we can faithfully reproduce the fine details and complex structures in the input shapes; see, e.g., the thin structures at the aircraft’s propeller and the chair’s back shown in the fourth and fifth rows of Figure 10; note that they cannot be achieved by the existing works.

Fig. 10.

View Figure

5.3 Shape Manipulation

Shape Interpolation. As Figure 11 shows, our method can produce smooth and plausible interpolations between two unseen shapes, thanks to the shape-guided refinement. From left to right, the source can morph smoothly towards the target; see especially the consistent changes in the armrests in the last row of Figure 11. These results manifest the superior capability of our framework to embed an unseen shape in a smooth and plausible latent space.

Fig. 11.

View Figure

Part Replacement. Besides, we can select a part in an input shape and replace it with a corresponding part in the donor shape. As Figure 12 shows, we replace the blue part in each input (a) with the blue part in the corresponding donor (b) to generate new shapes (d). Note that the new shapes exhibit high fidelity, while preserving the geometric details of (a) & (b). See particularly the last row of Figure 12; the complex armrests and back in the input shape can be well preserved in our new shape, while the chair’s seat and legs are seamlessly connected with clean surfaces and sharp edges.

Fig. 12.

View Figure

Further, we compare our method with SPAGHETTI [Hertz et al. 2022], the state-of-the-art implicit method for the same task. In particular, SPAGHETTI produces inverted shapes as a mixture of Gaussian distributions, where each of them is conditioned by a latent code. Part replacement on the input can then be conducted by replacing some particular latent codes with the corresponding ones in another shape. However, their results are somehow noisy in the connecting regions, while struggling to preserve the geometry and details of the input and part donor; see Figure 12(c). More visual comparisons on part replacement with SPAGHETTI [Hertz et al. 2022] and COALESCE [Kangxue Yin and Zhang 2020] are provided in supplementary material Section F. Note also that for the examples shown in the main paper, we cannot provide comparisons of COALESCE, since it needs additional part-level annotations, which are not available for these examples.

Part-wise Interpolation. Also, our method enables us to select a part (region) in the source shape and perform interpolation on it towards the target shape. See, e.g., Figure 13; during the interpolation, the green part of the source morphs smoothly towards the associated part in the target shape while preserving the remaining parts (marked in blue). See the drawer’s knobs of the table in the second row and the chair’s legs in the third row of Figure 13, the geometry and topology of the untouched parts are preserved faithfully. Besides, the selected part of the intermediate results is consistent with the untouched part; see the connecting regions between the chair’s seat and legs in the last row of Figure 13.

Fig. 13.

View Figure

Part-wise Re-generation. Further, our method allows us to specify a part in the input shape and replace it with a randomly-generated part; see Figure 14. The randomly-generated parts are diverse and plausible; see the table at top left of Figure 14. The connecting regions in the tables keep consistent across all the re-generation results, while the re-generated legs of the tables exhibit diverse topology structures and geometric details.

Fig. 14.

View Figure

More shape manipulation results. Please refer to supplementary material Section F for more results on shape interpolation, part replacement, part-wise interpolation, and part-wise re-generation.

5.4 Shape Reconstruction

Shape reconstruction from point clouds and single-view images. By utilizing the latent space learned by the encoder network, we can leverage our method to help reconstruct implicit shapes from various forms of inputs. As suggested by Chen and Zhang [2019], we propose to train another encoder that takes a point cloud or a single-view image as input and to predict a latent code that matches the one obtained by our trained encoder network in Section 4.2. Given an unseen input (a point cloud or a single-view image), we can then use the new encoder to generate a corresponding latent code and take the code to reconstruct the 3D shape.

Figure 15 shows visual examples of our reconstruction results. We can reconstruct shapes with fine details and thin structures solely by predicting the latent codes. The results show that the learned latent space is highly smooth and covers various unseen inputs. Interestingly, our method can reconstruct some occluded parts in the single-view images, e.g., the occluded legs of the chairs, by leveraging the shape priors learned in the latent space.

Fig. 15.

View Figure

5.5 Novelty Analysis on Shape Generation

Next, we analyze whether our method can generate shapes that are not necessarily the same as the training-set shapes, meaning that it does not simply memorize the training data. To do so, we use our method to generate 500 random shapes and retrieve the top-four most similar shapes in the training set separately via two different metrics, i.e., Chamfer Distance (CD) and Light Field Distance (LFD) [Chen et al. 2003]. It is noted that LFD is computed based on rendered images from multiple views on each shape, so it focuses more on the visual similarity between shapes and is considered to be more robust for shape retrieval. For the details on the metrics, please see supplementary material Section H.

Figure 16 (top) shows a shape generated by our method, together with top-four most similar shapes retrieved from the training set by the CD and LFD metrics. Further, we show another ten examples in supplementary material Section G. Comparing our shapes with the retrieved ones, we can see that the shapes share similar structures, showing that our method is able to generate realistic-looking structures like those in the training set. Beyond that, our shapes exhibit noticeable differences in various local structures.

Fig. 16.

View Figure

As mentioned earlier, a good generator should produce diverse shapes that are not necessarily the same as the training shapes. So, we further statistically analyze the novelty of our generated shapes relative to the training set. To do so, we use our method to generate 500 random chairs; for each generated chair shape, we use LFD to retrieve the most similar shape in the training set. Figure 16 (bottom) plots the distribution of LFDs between our generated shapes (in green) and retrieved shapes (in blue). Also, we show four shape pairs at various percentiles, revealing that shapes with larger LFDs are more different from the most similar shapes in the training set. From the LFD distribution, we can see that our method can learn a generation distribution that covers shapes in the training set (low LFD) and also generates novel and realistic-looking shapes that are more different (high LFD) from the training-set shapes.

5.6 Ablation Study

Ablation on shape generation. To evaluate the major components in shape generation, we successively ablate the full pipeline. First, we evaluate the effect of detail predictor on the generation performance. Then, we study the contributions of the diffusion model by replacing it with a variational autodecoder (VAD) [Zadeh et al. 2019; Hao et al. 2020; Cheng et al. 2022]. Last, we ablate the wavelet representation against directly producing the TSDF volumes with two settings. Specifically, the first setting directly generates a low-resolution TSDF volume of the same resolution as our coarse wavelet coefficient volume (\(46^3\)), whereas the second one applies a network of the same architecture as our detail predictor to upsample the TSDF volume to the same resolution as our detail wavelet coefficient volume (\(76^3\)).

First, from the results reported in Table 3 first vs. second rows, we can see the capability of the detail predictor, which introduces a substantial improvement on all metrics. Second, replacing our generator with the VAD model leads to a significant performance degradation; see first vs. third rows. Third, the ablation of the wavelet representation indicates that directly producing a low-resolution SDF leads to a consistent performance drop in all metrics; see first vs. fourth rows. In the second setting (first vs last rows), though the upsampling network can improve the generation quality, there still exhibit a significant performance gap compared with our full pipeline in terms of all metrics, especially “1-NNA.”

Figure 17 provides visual ablation results on shape generation (top) and inversion (bottom). First, comparing Figures 17(a) and (b) shows that the absence of the detail predictor leads to a loss of fine details and thin structures. Second, replacing the diffusion model with the VAD model, as proposed in Zadeh et al. [2019], leads to a considerable performance drop, as indicated in Figure 17(c). Third, comparing Figures 17(b) and (d) shows that our compact wavelet representation outperforms the TSDF in the same resolution \(46^3\) (the first setting). This is primarily due to the superior shape representation capability of our neural wavelet representation, while the TSDF of such a low resolution cannot well capture necessary information especially the high-frequency components. In the second setting, despite that the upsampling network (Figures 17 (e)) can capture more details than (d), it still falls short when compared with the full pipeline, as shown in Figures 17(a) and (e).

Fig. 17.

View Figure

Besides, we explored the possibility of using a GAN [Goodfellow et al. 2014] (with the same 3D convolution network architecture) to generate wavelet coefficients. However, we found that the model cannot converge well even though adopting the loss function introduced in WGAN [Arjovsky et al. 2017]. It can be partially attributed to the complexity of the wavelet coefficients in the frequency domain, making it challenging for the GAN model to converge. To ensure fairness, we compare our diffusion model with the VAD model, which can achieve a nearly monotonic decrease of objective loss in the training process, as shown in Figures 17(a) and (c).

Ablation on shape inversion. Next, we explore our shape inversion pipeline on the Chair category to evaluate its major components. First, we quantitatively evaluate the effect of directly reconstructing the input from the latent code predicted by the encoder without the shape-guided refinement. Second, we evaluate the shape inversion performance with/without the detail predictor. Last, we evaluate the effectiveness of the wavelet representation for shape inversion by constructing a baseline to predict the low-resolution and high-resolution TSDFs as introduced in the generation setting.

Table 4 reports the quantitative results, demonstrating the capability of our shape-guided refinement (first vs. second rows). Further, dropping the detail predictor largely degrades the performance (first vs. third rows).

Also, directly predicting the low-resolution TSDFs can harm the inversion performance (“Full model” vs. “Low-resolution TSDF”), which aligns with the above findings on shape generation (first vs. fourth rows). Similarly, introducing the upsampling network can improve the shape inversion quality (“Full model” vs. “Low-resolution TSDF”), yet it still yields inferior performance compared with the full pipeline (first vs. last rows).

Furthermore, Figures 17 (f–k) present visual ablation results on shape inversion. First, inversion using the latent code without refinement can already produce a reasonably similar shape to the input. However, it is difficult to recover complex topological structures, as depicted in Figures 17(g) and (h). Second, when the detail predictor is ablated, there is a large performance drop in inversion capabilities, particularly on the thin and fine structures of the shape, as demonstrated in Figures 17 (g) and (i). Also, directly predicting a low-resolution TSDF also proves to be inadequate, especially when dealing with shapes like thin bars at the chair’s back, as evidenced in Figures 17(i) and (j). Also, despite that adopting a high-resolution TSDF can enhance the quality of the inverted shapes as shown in Figures 17(j) and (k), it still encounters challenges to generate shapes with fine details, as shown in Figures 17(g) and (k).

Details of ablation study. Please refer to supplementary material Section M for the implementation details of each ablation baseline.

5.7 Limitations and Future Works

Discussion on shape generation. While our method can generate diverse and realistic-looking shapes, the generated shapes may not meet the desired functionality. As Figure 18(a) shows, the generated chairs, despite looking interesting and structurally reasonable, have an exceptionally tall seat back and low seat height for normal human bodies. In the future, we may incorporate functionality, e.g., [Blinn et al. 2021], into the shape generation process. Second, though our method can learn efficiently via the wavelet representation, it still requires a large number of shapes for training. So, for categories with few training samples, especially those with complex and very thin structures; see, e.g., Figure 18(b), the generated shape may exhibit artifacts like broken parts and structures. Last, while our method can better fit the data distribution than existing methods, the generated shapes still conform to structures/appearances in the training set. Exploring how to generate out-of-distribution shapes is an interesting but challenging future direction.

Fig. 18.

View Figure

Discussion on shape inversion and manipulation. Our method is able to invert and reconstruct shapes faithfully and facilitate manipulations with high fidelity, yet requiring users to manually select regions for manipulation. An interesting future direction is to guide the shape manipulation by sketches or texts [Li et al. 2017a; Liu et al. 2022]. Also, while our compact wavelet representation maintains a good spatial correspondence with the original TSDF domain and enables various part-aware manipulations, inter-shape correspondences still rely mainly on a canonically-aligned dataset. We hope to establish stronger correspondences in the compact wavelet domain to enable more compelling downstream applications such as texture and detail transfer. Besides, the diffusion process requires a large number of time steps, so producing manipulated shapes is time-consuming. Yet, simply reducing the time steps leads to a significant quality drop. We plan to explore acceleration strategies to promote interactive shape manipulation.