MoDA: Modeling Deformable 3D Objects from Casual Videos

Abstract

In this paper, we focus on the challenges of modeling deformable 3D objects from casual videos. With the popularity of NeRF, many works extend it to dynamic scenes with a canonical NeRF and a deformation model that achieves 3D point transformation between the observation space and the canonical space. Recent works rely on linear blend skinning (LBS) to achieve the canonical-observation transformation. However, the linearly weighted combination of rigid transformation matrices is not guaranteed to be rigid. As a matter of fact, unexpected scale and shear factors often appear. In practice, using LBS as the deformation model can always lead to skin-collapsing artifacts for bending or twisting motions. To solve this problem, we propose neural dual quaternion blend skinning (NeuDBS) to achieve 3D point deformation, which can perform rigid transformation without skin-collapsing artifacts. To register 2D pixels across different frames, we establish a correspondence between canonical feature embeddings that encodes 3D points within the canonical space, and 2D image features by solving an optimal transport problem. Besides, we introduce a texture filtering approach for texture rendering that effectively minimizes the impact of noisy colors outside target deformable objects.

Appendices

Appendix A: More details of MoDA

1.1 A.1 Skinning Weights for NeuDBS

We define the skinning weights for NeuDBS as $\textbf{W} = \{W_{1},..., W_{J}\} \in \mathbb {R}^{J}$, where J is the number of joint. Learning the skinning weights only from neural networks is difficult to optimize. To obtain the skinning weights for the proposed NeuDBS, we first calculate the Gaussian skinning weights and then learn the residual skinning weights with an MLP network following Yang et al. (2022).

Fig. 10

The texture filtering function

Firstly, we compute the Gaussian skinning weights based on the Mahalanobis distance between 3D points and the Gaussian ellipsoids,

$$\begin{aligned} \textbf{W}_{G} = (\textbf{X}-\textbf{O})^{T}\textbf{V}^{T}\varvec{\Lambda }^{0}\textbf{V}(\textbf{X}-\textbf{O}), \end{aligned}$$

(21)

where $\textbf{O} \in \mathbb {R}^{J \times 3}$ are the joint center locations, $\textbf{V} \in \mathbb {R}^{J \times 3 \times 3}$ are joint orientations and $\varvec{\Lambda }^{0} \in \mathbb {R}^{J \times 3 \times 3}$ are diagonal scale matrices. The joints represented by explicit 3D Gaussian ellipsoids are composed of these 3 elements: center, orientation, and scale. To learn better skinning weights for 3D deformation, we predict the residual skinning weights from an MLP network,

$$\begin{aligned} \textbf{W}_{r} = F_{skin}(\textbf{X}, \varvec{\psi }_{b}), \end{aligned}$$

(22)

then we have the final skinning weights,

$$\begin{aligned} \textbf{W} = \sigma _{softmax}(\textbf{W}_{G} + \textbf{W}_{r}). \end{aligned}$$

(23)

To be specific, the skinning weights $\textbf{W}^{t}_{o\xrightarrow c}$ are learned from 3D points in the observation space and the body pose code $\varvec{\psi }_{b}^{t}$ at time t, and $\textbf{W}^{t}_{c\xrightarrow o}$ are learned from 3D points in the canonical space and the rest pose code $\varvec{\psi }_{b}^{*}$.

1.2 A.2 Loss Functions

Optical flow loss. We render 2D flow to compute the optical flow loss. Specifically, we deform the canonical points to another time $t^{\prime }$ and get its 2D re-projection,

$$\begin{aligned} \textbf{x}^{t^{\prime }} = \sum _{k=1}^{N}\tau _{k}\textbf{P}^{t^{\prime }}(\mathcal {D}^{t^{\prime }}_{c\xrightarrow o}(\textbf{X}^{*}_{k})), \end{aligned}$$

(24)

where $\textbf{P}^{t^{\prime }}$ is the projection matrix of a pinhole camera. Then we can compute the 2D flow,

$$\begin{aligned} \textbf{f}(\textbf{x}^{t}, t\xrightarrow t^{\prime }) = \textbf{x}^{t^{\prime }} - \textbf{x}^{t}, \end{aligned}$$

(25)

and the optical flow loss $\mathcal {L}_{of}$ is defined as

$$\begin{aligned} \mathcal {L}_{of} = \sum _{\textbf{x}^{t}, (t, t^{\prime })}{\left\| \textbf{f}(\textbf{x}^{t}, t\xrightarrow t^{\prime }) - \widetilde{\textbf{f}}(\textbf{x}^{t}, t\xrightarrow t^{\prime })\right\| ^{2}}, \end{aligned}$$

(26)

where $\widetilde{\textbf{f}}$ is the observed optical flow that are extracted from off-the-shelf method (Yang & Ramanan, 2019).

3D cycle consistency loss. Similar to Li et al. (2021b); Yang et al. (2022), we introduce a 3D cycle consistency loss to learn better deformations. We deform the sampled points in the observation space to the canonical space and then deform them back to their original coordinates,

$$\begin{aligned} \mathcal {L}_{cyc} = \sum _{k}\tau _{k}{\left\| \mathcal {D}^{t}_{c\xrightarrow o}\left( \mathcal {D}^{t}_{o\xrightarrow c}(\textbf{X}^{t}_{k})) - \textbf{X}^{t}_{k}\right) \right\| ^{2}_{2}}, \end{aligned}$$

(27)

where $\tau _{k}$ weighs the sampled points to guarantee the points closer to the surface have stronger regularization.

Eikonal loss. Following Yang et al. (2022, 2023), we also adopt the implicit geometric regularization term (Gropp et al., 2020) as:

$$\begin{aligned} \mathcal {L}_{eikonal} = \sum _{\textbf{X} \in \textbf{V}^{*}}\left( \left\| \nabla F_{\textrm{SDF}}(\textbf{X})\right\| _{2} - 1\right) ^{2}. \end{aligned}$$

(28)

We observe that the eikonal loss often leads to instability and training failures. Following RAC (Yang et al., 2023), we solve this problem in the first stage by forcing the norm of the first-order derivative of signed distances d to be close to the mean norm rather than 1, which helps stabilize the training process. In subsequent stages, the target is changed from the mean norm to 1, ultimately contributing to a smoother geometry.

Fig. 11

Correspondence between different videos on casual-human and casual-cat. Distinct colors represent the correspondence

Fig. 12

2D-2D matching comparison. The mismatched results are highlighted with red boxes (Color figure online)

Fig. 13

Texture comparison of BANMo and our method on casual-cat. The fourth column is a screenshot of the example provided on BANMo’s website

1.3 A.3 Implementation Details

Training strategy. The optimization strategies of MoDA include three stages. Firstly, we optimize all losses and parameters. In this stage, MoDA already reconstructs good shape and deformation. Then we improve the articulated motions, where we only update the parameters related to the deformation model while keeping the shape parameters fixed. Finally, we improve the details of the reconstructions through importance sampling while freezing the camera poses. The design of MLP networks in MoDA is similar to BANMo (Yang et al., 2022). The hyperparameters $\gamma$ and $\lambda$ in the texture filtering function are set to 1.5 and 10 respectively (See Fig. 10 for the function). Our code will be available on GitHub once the paper is accepted.

Fig. 14

Novel view synthesis with or without texture filtering. There is noticeably more noise when texture filtering is removed

Sampling details for 2D–3D matching. In optimal transport, the sampling of 3D points is similar to BANMo (Yang et al., 2022). We establish a canonical 3D grid $\textbf{V}^{*} \in \mathbb {R}^{20\times 20\times 20}$ ($N_{point}= 8000$) to build correspondence between pixels and canonical points. This grid is centered at the origin and axis-aligned with bounds $[x_{min}, x_{max}], [y_{min}, y_{max}]$, and $[z_{min}, z_{max}]$, undergoes iterative refinement during optimization. Every 200 iterations, we update the bounds of the grid by approximating the object’s bounds. This approximation is obtained by applying marching cubes on a $64^{3}$ grid to extract a surface mesh.

Fig. 15

Qualitative comparison of different methods on casual-human and casual-adult (multiple-video setups). We show 2 views of the reconstruction results based on the reference images. ViSER (Yang et al., 2021) fails to learn detailed 3D shapes and accurate poses from the videos. BANMo (Yang et al., 2022) has obvious skin-collapsing artifacts (in the red circles) for motions with large joint rotations while our method performs well. The rest poses of casual-human and casual-adult are shown in the lower right corner

1.4 A.4 Dataset

We use 6 datasets in this work, where casual-cat, casual-human, eagle and hands are collected by BANMo (Yang et al., 2022). We have obtained permission to use these datasets. AMA dataset is the published dataset collected by Vlasic et al. (2008). The usage of casual-adult has also obtained consent.

Appendix B: More Experimental results

In this section, we show more experimental results.

Correspondence. In Fig. 11, we illustrate the correspondence between different videos in both the casual-human and casual-cat datasets. Distinct colors represent the correspondence. Besides, to compare the matching results between optimal transport and soft argmax method used in BANMo (Yang et al., 2022), we visualize the 2D–2D matching results in Fig. 12. As shown, optimal transport tends to produce more accurate matching results, while mismatches with soft argmax are highlighted with red boxes.

Texture rendering results. Here, we compare texture rendering results of BANMo (Yang et al., 2022) and our method on casual-cat. As shown in 13, we also provide a screenshot of the example on BANMo’s website.¹ The result of BANMo is aligned with the example on their website, appears unrealistic and potentially influenced by noise. In contrast, our method produces results more closely resembling the reference image.

Novel view synthesis. As the results reported in Fig. 9 and Fig. 13 are all rendered from meshes with texture, we also test the influence of texture filtering on volume rendering under novel views. For these novel view volume rendering results, we find that texture filtering is also beneficial. As shown in Fig. 14, the results with texture filtering exhibit less noise compared to those without it.

More results on multiple-video setups. We also show more results comparing MoDA with BANMo (Yang et al., 2022) and ViSER (Yang et al., 2021) on casual-human and casual-adult in Fig. 15.

Table 4 Training time comparison

Appendix C: Training Time

We compare the training times of our method and BANMo Yang et al. (2022) on different datasets. We train the models on two RTX 3090 GPUs. As shown in Table 4, MoDA and BANMo both have fast training on different datasets. BANMo takes around 8–10 h. MoDA takes about one hour more than BANMo. MoDA requires more computational time compared to BANMo due to two primary factors. Firstly, DBS employed in MoDA is inherently more time-consuming than Linear Blend Skinning (LBS), as reported in Figure 18 of Kavan et al. (2008). Secondly, the optimization process for solving the optimal transport problem adds additional computational overhead. However, the increased training time is acceptable considering MoDA’s superior performance.