CEDFlow++: Latent Contour Enhancement for Dark Optical Flow Estimation

Abstract

CEDFlow introduces a latent contour enhancement method into dark optical flow estimation and achieves advanced performance. Nevertheless, it largely focuses on addressing the motion boundary in a local manner. Unfortunately, it falls short in performance when addressing significant variations or large-scale degraded scenes. This paper introduces CEDFlow++, which features three innovative modules to address the key challenges of CEDFlow. Firstly, we introduce a decomposition-based feature encoder (DBFE), which captures both fine-grained and large-scale features through its local encoder and a uniquely designed sparse attention-based global encoder that suppresses noise and interference that only exist in the dark. Secondly, for reliable motion analysis, we propose a customized dual cost-volume reasoning (DCVR), which integrates important high-contrast feature correlations of the global cost volume into the local cost volume, effectively capturing salient yet holistic motion information while mitigating motion ambiguity caused by darkness. Importantly, we present a contour-guided attention (CGA) which enables context-adaptive extraction of contour features by modifying the sign properties of the Sobel kernel parameters in latent space, specifically targeting large-scale contours that are suitable for motion boundaries. Experimental results on the FCDN and VBOF datasets show that CEDFlow++ outperforms state-of-the-art methods in terms of the EPE index and produces more accurate and robust optical flow.

Appendix

Basic and Context Backbones We follow a similar design as the well-known RAFT (Teed & Deng, 2020); we incorporate motion backbone and context backbone into our PsFE framework. The motion backbone, it outputs features at 1/8 resolution from $\textbf{x} \in \mathbb {R}^{3 \times H \times W}$ $\rightarrow$ $\textbf{x} \in \mathbb {R}^{C \times H/8 \times W/8}$ where we set C = 256. The motion backbone consists of 6 residual blocks, 2 at 1/2 resolution, 2 at 1/4 resolution, and 2 at 1/8 resolution. Furthermore, the structure of the context backbone is identical to that of the feature extraction network, except that BatchNorm regularization is used in the context branch and InstanceNorm is used in the basic branch.

The Local Feature Encoder. We used the local encoder in CEDFlow as a continuation. In detail, the local encoder employs three 2D residual convolutional blocks (with a small receptive field) to encode the local properties of each point, followed by ReLU activation to ensure robust information propagation of the fine-grained local feature.

The Global Feature Encoder. Our global encoder consists of a sparse attention method and an interaction layer. As shown in Fig. 19, the interaction layer first predicts a set of scale weights from the local feature $\hat{f^L}$, and then enhances the appearance information in the coarse $f^H$ while retaining high-contrast feature. The process is as follows,

$$\begin{aligned} V_p = L^V_p((1 + \mathcal{E}\mathcal{A}(\hat{f^L})) \cdot f^H), \end{aligned}$$

(15)

where ‘$\cdot$’ denotes an element-wise dot product. $\mathcal{E}\mathcal{A}()$ is a spatial attention module, which consists of a convolutional block and a sigmoid function to extract the importance weight of each point, thereby improving the spatial expression of feature.

Fig. 19

An overview of our sparse attention-based global encoder. It mainly contains a interaction layer and a sparse attention module. ‘$\times$’ is a matrix multiplication. ‘$\cdot$’ represents the dot product. ‘$+$’ denotes the pixel-wise addition