Indoor Obstacle Discovery on Reflective Ground via Monocular Camera

Abstract

Visual obstacle discovery is a key step towards autonomous navigation of indoor mobile robots. Successful solutions have many applications in multiple scenes. One of the exceptions is the reflective ground. In this case, the reflections on the floor resemble the true world, which confuses the obstacle discovery and leaves navigation unsuccessful. We argue that the key to this problem lies in obtaining discriminative features for reflections and obstacles. Note that obstacle and reflection can be separated by the ground plane in 3D space. With this observation, we firstly introduce a pre-calibration based ground detection scheme that uses robot motion to predict the ground plane. Due to the immunity of robot motion to reflection, this scheme avoids failed ground detection caused by reflection. Given the detected ground, we design a ground-pixel parallax to describe the location of a pixel relative to the ground. Based on this, a unified appearance-geometry feature representation is proposed to describe objects inside rectangular boxes. Eventually, based on segmenting by detection framework, an appearance-geometry fusion regressor is designed to utilize the proposed feature to discover the obstacles. It also prevents our model from concentrating too much on parts of obstacles instead of whole obstacles. For evaluation, we introduce a new dataset for Obstacle on Reflective Ground (ORG), which comprises 15 scenes with various ground reflections, a total of more than 200 image sequences and 3400 RGB images. The pixel-wise annotations of ground and obstacle provide a comparison to our method and other methods. By reducing the misdetection of the reflection, the proposed approach outperforms others. The source code and the dataset will be available at https://github.com/xuefeng-cvr/IndoorObstacleDiscovery-RG

Appendices

Appendix A: Principle of Ground-Pixel Parallax

In this section, we state the correctness of Eq. 4 in the main manuscript, and prove that this equation determines the relationship between an observed point and the ground.

Fig. 18

The ground-pixel parallax in two-view geometry

First of all, we give the proof of Eq. 4 in the main manuscript. Supposing $I^t$ and $I^{t-1}$ denote two consecutive images from robot’s view, $\pi$ denotes the ground plane, and $e^t, e^{t-1}$ denote the epipoles on the two images, which respectively are two intersection points of the two images and a line, i.e., the line connecting two camera optical centers, as the yellow points in Fig. 18. $x^t_i=\{u^t_i,v^t_i,1\}$ denotes an occlusion edge points of image $I^t$ in its homogeneous form, and it is also the 2D projection of a 3D point $X'$ on image $I^t$. A ray emitted from $I^t$’s optical center, denoted as $\textbf{l}$, penetrates $X'$ and $x^t_i$, and intersects the ground $\pi$ at a 3D point X. In addition, with the representation of the main manuscript, the geometric and appearance corresponding points are denoted as $g^{t-1}_i$ and $a^{t-1}_i$. According to Epipolar Constraints, the 3D line $\textbf{l}$ can be projected to image $I^{t-1}$ to form a projected 2D line, denoted as $\overrightarrow{g^{t-1}_i e^{t-1}}$. Since the 3D point $X'$ is on the 3D line $\textbf{l}$, its projection on image $I^{t-1}$, namely $a^{t-1}_i$, is on the projected 2D line $\overrightarrow{g^{t-1}_i e^{t-1}}$. Hence, these 2D points $a^{t-1}_i$, $g^{t-1}_i$, and $e^{t-1}$ are collinear. Based on this, 2D point $a^{t-1}_i$ can be stated as:

$$\begin{aligned} a^{t-1}_i = g^{t-1}_i + \rho (g^{t-1}_i-e^{t-1}) \end{aligned}$$

(14)

where $\rho$ is a scalar. This equation can be reformulated as Eq. 4 in the main manuscript.

Then, we discuss the reason that Eq. 4 in the main manuscript can be used to distinguish the points above the ground and below the ground. Noteworthily, since $g^{t-1}_i$ uniquely represents the projection of 3D ground point X, $g^{t-1}_i$ can be considered as the dividing point, and all points on both sides of this 2D line are partitioned into two spaces, above the ground and below the ground, as shown in Fig. 18. Hence, the space of point $x^{t}_i$ can be determined by comparing $a^{t-1}_i$ and $g^{t-1}_i$.

Fig. 19

Example results of the single frame reflection removal algorithm (Dong et al., 2021) and the multi-frame reflection removal algorithm (Nam et al., 2022) on multiple training scenes

Appendix B: Feasibility of Reflection Removal Methods

In scenes of reflection ground, it is intuitive to incorporate reflection removal approaches into the feature extraction part. Therefore, we employ the state-of-the-art single frame reflection removal algorithm (Dong et al., 2021) (ICCV 2021) and multi-frame reflection removal algorithm (Nam et al., 2022) (ECCV 2022), both of which have publicly available code. Their results on our dataset are visualized in Fig. 19. Note that since the real-world interfaces the reconstruction of multi-frame-based methods, we only conduct the reflection removal in the bottom half of the image, i.e., in the ground area.

Observably, both reflection removal methods are ineffective in removing reflections in our benchmark scenarios. In fact, they even damage the information of obstacles that needed to be detected. The single-frame-based method (Dong et al., 2021) produced confidence maps that failed to highlight the reflection, and the non-reflection images were almost identical to the original RGB images. Despite being free from the interface of the real world, the multi-frame-based method (Nam et al., 2022) still fails to eliminate reflections. The reason is that both algorithms require strong textures of main object for adequate reconstruction, but the ground texture is too weak to be perceived. In contrast, the reflection has a stronger texture than ground, making it appear as the main object. Overall, existing reflection removal algorithms cannot be directly used in the scene with reflective ground, and even damage obstacle information.

Appendix C: Feasibility of Depth Sensors

In recent years, multi-modal sensors have been increasingly popular in autonomous driving. Thus, we evaluate the usability of radar or depth cameras in reflective ground environments. To this end, we collect depth data of several reflective scenes by two classical sensors, i.e., the structured light camera (Kinect v1, released in 2010, priced at $150), the stereo camera (RealSense D455, released in 2019, priced at $249). The exemplar RGBD data is visualized in Fig. 20. Intuitively, the depth data obtained by these cameras in reflective environments is of such low quality that it cannot be applied in reflective scenes. Specifically, the structured light camera generates many void areas on the ground plane, while the stereo camera matches corresponding pixels erroneously between the cameras and results in completely incorrect depth data. Clearly, both types of cameras are unsuitable for reflective ground scenes.

Fig. 20

RGBD data captured by Kinect v1 and Realsense D455

Fig. 21

Capability tests of laser sensor. a Side view of scanning ground with a single-beam LiDAR. b Laser scan obtained in different angle

Furthermore, we conduct capability tests of laser sensor using a single-beam 360-degree LiDAR, which is illustrated in Fig. 21. The results illustrate that the single-beam LiDAR is not affected by reflections in all angles, which means that 3D LiDAR can obtain reliable depth information in reflective scenes. Unfortunately, although 3D LiDARs almost avoid the issue brought by reflective ground, it is too expensive to be deployed on a robot compared to other depth sensor.

Table 8 Formulation of each feature representing a bounding box

Appendix D: Detailed Formulation of Feature Vector

To clearly represent the feature used in this paper, Table 8 shows the formulation of each feature channel. Note that, according to Sect. 3.4, $b_j^t$ denotes the j-th bounding boxes in the t-th image, and its feature vector is denoted as $v_j^t$, which consists 19 channels grouped into five categories. To simplify the notation, we use $\textsf{b}$ to represent the bounding box $b_j^t$, specified by its top-left pixel coordinates $\left( \textsf{u},\textsf{v}\right)$ and its width and height $\left( \textsf{w},\textsf{h}\right)$. In Table 8, the notation $\check{\textsf{b}}$ refers to the inner ring of the bounding box $\textsf{b}$, while $\hat{\textsf{b}}$ represents the outer ring.

In the 2-nd channel, the notation $\left[ .\right]$ represents an indicator function that outputs 1 if the input is correct and 0 otherwise. In the 5-th channel, $\left( \textsf{W},\textsf{H}\right)$ denotes the width and height of the input image. For the 12-th - 17-th channels, the variables $\mathcal {H}$, $\mathcal {S}$, and $\mathcal {V}$ correspond to the input image’s HSV channels, and $\mathcal {H}\left( p\right)$ denotes the p-th pixel in the channel $\mathcal {H}$. Note that, the color contrast formulation involves the normalized histogram of box $\textsf{b}$’s H channel, represented as $hist_\textsf{b}^\mathcal {H}$, which is discretized into 18 bins denoted by $\{h_k^\mathcal {H},k\in \left[ 1,18\right] \}$, where k ranges from 1 to 18. The value of $h_k^\mathcal {H}$ is obtained as the sum over all pixels p in box $\textsf{b}$ such that $h_k^\mathcal {H}=\sum _{p\in \textsf{b}}\left[ \left\lfloor \frac{\mathcal {H}\left( p\right) }{360/18}\right\rfloor =k\right]$. Additionally, similar histograms $hist_\textsf{b}^\mathcal {S}$, $hist_\textsf{b}^\mathcal {V}$, $hist_{\hat{\textsf{b}}}^\mathcal {H}$, $hist_{\hat{\textsf{b}}}^\mathcal {S}$, $hist_{\hat{\textsf{b}}}^\mathcal {V}$ are computed in the same way.