APPTracker+: Displacement Uncertainty for Occlusion Handling in Low-Frame-Rate Multiple Object Tracking

Abstract

Multi-object tracking (MOT) in the scenario of low-frame-rate videos is a promising solution to better meet the computing, storage, and transmitting bandwidth resource constraints of edge devices. Tracking with a low frame rate poses particular challenges in the association stage as objects in two successive frames typically exhibit much quicker variations in locations, velocities, appearances, and visibilities than those in normal frame rates. In this paper, we observe severe performance degeneration of many existing association strategies caused by such variations. Though optical-flow-based methods like CenterTrack can handle the large displacement to some extent due to their large receptive field, the temporally local nature makes them fail to give reliable displacement estimations of objects that newly appear in the current frame (i.e., not visible in the previous frame). To overcome the local nature of optical-flow-based methods, we propose an online tracking method by extending the CenterTrack architecture with a new head, named APP, to recognize unreliable displacement estimations. Further, to capture the fine-grained and private unreliability of each displacement estimation, we extend the binary APP predictions to displacement uncertainties. To this end, we reformulate the displacement estimation task via Bayesian deep learning tools. With APP predictions, we propose to conduct association in a multi-stage manner where vision cues or historical motion cues are leveraged in the corresponding stage. By rethinking the commonly used bipartite matching algorithms, we equip the proposed multi-stage association policy with a hybrid matching strategy conditioned on displacement uncertainties. Our method shows robustness in preserving identities in low-frame-rate video sequences. Experimental results on public datasets in various low-frame-rate settings demonstrate the advantages of the proposed method.

Appendices

Appendix A: Detailed Experimental Setups

This section elaborates on the detailed setups of experiments on MOT17 (Milan et al., 2016) and MOT20 (Dendorfer et al., 2020).

FairMOT (Zhang et al., 2021). It is a ReID-based model and it randomly samples frames during training, making its training stage frame-rate-agnostic.

ByteTrack (Zhang et al., 2022). The tracking module of ByteTrack only includes a Kalman filter without learnable parameters. In our experiments, we evaluated ByteTrack with the same detection set obtained from our model.

Tracktor (Bergmann et al., 2019). The trainable module of Tracktor is an object detector (Ren et al., 2015) and thus is frame-rate-agnostic. In the inference stage, tracking is achieved by propagating the bounding boxes from the previous frame to the next frame as proposals and leveraging the denoising capability of the detector’s regression branch to obtain the refined boxes.

APPTracker/APPTracker+. Our model took two adjacent frames as input and was trained on 1/10-frame-rate videos. Following CenterTrack (Zhou et al., 2020a), the time interval between the two input frames is uniformly sampled from {-20, -10, 0, 10, 20} where pairs in temporally reverse order are allowed.

CenterTrack (Zhou et al., 2020a). To ensure a fair comparison on the same detection set, CenterTrack is evaluated using detections and displacements estimated by our model, while employing its own association logic. The simulated performance of CenterTrack slightly exceeds that reported by the original authors, as shown in Table 9. This is due to the different ignoring policies on low-visible objects during training, as illustrated in Sect. 3.3.1 of the main text.

Table 10 Buffer sizes for different settings of $n_d$

Buffer size. Table 10 lists the buffer sizes (above which an inactive tracklet is terminated) we adopted for different frame rate settings.

Fig. 10

Cross-dataset evaluation (trained on MOT17 (Milan et al., 2016) and tested on MOT20 (Dendorfer et al., 2020)) with multiple frame rate downsampling factor $n_d$ (x-axis label). Arrows indicate the direction of better performance

For each method, only one model is trained on each dataset and is tested at different frame rates.

Appendix B: Cross-Dataset Evaluation with Multi-Frame-Rate Settings

Figure 10 shows the multi-frame-rate experiments for the cross-dataset evaluation (trained on MOT17 (Milan et al., 2016) and tested on MOT20 (Dendorfer et al., 2020)) where we maintained our original experimental setup, using the same detection set for all methods except FairMOT and FairMOT*. As shown in Fig. 10, the shape of the curves approximately aligns with the previous results on the MOT17 validation set except the ReID-based method denoted by FairMOT*. Besides, we note that (1) in high-frame-rate videos, our method slightly falls behind ByteTrack (Zhang et al., 2022) and FairMOT (Zhang et al., 2021), mainly due to the advantages of the Kalman filter (Kalman, 1960) in high-frame-rate scenarios, as has been analyzed in Table 3 of the main text. This gap can be mitigated on the engineering side by switching the motion model according to target frame rates; (2) the detection performance of our model is slightly affected by frame rate. This is because our detection branch is learned jointly with the displacement estimation branch which shares the input of two adjacent frames. On the engineering side, one may address this influence by decoupling the detection and displacement estimation modules.

Appendix C: Further Investigations on Re-Identification Models

1.1 C.1 FairMOT

In this section, we conduct a comprehensive investigation of FairMOT (Zhang et al., 2021). The conclusions can be summarized in two points: (1) FairMOT’s excessive confidence in historical appearance embeddings leads to significant additional failures in low-frame-rate videos. (2) The differences in detection entries are one of the main reasons for the performance degradation of ReID-based trackers in low-frame-rate videos, and it is not well addressed by adjusting the matching threshold. We elaborate on the details below. To investigate the ReID performance of FairMOT, in the following experiments, we disable the Kalman filter together with the strict spawning strategy.

Impact of matching threshold. We first study the impact of adjusting the matching threshold $\tau$. Given two appearance embeddings, ${\varvec{e}}_a$ and ${\varvec{e}}_b$, FairMOT takes the cosine distance $1-\frac{{\varvec{e}}_a \cdot {\varvec{e}}_b}{\Vert {\varvec{e}}_a \Vert _2 \Vert {\varvec{e}}_b \Vert _2}$ as the matching cost, which ranges from 0 to 2. According to Table 11, adjusting the matching threshold does not receive significant performance improvements. We set the matching threshold $\tau$ to 0.5 throughout our experiments (which was officially set to 0.4 in normal-frame-rate cases).

Table 11 IDF1 score vs. matching threshold $\tau$ on MOT17 validation set (1/10-frame-rate)

Table 12 IDF1 score vs. $\alpha$ and $n_d$

Fig. 11

IDF1 scores vs. $\alpha$. Results on MOT17 validation set (1/10 frame rate)

Impact of online appearance embedding update. Further, we conducted additional investigations on FairMOT and found that its excessive confidence in historical appearance embeddings leads to significant additional failures in low-frame-rate videos. Specifically, FairMOT (Zhang et al., 2021) smoothly updates the latest appearance embeddings of tracklets. In a new time step t, the registered appearance embedding $\varvec{e}^{t}_{\text {smooth}}$ of an associated tracklet is updated by

$$\begin{aligned} \varvec{e}^{t}_{\text {smooth}} = \alpha \varvec{e}^{t-1}_{\text {smooth}} + (1-\alpha )\varvec{e}^{t}, \end{aligned}$$

(10)

where $\alpha$ is a smoothness factor and is set to 0.9 by default, indicating the low confidence on the new appearance observation $\varvec{e}^{t}$. We performed a grid search on $\alpha$ for various frame rate settings in Table 12 and plot the curves of $\alpha = \{0.0, 0.3, 0.6, 0.9\}$ in Fig. 11 for ease of observation. We use a fixed $\alpha$ of 0.6 in the main text to avoid over-tailored hyperparameters.

Fig. 12

Typical failure cases of FairMOT* (1/10 frame rate). In the first row of each case, we present cosine similarity heatmaps of appearance embeddings and annotate the query object in the first frame. The second and third rows provide tracking results from FairMOT* and our method, respectively. The appearance embedding is updated frame-by-frame with the historical embedding weight $\alpha = 0$. We highlight the recognized appearing objects in our results by indicating the predicted APP scores on the right top of corresponding bounding boxes

Table 13 IDF1 score vs. matching threshold $\tau$ (1/10-frame-rate)

Fig. 13

Results on MOT17 validation set

Fig. 14

Results on MOT20 validation set (APPTracker+ and FairMOT* are trained on MOT17)

According to Table 12, the optimal values for parameter $\alpha$ are roughly distributed along the diagonal. This is reasonable, considering that targets exhibit larger appearance changes as the frame rate goes lower. According to Fig. 8a and b of the main text, while FairMOT* achieves comparable IDF1 scores to CenterTrack (Zhou et al., 2020a), there is still a gap in terms of AssA. This is because the IDF1 score encourages more about estimating the total number of unique objects in a scene than about good detection or association (Luiten et al., 2021). Without the Kalman filter and the strict spawning strategy, FairMOT* tends to transfer identities between targets rather than spawning new tracks with new identities, which is preferred by the IDF1 score. Another weakness of using smaller values of $\alpha$ is that it may compromise long-term re-identification across occlusions.

Finally, we once again conducted a grid search of matching threshold $\tau$ with $\alpha =0.5$. As shown in Table 13, using a small value of $\alpha$ makes the performance more sensitive to the matching threshold $\tau$. However, compared to the default threshold $\tau$ of 0.4, parameter tuning still brings minor gains.

Impact of difference in detection entries. Further, we visualize the typical failure cases of the ReID-based tracking in Fig. 12, where we find that the differences in detection entries caused by occlusion are one of the main reasons for the performance degradation of ReID-based trackers in low-frame-rate videos. Considering that identifying the detection variations is one of the main ideas in our work, we believe our work has the potential to enhance the performance of ReID-based trackers in low-frame-rate scenarios.

1.2 C.2 DeepSORT

Besides FairMOT (Zhang et al., 2021), we also study the effectiveness of DeepSORT with a pre-trained ReID model and our detection set. Specifically, we follow the same DeepSORT implementation adopted by ByteTrack (Zhang et al., 2022), where the ReID model was trained on a large-scale ReID dataset, MARS (Zheng et al., 2016), containing over 1,100,000 images of 1,261 pedestrians. We provide the results on the MOT17 (Milan et al., 2016) validation set in Fig. 13. To study the tracking performance through ReID alone, we also include a version with the Kalman filter (Kalman, 1960) turned off (denoted as DeepSORT*). Additionally, we experiment with different matching thresholds⁴$\tau$, but receive no benefit compared to the default setting of $\tau =0.1$.

For comparison, we attach the results of APPTracker+ (our method) and FairMOT* in Fig. 13, but set them in to avoid potential unfair comparisons, as APPTracker+ and FairMOT* are trained on half of the MOT17 data. From Fig. 13, the pre-trained model demonstrates better appearance discrimination capability compared to FairMOT*, especially at high frame rates.

Table 14 IDF1 score vs. matching threshold on MOT17 validation set (1/10-frame-rate)

Further, in Fig. 14, we provide results on the MOT20 (Dendorfer et al., 2020) validation set, which is characterized by crowded scenes and poor lighting conditions. Similarly, we attach the results of APPTracker+ and FairMOT*, which are trained on half of the MOT17 data. Shown in Fig. 14, under crowded and poor lighting conditions, (1) the pre-trained ReID also experiences reduced effectiveness, and (2) by comparing the results of DeepSORT and DeepSORT*, the motion model is found to consistently bring tracking performance improvements (as shown by the IDF1 score and the association accuracy, AssA), but lead to a huge decrease in DetA at low frame rates.

Appendix D: ByteTrack with GIoU

In Table 14, we report the results of adopting GIoU (Rezatofighi et al., 2019) in the Kalman-filter-based tracker, ByteTrack (Zhang et al., 2022), with various matching threshold settings but receive minor performance gain. The failure of the motion model in low-frame-rate videos is mainly due to (1) the failure of zero-velocity initialization, and (2) quick changes in target velocity.

Table 15 Computational cost

Appendix E: Computational Cost

We reported the FLOPs, MACs, and parameter count in Table 15. Compared with CenterTrack (Zhou et al., 2020a), our method introduces 7.7% extra FLOPs/MACs and 0.7% extra parameters when running on MOT17 (Milan et al., 2016) and MOT20 (Dendorfer et al., 2020). With the absence of the APP prediction head when running on KITTI (Geiger et al., 2012), our method only introduces negligible additional computational cost.