Procedure-Aware Action Quality Assessment: Datasets and Performance Evaluation

Abstract

In this paper, we investigate the problem of procedure-aware action quality assessment, which analyzes the action quality by delving into the semantic and spatial-temporal relationships among various composed steps of the action. Most existing action quality assessment methods regress on deep features of entire videos to learn diverse scores, which ignore the relationships among different fine-grained steps in actions and result in limitations in visual interpretability and generalization ability. To address these issues, we construct a fine-grained competitive sports video dataset called FineDiving with detailed semantic and temporal annotations, which helps understand the internal structures of each action. We also propose a new approach (i.e., spatial-temporal segmentation attention, STSA) that introduces procedure segmentation to parse an action into consecutive steps, learns powerful representations from these steps by constructing spatial motion attention and procedure-aware cross-attention, and designs a fine-grained contrastive regression to achieve an interpretable scoring mechanism. In addition, we build a benchmark on the FineDiving dataset to evaluate the performance of representative action quality assessment methods. Then, we expand FineDiving to FineDiving+ and construct three new benchmarks to investigate the transferable abilities between different diving competitions, between synchronized and individual dives, and between springboard and platform dives to demonstrate the generalization abilities of our STSA in unknown scenarios, scoring rules, action types, and difficulty degrees. Extensive experiments demonstrate that our approach, designed for procedure-aware action quality assessment, achieves substantial improvements. Our dataset and code are available at https://github.com/xujinglin/FineDiving.

Appendix

1.1 About Annotator’s Disagreement

During the annotation process, we assess and measure inter-annotator agreement using metrics like Fleiss’ kappa $\kappa$, which can help identify and address inconsistencies between annotators. $\kappa$ is calculated by:

$$\begin{aligned} \kappa =\frac{P_A-P_e}{1-P_e} \end{aligned}$$

(19)

where $1-P_e$ gives the degree of agreement that is attainable above chance, and $P_A-P_e$ gives the degree of agreement actually achieved above chance. If the annotators are in complete agreement, then $\kappa \!=\!1$. If there is no agreement among the annotators (other than what would be expected by chance), then $\kappa \!\le \!0$. When $\kappa$ is below 0.6, we regard it as inter-annotator disagreement, and then additional clarification or higher-level judgments are necessary to resolve ambiguities. For instance, the guideline states that the somersault pike or somersault tuck in the flight is considered over when the torso opens at an angle of more than 90 degrees. If guidelines are well-defined and leave little room for interpretation, annotators are more likely to agree on the annotations. In this work, we provide clear and detailed annotation guidelines according to official FINA diving rules¹, including various examples and extreme cases, minimizing ambiguities and enhancing annotated data’s reliability. Taking defining where the sub-action “Entry” starts and ends as an example, when 1) the torso opens at an angle of more than 90 degrees, 2) the head, torso, and legs tend to be a straight line, and 3) extending arms straight above the head, with the hands close together, the sub-action “Entry” starts; when the hands enter first, followed by the head, shoulders, and the rest of the body, the sub-action “Entry” ends.