State-aware video procedural captioning

Abstract

Video procedural captioning (VPC), which generates procedural text from instructional videos, is an essential task for scene understanding and real-world applications. The main challenge of VPC is to describe how to manipulate materials accurately. This paper focuses on this challenge by designing a new VPC task, generating a procedural text from the clip sequence of an instructional video and material set. In this task, the state of materials is sequentially changed by manipulations, yielding their state-aware visual representations (e.g., eggs are transformed into cracked, stirred, then fried forms). The essential difficulty is to convert such visual representations into textual representations; that is, a model should track the material states after manipulations to better associate the cross-modal relations. To achieve this, we propose a novel VPC method, which modifies an existing textual simulator for tracking material states as a visual simulator and incorporates it into a video captioning model. Our experimental results show the effectiveness of the proposed method, which outperforms state-of-the-art video captioning models. We further analyze the learned embedding of materials to demonstrate that the simulators capture their state transition.

Appendices

Appendix A: Details of simulator

In this section, we describe the details of the visual simulator for the reproducibility of the proposed method. Given the encoded vectors of the clip sequence and material list $(\boldsymbol {{\mathscr{H}}}, \boldsymbol {\mathcal {E}}^{0})$, the visual simulator, shown in Fig. 3 in the main paper, recurrently reasons the state transition of materials at each step. Specifically, at the n-th step, given the n-th clip h_n and (n − 1)-th material list $\boldsymbol {\mathcal {E}}^{n-1}$, the visual simulator predicts executed actions and involved materials in (1) the action and (2) material selector and it then updates the state of materials in (3) the updater. After n-th reasoning, it outputs a state-aware step vector $\boldsymbol {u}_{n} \in \mathbb {R}^{3 \times d}$, which concatenates the n-th clip h_n, selected action $\bar {\boldsymbol {f}}_{n}$ and material vectors $\bar {\boldsymbol {e}}_{n}$ (d represents the dimension of these vectors). The visual simulator recurrently repeats the above process until processing the end element of the clip sequence. For clarity, we explain the simulation process of the visual simulator at the n-th step.

Action selector

Given a clip vector h_n, the action selector outputs the selected action vector $\bar {\boldsymbol {f}}_{n}$ by choosing actions executed in the clip from predefined action embedding $\boldsymbol {\mathcal {F}}$. For example, in Fig. 3 in the main paper, the actions “crack” and “stir” are executed in the clip, thus both f_crack and f_stir should be selected. To consider multiple actions, the action selector computes a soft selection w_p as action probability for each action in $\boldsymbol {\mathcal {F}}$. Then it outputs the selected action vector $\boldsymbol {\bar {f}}_{n}$ as a weighted sum of the action embedding $\boldsymbol {\mathcal {F}}$ and action probability w_p:

$$ \begin{array}{@{}rcl@{}} \boldsymbol{w}_{p} &=& \text{MLP}(\boldsymbol{h}_{n}) \end{array} $$

(A1)

$$ \begin{array}{@{}rcl@{}} \bar{\boldsymbol{w}}_{p} &=& \frac{\boldsymbol{w}_{p}}{{\sum}_{j}\boldsymbol{w}_{p}^{j}} \end{array} $$

(A2)

$$ \begin{array}{@{}rcl@{}} \bar{\boldsymbol{f}}_{n} &=& \bar{\boldsymbol{w}}_{p}^{T} \boldsymbol{\mathcal{F}}, \end{array} $$

(A3)

where MLP(⋅) represents two-layer MLPs with the sigmoid function and $\boldsymbol {w}_{p} \in \mathbb {R}^{\|\boldsymbol {\mathcal {F}}\|}$ is the attention distribution over $\|\boldsymbol {\mathcal {F}}\|$ possible actions⁸.

Material selector

Based on the action probability w_p and clip vector h_n, the material selector outputs the selected material vector $\boldsymbol {\bar {e}}_{n}$ by choosing materials involved in the clip from the material list $\boldsymbol {\mathcal {E}}^{n-1}$. For example, in Fig. 3 in the main paper, the raw “cheese” and manipulated “eggs” and “butter” should be selected. To consider such a combination of raw and manipulated material selection, the material selector has two attention modules: (1) clip attention and (2) recurrent attention.

• (1) The clip attention chooses relevant materials from the clip vector h_n and action probability w_p:

  $$ \begin{array}{@{}rcl@{}} \hat{\boldsymbol{\textit{h}}}_{n} &=& \text{ReLU}(\boldsymbol{W}_{1} \boldsymbol{\textit{h}}_{n} + \boldsymbol{b}_{1}) \end{array} $$

  (A4)
  $$ \begin{array}{@{}rcl@{}} \boldsymbol{d}_{m} &=& \sigma((\boldsymbol{\textit{e}}_{m}^{n-1})^{\textsf{T}} \boldsymbol{W}_{2} [\hat{\boldsymbol{\textit{h}}}_{n};\boldsymbol{w}_{p}]) \end{array} $$

  (A5)

  where W₁ and W₂ are linear and bilinear mapping, b₁ and b₂ are biases, and $\boldsymbol {\textit {e}}_{m}^{n-1}$ and d_m represent the m-th material vector and its attention weight.

• (2) Recurrent attention selects materials based on information from both the current and previous clips. Using the result of clip attention, it computes a soft selection aⁿ as a material probability for each material in the material list:

  $$ \begin{array}{@{}rcl@{}} \boldsymbol{c} &=& \text{softmax}(\boldsymbol{W}_{3} \hat{\boldsymbol{\textit{h}}}_{n} + \boldsymbol{b}_{3}) \end{array} $$

  (A6)
  $$ \begin{array}{@{}rcl@{}} \boldsymbol{a}_{m}^{n} &=& \boldsymbol{c}_{1}\boldsymbol{d}_{m} + \boldsymbol{c}_{2}\boldsymbol{a}_{m}^{n-1} + \boldsymbol{c}_{3} \boldsymbol{0} \end{array} $$

  (A7)

  where W₃ is a linear mapping, $\boldsymbol {c} \in \mathbb {R}^{3}$ is the choice distribution, $\boldsymbol {a}_{m}^{n-1}$ is the attention weight of the previous clip for each material, $\boldsymbol {a}_{m}^{n}$ is the final distribution for each material, and 0 is a vector of zeros (providing the option not to select any materials). Finally, using the calculated attention weights, the selected material vector $\boldsymbol {\bar {e}}_{n}$ is computed as the normalized weighted sum of the selected materials.

  $$ \begin{array}{@{}rcl@{}} \boldsymbol{\alpha}_{m}^{n} &=& \frac{\boldsymbol{a}_{m}^{n}}{{\sum}_{j}\boldsymbol{a}_{j}^{n}} \end{array} $$

  (A8)
  $$ \begin{array}{@{}rcl@{}} \bar{\boldsymbol{\textit{e}}}_{n} &=& \sum\limits_{m}\boldsymbol{\alpha}_{m}^{n}\boldsymbol{e}_{m}^{n-1}. \end{array} $$

  (A9)

Updater

Based on the selected actions and materials, the updater represents the state transition of materials by computing a new material vector $\hat {\boldsymbol {e}}_{m}$. To this end, it first calculates an action-aware proposal vector l_n of materials with a bilinear transformation of the selected action and material vectors $(\boldsymbol {\bar {f}}_{n},\boldsymbol {\bar {e}}_{n})$:

$$ \boldsymbol{l}_{n} = \text{ReLU}(\bar{\boldsymbol{f}}_{n}\boldsymbol{W}_{4}\bar{\boldsymbol{\textit{e}}}_{n} + \boldsymbol{b}_{4}), $$

(A10)

where W₄ is a bilinear mapping.

Then, based on the material probability aⁿ, it computes the new material vector $\hat {\boldsymbol {e}}_{m}$ by interpolating the action-aware proposal vector l_n and current material vector $\hat {\boldsymbol {e}}_{m}^{n-1}$:

$$ \hat{\boldsymbol{e}}_{m} = \boldsymbol{a}_{n}^{m}\boldsymbol{l}_{n} + (1 - \boldsymbol{a}_{n}^{m}) \boldsymbol{e}_{m}^{n-1}. $$

(A11)

The new m-th material vector $\hat {\boldsymbol {e}}_{m}$ is assigned to $\boldsymbol {\mathcal {E}}_{m}^{n}$, which is forwarded to the next (n + 1)-th step.

Appendix B: Detailed annotation process

We additionally annotated ingredients with the rest of the recipes (126 recipes) to the YouCook2-ingredient dataset, and built the YouCook2-ingredient+ dataset.

We increased the dataset size by obtaining the missing videos through YouCook2 author and annotating ingredients with these additional videos by hiring one annotator to use the web tool shown in Fig. 10. This annotation tool presents a recipe, corresponding video, and text boxes for writing ingredients. In this paper, ingredients are defined as raw materials that are necessary to complete the dish. For example, “tomato” and “cucumber” should be written as ingredients, although “salad” should not be written because it represents a mixture of ingredients.

Fig. 10

A screen of our browser-based web annotation tool. Annotators write ingredients that appear in the recipes. To ease annotation, we preliminarily estimated ingredients using the NER method [1] pre-trained on the English recipe flow graph corpus [50], and we set the default values of inputs

To annotate ingredients easily, “jump” buttons, which allow annotators to see a clip corresponding to a step, are implemented based on the start/end timestamp from the original YouCook2 dataset. Moreover, to encourage annotators to write ingredients easily, this tool displays estimated ingredients using the named entity recognition (NER) model, flair⁹ [1] pre-trained on the English recipe flow graph corpus (E-rFG corpus) [50]¹⁰. If the estimated words are not appropriate for the ingredients, the ingredients can be deleted or rewritten.

Appendix C: Baseline implementation details

As our comparative models, we employed two state-of-the-art transformer-based video captioning models: Transformer-XL [9] and MART [22]. These models originally have no ingredient set in their inputs and copy mechanism in their decoder; thus for a fair comparison, we prepare for additional baseline +ingredient (-I) models, which incorporate the material encoder (Section 3.2) and the copy mechanism into the baselines.

These models are based on the transformer that encodes sequential inputs and decodes a sentence by attending all of the elements in the input sequence. Thus, to fit this characteristic, we concatenate the encoded ingredient and video vectors, and input them to the model, as shown in Fig. 11. When decoding, based on the output of the decoder o_k and ingredient vectors $\boldsymbol {\mathcal {E}}_{0}$, the copy mechanism calculates the copying gate to make a soft choice between selecting an ingredient from the ingredient set or generating a word from the vocabulary.

Fig. 11

An overview of our baseline +ingredients (-I) implementation. These models also incorporate the material encoder described in Section 3.2 and copy mechanism into the baselines

Appendix D: Implementation and training details on full prediction settings

Here, we discuss the implementation and training details of the full prediction settings, where the material set is not given, but is predicted from the video clips in advance. To address this, as described in Section 4.7, we added an ingredient decoder of the multi-label classifier and trained the entire model as multi-task learning.

Figure 12 shows how to integrate the ingredient decoder into the model. The ingredient decoder consists of a two-layered MLP with a sigmoid function, and converts $\hat {\boldsymbol {h}}$ of a max-pooled vector of clip vectors into a probability vector of materials, where q indicates the number of unique ingredients appearing more than three times in the training set (we obtained q = 668 in the experiment). During training, we compute the ingredient decoder loss ${\mathscr{L}}_{ingr}$, which is an asymmetric loss [51] on the multi-label classification settings, and add it to the total loss defined in (4). Note that we adopt teacher-forcing [48] to stabilize the training; while the models learn using the ground-truth ingredients for the downstream process in the training phase, they generate a recipe based on predicted ingredients at the inference phase (we sample the top k = 15 ingredients from the probability). Another modification of the model is to remove the copy mechanism because we find that it degrades the captioning performance with this setting.

Fig. 12

An overview of how to integrate ingredient decoder into the model