Towards Robust Semantic Segmentation against Patch-Based Attack via Attention Refinement

Abstract

The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both convolution neural networks and vision transformer as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field brought by global attention may lead to the spread of the adversarial patch. To address this issue, in this paper, we propose a robust attention mechanism (RAM) to improve the robustness of the semantic segmentation model, which can notably relieve the vulnerability against patch-based attacks. Compared to the vallina attention mechanism, RAM introduces two novel modules called max attention suppression and random attention dropout, both of which aim to refine the attention matrix and limit the influence of a single adversarial patch on the semantic segmentation results of other positions. Extensive experiments demonstrate the effectiveness of our RAM to improve the robustness of semantic segmentation models against various patch-based attack methods under different attack settings.

Appendices

Appendix A The Details of Attack Methods

In this section, we first introduce the general framework of attack methods, and then describe ten attack methods (i.e., PGD Madry et al. (2018), DAG Xie et al. (2017), IPatch Mirsky (2021), SSAP Nesti et al. (2022), Patch-Fool Fu et al. (2022), Attention-Fool Lovisotto et al. (2022), AutoAttack Croce and Hein (2020), EOT Athalye et al. (2018), MaxVarDAG and MaxAttnDAG) in detail.

In order to comprehensively evaluate the defense effect of our proposed RAM, we first use the widely used PGD (Madry et al., 2018) method, together with three strong attack methods designed for semantic segmentation models (i.e., DAG (Xie et al., 2017), IPatch (Mirsky, 2021), SSAP (Nesti et al., 2022)) to evaluate the robustness of models. In addition, we further adopt recent proposed attention-based attack methods (e.g., Patch-Fool Fu et al. (2022) and Attention-Fool Lovisotto et al. (2022)), together with several strong attack methods which often used as the benchmark (i.e., AutoAttack Croce and Hein (2020) and EOT Athalye et al. (2018)), to further evaluate the robustness of segmentation models. Moreover, in order to more comprehensively evaluate the effectiveness of our defense method, we use several adaptive attack methods to conduct the evaluation. Specifically, based on the DAG method, we propose two adaptive attack methods, MaxVarDAG and MaxAttnDAG to elaborately attack the attention matrix.

1.1 A.1 The General Framework of Attack Methods

We use the process of adversarial attack against the semantic segmentation model in DAG Xie et al. (2017) method as the general framework, and add the limitation of a patch-based mask to it. The algorithm is shown in Algorithm 1, where T is the number of iterative steps in the attack, $\gamma$ is a hyperparameter to control the scale of the gradient, $\text {Clip}$ means to clip the generated adversarial examples to the image scale range, i.e., [0, 1]. The specific loss functions used in different attack methods are introduced below.

In the experiments, we set $T=400$ for all attack methods. For the hyperparameter of $\gamma$, we set $\gamma =0.005$ for PGD Madry et al. (2018), and $\gamma =1$ for the other all attack methods.

Algorithm 1

The General Framework of Attack Methods

1.2 A.2 Projected Gradient Descent

Projected Gradient Descent (PGD) (Madry et al., 2018) method is one of the earliest classic adversarial attack methods, and it focuses on the adversarial attack of the classification model under $L_p$ norm. In the targeted attack scenario, it generates adversarial examples by maximizing the cross-entropy loss between model predictions and target labels. In this paper, we migrate it to the task of patch-based attacks on semantic segmentation models. Assume that the predicted segmentation result of the model for the input image $\textbf{X}\in \mathbb {R}^{H\times W\times 3}$ is $\textbf{Y}\in [0,1]^{H\times W\times C}$, the ground truth label is $\mathbf {Y^g}\in \{0,1,\cdots , C-1\}^{H\times W}$ and the target label specified by the attacker is $\mathbf {Y^t}\in \{0,1,\cdots , C-1\}^{H\times W}$, where H and W are the height and width of the input image, and C is the number of categories for image segmentation. The loss function of PGD method is as follows:

$$\begin{aligned} \mathcal {L}_{\text {PGD}}=\sum _{i=0}^{H-1}\sum _{j=0}^{W-1}\log \textbf{Y}_{ij\mathbf {Y^t}_{ij}}, \end{aligned}$$

(A1)

where $\mathbf {Y^t}_{ij}$ represents the target label at position (i, j) in the image, and $\textbf{Y}_{ijc}$ represents the probability predicted by the model that the pixel at position (i, j) in the image belongs to category c.

1.3 A.3 Dense Adversary Generation

Dense Adversary Generation (DAG) (Xie et al., 2017) method is an attack method specific to dense prediction tasks, such as semantic segmentation and object detection. The basic idea is to define a dense set of targets as well as a different set of desired labels, and optimize a loss function in order to produce incorrect recognition results on all the targets simultaneously. The loss function used in DAG is as follows:

$$\begin{aligned} \mathcal {T} = \{(i,j)|\arg \max _c \textbf{Y}_{ijc} = \mathbf {Y^g}_{ij} \}, \end{aligned}$$

(A2)

$$\begin{aligned} \mathcal {L}_{\text {DAG}}=\sum _{(i,j)\in \mathcal {T}} \textbf{Y}_{ij\mathbf {Y^g}_{ij}} - \textbf{Y}_{ij\mathbf {Y^t}_{ij}}. \end{aligned}$$

(A3)

1.4 A.4 IPatch

IPatch (Mirsky, 2021) aims to change the semantics of locations far from the patch to a specific category by placing a universal adversarial patch anywhere within an image. When we utilize the method of IPatch to conduct the patch-based attack in our work, we fix the location of the adversarial patch at the lower right corner of the image and generate the adversarial patch for each image separately. The loss function used in IPatch is as follows:

$$\begin{aligned} \mathcal {L}_{\text {IPatch}}&=\mathcal {L}_{\text {KL}}(\text {onehot}(\mathbf {Y^t}), \textbf{Y})\end{aligned}$$

(A4)

$$\begin{aligned}&=\sum _{i=0}^{H-1}\sum _{j=0}^{W-1}\sum _{c=0}^{C-1}\text {onehot}(\mathbf {Y^t})_{ijc}\nonumber \\&\quad \cdot \log \frac{\text {onehot}(\mathbf {Y^t})_{ijc}+\epsilon }{\textbf{Y}_{ijc}}, \end{aligned}$$

(A5)

where $\epsilon =1\textrm{e}{-10}$.

1.5 A.5 Scene-specific Patch Attack

The Scene-specific Patch Attack (SSPA) (Nesti et al., 2022) method designs a new loss function, which considers the pixels that have been successfully attacked and the pixels that have not yet been successfully attacked simultaneously, and adjusts the weights of the two terms through adaptive coefficients. The attack process is formalized as follows:

$$\begin{aligned}{} & {} \mathcal {T}=\{(i,j)|\arg \max _c \textbf{Y}_{ijc}\ne \mathbf {Y^t}_{ij} \}, \end{aligned}$$

(A6)

$$\begin{aligned}{} & {} \eta = \frac{|\mathcal {T}|}{H\cdot W}, \end{aligned}$$

(A7)

$$\begin{aligned}{} & {} \mathcal {L}_{\text {SSAP}}=\eta \cdot \sum _{(i,j)\in \mathcal {T}}\log \textbf{Y}_{ij\mathbf {Y^t}_{ij}} + (1-\eta )\nonumber \\{} & {} \quad \cdot \sum _{(i,j)\notin \mathcal {T} }\log \textbf{Y}_{ij\mathbf {Y^t}_{ij}}. \end{aligned}$$

(A8)

1.6 A.6 Patch-Fool

Given the insensitivity of ViTs’ self-attention mechanism to local perturbations in the task of classification, Patch-Fool (Fu et al., 2022) is designed to fool the self-attention mechanism by attacking its basic component (i.e., a single patch) with a series of attention-aware optimization techniques. It should be noted that the Patch-Fool method is an attack method designed for the classification model, so we have partially modified the loss function in order to be suitable for the attack on semantic segmentation models.

The loss function used in Patch-Fool consists of two parts, one of which is the commonly used cross-entropy loss:

$$\begin{aligned} \mathcal {L}_{\text {CE}}=\sum _{i=0}^{H-1}\sum _{j=0}^{W-1}\log \textbf{Y}_{ij\mathbf {Y^t}_{ij}}. \end{aligned}$$

(A9)

The other one is the attention-aware loss:

$$\begin{aligned} \mathcal {L}_{\text {ATTN}}^{(l)}=\sum _{h,i}a_p^{(l,h,i)}, \end{aligned}$$

(A10)

where $\mathbf {a^{(l,h,i)}}=[a_1^{(l,h,i)}, \cdots , a_n^{(l,h,i)}]\in \mathbb {R}^n$ is the attention distribution for the i-th token of the h-th head in the l-th layer, $a_p^{(l,h,i)}$ denotes the attention value corresponding to the adversarial patch p.

Table 15 The robustness of semantic segmentation models against two patch-based targeted attack settings

The adversarial patch is then updated based on both the cross-entropy loss and the layer-wise attention-aware loss:

$$\begin{aligned} \mathcal {L}_{\text {Patch-Fool}}=\mathcal {L}_{\text {CE}}+\alpha \sum _l \mathcal {L}_{\text {ATTN}}^{(l)} - \alpha \sum _l \beta _l \mathcal {L}_{\text {CE}}, \end{aligned}$$

(A11)

where

$$\begin{aligned} \beta _l= {\left\{ \begin{array}{ll} 0,&{} \langle \nabla _\textbf{x} \mathcal {L}_{\text {CE}}, \nabla _\textbf{x} \mathcal {L}_{\text {ATTN}}^{(l)} \rangle >0, \\ \frac{\langle \nabla _\textbf{x} \mathcal {L}_{\text {CE}}, \nabla _\textbf{x} \mathcal {L}_{\text {ATTN}}^{(l)} \rangle }{\Vert \nabla _\textbf{x} \mathcal {L}_{\text {CE}}\Vert ^2},&{} \text {otherwise}, \end{array}\right. } \end{aligned}$$

(A12)

where l is the predefined self-attention layer and $\alpha$ is a weighted coeffficient.

1.7 A.7 Attention-Fool

Attention-Fool (Lovisotto et al., 2022) claims that Vision Transformer (ViT) models may give a false sense of robustness due to dot-product attention, and can be easily attacked with a specific kind of attack called Attention-Fool attack. It should be noted that the Attention-Fool method is also an attack method designed for the classification model, so we have partially modified the loss function in order to be suitable for the attack on semantic segmentation models.

Fig. 5

More visualizations about the comparison of segmentation results of different models

Table 16 The robustness of semantic segmentation models against two patch-based targeted attack settings

We denote projected queries by $\mathbf {P_Q^{hl}=X^{hl}W_Q^{hl}}\in \mathbb {R}^{n\times d_q}$ and projected keys by $\mathbf {P_K^{hl}=X^{hl}W_K^{hl}}\in \mathbb {R}^{n\times d_k}$, where $\mathbf {X^{hl}}$ is the feature of images fed to the h-th attention head of l-th layer, $\mathbf {W_Q^{hl}}$ and $\mathbf {W_K^{hl}}$ are the learned projection matrices, respectively. Then $\mathbf {B^{hl}}=\frac{\mathbf {P_Q^{hl}}(\mathbf {P_K^{hl}})^\top }{\sqrt{d_k}}\in \mathbb {R}^{n\times n}$ quantifies the dot-product similarity between each pair of key and query in the attention head h and layer l.

$\mathcal {L}_{kq}$ is proposed to attack all attention layers and heads simultaneously, which is formalized as follows:

$$\begin{aligned}{} & {} \mathcal {L}_{kq}^{hl}=\frac{1}{n}\sum _j \mathbf {B_{jp}^{hl}}, \end{aligned}$$

(A13)

$$\begin{aligned}{} & {} \mathcal {L}_{kq}=\log \sum _l \exp (\log \sum _h \exp (\mathcal {L}_{kq}^{hl})), \end{aligned}$$

(A14)

where $\mathbf {B_{jp}^{hl}}$ denotes the corresponding feature of adversarial patch p.

The loss function used in Attention-Fool attack is the combination of the cross-entropy loss and $\mathcal {L}_{kq}$:

$$\begin{aligned} \mathcal {L}_{\text {Attention-Fool}} = \mathcal {L}_{\text {CE}}+\alpha \mathcal {L}_{kq}, \end{aligned}$$

(A15)

where $\alpha$ is a weighted coefficient.

1.8 A.8 AutoAttack

AutoAttack (Croce & Hein, 2020) is a strong adaptive attack method that has been widely used in recent years to evaluate the robustness of the model. It is a kind of parameter-free ensemble attack, which consists of four attack methods, i.e., APGD$_{\text {CE}}$, APGD$_{\text {DLR}}$, FAB (Croce & Hein, 2020) and Square Attack (Andriushchenko et al., 2020). It should be noted that AutoAttack is also an attack method designed for the classification model, so we have partially modified the loss function in order to be suitable for the attack on semantic segmentation models.

1.9 A.9 EOT

Expectation Over Transformation (EOT) (Athalye et al., 2018) method is first proposed by Athalye et al., which calculates the expected value of the loss function over the distribution of possible transformations (e.g., random rotation, translation, addition of noise), and then uses this expected value to guide the generation of adversarial examples. EOT method allows the attacker to craft adversarial examples that are effective even when the defense mechanism introduces randomness into the input image.

1.10 A.10 MaxVarDAG

In order to further explore the defense effect of our method under adaptive attack, we designed two additional adaptive attack methods (i.e., MaxVarDAG and MaxAttnDAG) based on DAG, that is, to elaborately attack the design of our attention mechanism, so as to more comprehensively evaluate the effectiveness of our defensive approach. Specifically, MaxVarDAG extends the DAG method by introducing an additional regularization term that maximizes the variance of the attention matrix in order to amplify the influence of the dirty patch. The loss function of MaxVarDAG is formalized as follows:

$$\begin{aligned} \mathcal {L}_{\text {MaxVarDAG}} = \mathcal {L}_{\text {DAG}}+\alpha \cdot \text {Var} (\textbf{M}), \end{aligned}$$

(A16)

where $\textbf{M}$ is the normalized attention matrix in Eq. (5), $\text {Var}$ represents the funciton of variance, $\alpha$ is a weighted coefficient.

1.11 A.11 MaxAttnDAG

MaxAttnDAG also extends DAG by directly adding a regularization term that maximizes the attention value corresponding to the dirty patch position. Through this design, the inference of the dirty patch to the result of image semantic segmentation can be maximized, so as to conduct the adaptive attack based on our proposed defense method. The loss function of MaxVarDAG is formalized as follows:

$$\begin{aligned} \mathcal {L}_{\text {MaxAttnDAG}} = \mathcal {L}_{\text {DAG}}+\alpha \cdot \text {resize}(\sum _i \mathbf {M_{i\cdot }}) \odot \mathbf {M_{mask}},\nonumber \\ \end{aligned}$$

(A17)

where $\textbf{M} \in \mathbb {R}^{N\times N}$ is the normalized attention matrix in Eq. (5), $\text {resize}$ refers to the process of resizing the attention value vector of each position $\sum _i \mathbf {M_{i\cdot }}$ to a new size of $H\times W$, $\mathbf {M_{mask}} \in \{0,1\}^{H\times W}$ is the mask of dirty patch, $\alpha$ is a weighted coefficient.

Appendix B More Visualization Results

In this section, we visualize more segmentation results of semantic segmentation models under different attack scenarios in Fig 5.

In Fig 5a, we compare the segmentation results of UPerNet (Xiao et al., 2018)/DeiT (Touvron et al., 2021) model with our proposed RAM and the baseline under the attack setting of Permute. When the baseline model is faced with an input image with the adversarial patch, the segmentation result almost completely meets the adversary’s intention, that is, most areas of the segmentation results are consistent with the target labels determined by the adversary. As a comparison, when the model is with our proposed RAM, the adversarial patch less affects the output of the model, and most of the regions maintain correct segmentation results.

In Fig. 5b, we compare the segmentation results of SegFormer (Xie et al., 2021)/MiT (Xie et al., 2021) model with our proposed RAM and the baseline under the attack setting of Strip. The segmentation results of the baseline model are very close to the strip pattern designed by the adversary, while the segmentation results of our RAM are rarely affected.

Both of the visualization results demonstrate that the baseline model is quite vulnerable to patch-based attacks, while our RAM method significantly improves the robustness of semantic segmentation models against various attack scenarios.