1 Introduction

Thanks to the abundant human annotated data, deep learning has achieved great success in many supervised learning problems, such as image classification and retrieval (Krizhevsky et al., 2017; Zhou et al., 2019, 2020), object detection (Ren et al., 2015; Wang et al., 2018a), semantic segmentation (Long et al., 2015; Chen et al., 2017). However, labeling large-scale training data for each task is time consuming and expensive. Inspired by the human’s remarkable ability in recognizing instances of unseen classes solely based on class descriptions without seeing any visual example of such classes, zero-shot learning (ZSL) was proposed as an image classification setting to mimic the human learning process (Lampert et al., 2009). Given the semantic descriptions of both seen and unseen classes but only the seen class training images, ZSL aims to classify test images of unseen classes.

Based on the images and labels that a model can see in the training phase, ZSL includes two settings which are Inductive ZSL(IZSL) and Transductive ZSL(TZSL). IZSL can only utilize the images and labels of seen classes during training. TZSL can use extra images of unseen classes without labels during training. In the test phase, the ZSL problem is further categorized into two settings: conventional ZSL and generalized ZSL. In conventional ZSL, the images to be recognized at test time belong only to unseen classes. In the generalized ZSL (GZSL) setting, the images at test time may belong to both seen or unseen classes. The GSZL setting is practically more useful and challenging, since the assumption that images at test time come only from unseen classes need not hold.

Fig. 1
figure 1

Comparison of existing direct mapping methods and our latent bidirectional embedding based generative model. a Traditional ZSL frameworks are based on direct mapping. The hubness problem and calibration deviation make it difficult to accurately align visual and semantic distributions in respective spaces. b Our bidirectional embedding based generative model with a unified latent space. Firstly, an information bottleneck constraint on the latent bidirectional embedding preserves more essential attribute information while eliminating the non-semantic information of visual features. Secondly, uncertainty estimation is utilized to alleviate the visual noises and a bias passing mechanism is designed to solve the unicity of human annotated semantics. Thirdly, a wake-sleep procedure uses both real and generated data for joint training to improve the model representation and abstraction capability

Full size image

Most early ZSL methods learn a direct or indirect mapping between the visual space and the semantic space (Akata et al., 2013; Romera-Paredes & Torr, 2015; Akata et al., 2015b; Xian et al., 2016; Guo et al., 2016; Kodirov et al., 2017; Xie et al., 2019, 2020; Liu et al., 2021). However, the performance of these methods is often poor on GZSL setting. The reason is that the embedding model is learned only from seen classes, which leads to a bias towards predicting seen classes. To address this issue, more recent approaches (Xian et al., 2018b; Mishra et al., 2018; Wang et al., 2018b; Li et al., 2019a; Schonfeld et al., 2019; Ma & Hu, 2020; Verma et al., 2020; Yu et al., 2020) utilize generative models, e.g., generative adversarial networks (GAN) (Goodfellow et al., 2014) or variational autoencoders (VAE) (Kingma and Welling 2013), to generate synthetic features of unseen classes. This transfers the ZSL task to a supervised classification problem. Since GAN-based loss functions are unstable in training, VAE-based methods (Schonfeld et al., 2019; Ma & Hu, 2020) were developed to tackle this problem and achieved better performance. However, most of these generative models still suffer from the deviation between generated features and unseen classes due to the lack of tight visual-semantic coupling.

Fig. 2
figure 2

Illustration of the proposed model. We learn a latent bidirectional embedding based generative model via a modified VAE network. A unified latent space is simultaneously learned to align visual and semantic distributions. A novel Information Bottleneck (IB) loss (\({\mathcal {L}}_{IB}\)) is proposed as the constraint between the latent space and the other two spaces. We exploit the data uncertainty estimation to learn the bias (\(\varvec{\sigma }\)) of the original visual data and share the bias from visual distribution to the semantic distribution (bias passing). In addition, a wake-sleep procedure is used for joint training of real data and generated data

Full size image

Since high-dimensional visual features contain non-semantic information which is redundant for semantic discrimination (Tong et al., 2019; Han et al., 2020b; Shen et al., 2020), it is difficult to well align the semantic categories to the centers of visual sample distributions when mapping the semantic features to the visual space. This causes a calibration deviation problem as illustrated in Fig. 1. In addition, when high-dimensional visual features are mapped to a low-dimensional semantic space, the shrink of feature space would aggravate the hubness problem that some instances in the high-dimensional space become the nearest neighbors of a large number of instances (Radovanovic et al., 2010). To address the above challenges, we propose an information bottleneck (IB) (Tishby et al., 2000) constrained bidirectional embedding based generative model which utilizes advantages of both embedding model and generative model to align visual and semantic distributions in a unified latent space. As shown in Fig. 2, our proposed method first learns a latent bidirectional embedding via a modified VAE network. Visual features \({\mathbf {x}}\) and attributes semantic \({\mathbf {c}}\) are embedded to the unified latent space respectively by two encoders. Then the visual-semantic distributions alignment is constrained in the latent space by KL-Divergence and Center-Calibration. To facilitate the distribution alignment, the redundant non-semantic information in the visual space should be discarded to preserve the attributed related part when it is flowing to the latent space. To achieve this, we design an IB loss (\({\mathcal {L}}_{IB}\)) on the latent bidirectional embedding to impose the mutual information relationships between feature spaces. Due to the wide existence of noises such as the labeling noise (Kunran Xu et al., 2020), the human annotated semantics are insufficient to fully describe the diversified visual samples (Ding & Liu, 2019). The deviation between visual and semantic distributions will accumulate during the embedding process. Therefore, we learn the bias of the original visual distribution by introducing an uncertainty estimation technique (Kendall & Gal, 2017) to alleviate the influence of noises. Since one semantic class may correspond to a variety of visual samples, we also propose a bias passing mechanism to share this variety bias to the latent semantic distribution to benefit the distributions alignment. The two decoders are respectively used to generate attributes semantic and visual features for auxiliary training, i.e., the reconstruction losses \({\mathcal {L}}_{VCE}\) and \({\mathcal {L}}_{SCE}\). Since VAE does not incorporate the generated samples for learning, the latent features generated by VAE are largely randomized and uncontrollable (Hu et al., 2017, 2018). Therefore, we introduce a wake-sleep procedure (Hinton et al., 1995) that uses both real and generated data for joint training to improve the model representation and abstraction capability. Furthermore, base on the proposed framework, we can easily extend our method to solve the TZSL task. We use the generated semantics as pseudo labels for unseen images and regard the problem as a label noise circumstance. Then a robust loss is introduced to solve this label noise problem. Finally, with the generated latent features, we can solve both the IZSL and TZSL as a supervised classification problem.

The contributions of this paper are as follows.

  • We propose a novel ZSL method based on an information bottleneck (IB) constrained generative model with a tight visual-semantic bidirectional embedding. The IB loss minimizes the non-semantic information when embedding the visual domain to latent space. To the best of our knowledge, this is the first work that adopts the IB theory in ZSL.

  • We exploit the data uncertainty estimation technique for the first time in ZSL to learn the bias of visual distribution and design a bias passing mechanism, which alleviates the noises and gap between visual features and human annotated semantics.

  • We train the model on both real and generated data with a wake-sleep training mechanism to improve the model representation and abstraction capability via a VAE model.

  • We further extend our method to adapt the transductive ZSL setting with a robust label noise loss.

Extensive experimental results on four widely used ZSL benchmarks for both generalized ZSL and conventional ZSL show the superiority of our method under both inductive ZSL and transductive ZSL settings compared with state-of-the-art ZSL methods.

The rest of this paper is organized as follows: We first review the related works on two different ZSL settings in Sect. 2. In Sect. 3, we describe the problem setting and our proposed method. The extended method for TZSL is introduced at the end of Sect. 3. Section 4 shows the experimental results on four benchmark datasets under different ZSL settings. Finally, we conclude our method in Sect. 5.

2 Related works

In this section, we review related works on IZSL and TZSL. For the IZSL, we divide the existing methods into two categories, embedding models and generative models.

2.1 Embedding models for IZSL

Embedding models for ZSL focus on learning a direct or indirect mapping between visual and semantic spaces to transfer semantic knowledge from seen classes to unseen classes. There are three typical embedding strategies. The earliest methods learned the mapping function from the visual space to the semantic space, which include, for example DAP and IAP (Lampert et al., 2013), ALE (Akata et al., 2015a), DeViSE (Frome et al., 2013) and ESZSL (Romera-Paredes & Torr, 2015). To alleviate the severe hubness problem caused by embedding from the high-dimensional visual space to the low-dimensional semantic space, reverse mapping from the semantic space to the visual space was proposed for the nearest neighbor classification in the visual space (Changpinyo et al., 2017; Zhang et al., 2017). Some models such as SSE (Zhang et al., 2015), SYNC (Changpinyo et al., 2016) and BiDiLEL (Wang & Chen, 2017) explore the idea of embedding both visual and semantic features into a common intermediate space. Though these methods perform well in the conventional ZSL setting, their performance deteriorates on the GZSL setting since there are only seen class features for model training.

2.2 Generative models for IZSL

Recently, abundant generative models (Guo et al., 2017; Chen et al., 2018; Felix et al., 2018; Kumar Verma et al., 2018; Xian et al., 2018b; Zhu et al., 2018; Li et al., 2019a; Schonfeld et al., 2019; Ma & Hu, 2020; Keshari et al., 2020) were proposed to address the training data imbalance problem between seen and unseen classes by synthesizing unseen class features. Among these, both generative adversarial networks (GAN) (Chen et al., 2018; Felix et al., 2018; Zhu et al., 2018; Li et al., 2019a), and variational autoencoders (VAE) (Bucher et al., 2017; Kumar Verma et al., 2018; Schonfeld et al., 2019; Ma & Hu, 2020; Keshari et al., 2020) have been used for ZSL. f-CLSWGAN (Xian et al., 2018b) adapts the Wasserstein GAN (WGAN) (Arjovsky et al., 2017; Gulrajani et al., 2017) by adding a classification loss to enforce the generator to synthesize features that are suited for ZSL. Motivated by the cycle consistency loss (Zhu et al., 2017), cycle-CLSWGAN (Felix et al., 2018) utilizes a multi-modal cycle consistency loss to enforce that the generated visual features map back to their original semantic features, which can generate more robust unseen samples. LisGAN (Li et al., 2019a) exploits conditional WGAN to generate fake unseen classes from random noises and introduces soul samples regularizations to guarantee the generated sample is close to real. Due to the hardness of training GAN based models, CADA-VAE (Schonfeld et al., 2019) adopts a cross-aligned VAE to align the visual and semantic distributions in a latent space. More recently, a new flow-based generative model (Shen et al., 2020) was introduced to ZSL which utilizes an invertible generative flow network to generate distinguishable samples.

Although these generative models have achieved encouraging performance for GZSL, feature generation for unseen classes still needs tight visual-semantic coupling constraints to alleviate the deviation. Our proposed method combines the advantages of both embedding and generative models for an accurate alignment of visual-semantic distributions while generating discriminative image features.

2.3 Transductive ZSL

Different from the IZSL, the TZSL assumes the availability of unlabeled target unseen images (Fu et al., 2015) during training. But the relationship between the unseen image and label is still unavailable. With unseen images, the distribution discrepancy between seen and unseen domains can be exploited to alleviate the domain shift problem. Therefore, the TZSL methods usually achieve better performance than the IZSL setting. QFSL (Song et al., 2018) alleviates the bias problem with a proposed Quasi-Fully Supervised Learning framework. GXE (Li et al., 2019b) utilizes semantic attributes to train a visual feature classifier and calibrates the classifier with unlabeled data. SABR (Paul et al., 2019) learns two different GANs to generate the latent space features for seen and unseen classes, respectively. Recently, SDGN (Wu et al., 2020) integrates a self-supervised learning mechanism into the feature generating model to effectively exploit the unlabeled data and uses the self-supervision lurking in the data structure of different domains to conduct cross-domain mining.

Our method can be easily extended to adapt the TZSL setting. The latent space we learned can effectively eliminate the possible noise in visual and semantic features. We select unseen data with higher confidence and mark these data with pseudo labels. Unlike training the seen classes, the labels of the unseen classes are noisy, hence we introduce a robust loss for the label noise problem during the unseen classes training.

3 Proposed method

In this section, we first define the problem setting, notations and then present the details of each module of our method. Finally, we extend our method to adapt the TZSL setting with a robust loss for the noisy label.

3.1 Problem setting and notations

The GZSL problem is defined as follows. Let \({\mathcal {X}}^S\) and \({\mathcal {X}}^U\) denote the image feature sets of seen classes and unseen classes respectively, \({\mathcal {X}}= {\mathcal {X}}^S \cup {\mathcal {X}}^U\). \({\mathcal {S}}=\{({\mathbf {x}},y,{\mathbf {c}}(y))|{\mathbf {x}}\in {\mathcal {X}}^S,y\in {\mathcal {Y}}^S,{\mathbf {c}}(y)\in {\mathcal {C}}^S\}\) denotes the training set, where \({\mathbf {x}}\in {\mathbb {R}}^D\) are image features extracted by a plain CNN model. y are the seen class labels which are available during training and \({\mathbf {c}}(y)\in {\mathbb {R}}^K\) are attribute features. The auxiliary training set is \({\mathcal {U}}=\{(u,{\mathbf {c}}(u))|u \in {\mathcal {Y}}^U,{\mathbf {c}}(u)\in {\mathcal {C}}^U\}\), where u denote unseen class labels. The seen classes and unseen classes are disjoint, i.e., \({\mathcal {Y}}^S\cap {\mathcal {Y}}^U=\emptyset \). Here, \({\mathcal {C}}={\mathcal {C}}^S\cup {\mathcal {C}}^U\) is used to transfer information between seen classes and unseen classes. \({\mathcal {C}}\) can be human-annotated attributes (Xian et al., 2018b) or articles describing the classes (Zhu et al., 2018). In the conventional ZSL, the task is to learn a classifier \(f_{ZSL} : {\mathcal {X}}^U \rightarrow {\mathcal {Y}}^U\). However, in more realistic and challenging setup of GZSL, the aim is to learn a classifier \(f_{GZSL} : {\mathcal {X}} \rightarrow {\mathcal {Y}}^U\cup {\mathcal {Y}}^S\).

The architecture of the proposed model is shown in Fig. 2. It consists of two sets of latent embedding VAEs with \((E_{v\rightarrow l}, D_{l\rightarrow a})\) and \((E_{a\rightarrow l}, D_{l\rightarrow v})\). These two sets of VAEs share the same latent space l. \(E_{v\rightarrow l}\) maps visual space v to latent space l, and \(D_{l\rightarrow a}\) maps latent space l to semantic space a. \(E_{a\rightarrow l}\) maps semantic space a to latent space l, and \(D_{l\rightarrow v}\) maps latent space l to visual space v. The visual-semantic distributions are aligned in the latent space l by KL-Divergence and Center-Calibration. Information bottleneck loss and uncertainty estimation are used to facilitate the distribution alignment. A wake-sleep procedure is exploited to improve the quality of generated features by VAE decoder. Following we give the detailed descriptions of each module.

3.2 Latent bidirectional embedding with uncertainty estimation

The goal of our model is to learn a latent space that can accurately align visual and semantic distributions. We first learn a Visual to Semantic (VS) network \(\text{ VS }=E_{v\rightarrow l}\circ D_{l\rightarrow a}: {\mathbb {R}}^D\rightarrow {\mathbb {R}}^K\) to project the visual features through latent space into semantic space. The latent embedding model is shown in Fig. 3. Because there may be inherent noise in the visual features (Chang et al., 2020). To reduce the impact of data uncertainty, we define the latent representation \({\mathbf {z}}_i^{(v)}\) embedded from each visual sample \({\mathbf {x}}_i\) as a Gaussian distribution:

$$\begin{aligned} p({\mathbf {z}}_i^{(v)}|{\mathbf {x}}_i)={\mathcal {N}}({\mathbf {z}}_i^{(v)} ;\varvec{\mu }_i^{(v)},\varvec{\sigma }_i^{2}{\mathbf {I}}), \end{aligned}$$
(1)

where \(\varvec{\mu }_i^{(v)}\) and \(\varvec{\sigma }_i^2\) are the mean and variance of the Gaussian distribution learned by the encoder \(E_{v\rightarrow l}: \varvec{\mu }_i^{(v)}=E_{v\rightarrow l,\varvec{\phi }_1}({\mathbf {x}}_i), \log \varvec{\sigma }_i^2=E_{v\rightarrow l,\varvec{\phi }_2}({\mathbf {x}}_i)\), where \(\varvec{\phi }_1\) and \(\varvec{\phi }_2\) refer to the model parameters. The re-parameterization trick (Kingma & Welling, 2013) is used to keep gradients of the model as usual. With this trick, we generate the latent sampling representation \({\mathbf {s}}_i^{(v)}\) as

$$\begin{aligned} {\mathbf {s}}_i^{(v)}=\varvec{\mu }_i^{(v)}+\varvec{\epsilon }\varvec{\sigma }_i,\ \varvec{\epsilon }\sim {\mathcal {N}}({\mathbf {0}},{\mathbf {I}}), \end{aligned}$$
(2)

where \(\varvec{\epsilon }\) is a random noise.

Fig. 3
figure 3

Illustration of the latent bidirectional embedding and data uncertainty estimation. Left: the visual to semantic network. Here we define the latent representation as a Gaussian distribution to learn the data uncertainty. Right: the semantic to visual network

Full size image

Then, \(\widetilde{{\mathbf {c}}}(y_i)=D_{l\rightarrow a}({\mathbf {s}}_i^{(v)})\) projects the latent feature \({\mathbf {s}}_i^{(v)}\) into semantic space, i.e., the mapping of a visual sample \({\mathbf {x}}_i\) is calculated as \(\text{ VS }({\mathbf {x}}_i)\):

$$\begin{aligned} \text{ VS }({\mathbf {x}}_i)=\widetilde{{\mathbf {c}}}(y_i)=D_{l\rightarrow a}\left( {\mathbf {s}}_i^{(v)}\right) =E_{v\rightarrow l}\circ D_{l\rightarrow a}({\mathbf {x}}_i). \end{aligned}$$
(3)

The affinity between \(\text{ VS }({\mathbf {x}}_i)\) and the \(y_i\)-th attribute feature \({\mathbf {c}}(y_i)\) could be measured by their inner product \(\text{ VS }({\mathbf {x}}_i)^T{\mathbf {c}}(y_i)\). Then the probability of \({\mathbf {x}}_i\) belong to the \(y_i\)-th category in semantic space can be calculated as:

$$\begin{aligned} p^A(y_i|{\mathbf {x}}_i) = \frac{\exp \left( \text{ VS }({\mathbf {x}}_i)^T{\mathbf {c}}(y_i)\right) }{\sum \nolimits _{y\in {\mathcal {Y}}^S}\exp \left( \text{ VS }({\mathbf {x}}_i)^T{\mathbf {c}}(y)\right) }. \end{aligned}$$
(4)

Then the semantic cross-entropy (SCE) loss can be written as:

$$\begin{aligned} {\mathcal {L}}_{SCE}=-\sum \limits _{i}\log p^A(y_i|{\mathbf {x}}_i). \end{aligned}$$
(5)

Similarly, we learn a semantic to visual (SV) network \(\text{ SV }=E_{a\rightarrow l}\circ D_{l\rightarrow v}: {\mathbb {R}}^K\rightarrow {\mathbb {R}}^D\), which first projects the semantic feature \({\mathbf {c}}(y_i)\) to latent space as \(\varvec{\mu }_i^{(a)}\), then projects \(\varvec{\mu }_i^{(a)}\) to visual space as the generated visual prototype \(\widetilde{{\mathbf {x}}}(y_i)\) for the \(y_i\)-th category:

$$\begin{aligned} \text{ SV }({\mathbf {c}}(y_i))=\widetilde{{\mathbf {x}}}(y_i)=D_{l\rightarrow v}\left( \varvec{\mu }_i^{(a)}\right) =E_{a\rightarrow l}\circ D_{l\rightarrow v}({\mathbf {c}}(y_i)). \end{aligned}$$
(6)

The probability of \({\mathbf {x}}_i\) belong to the \(y_i\)-th category in visual space is calculated as:

$$\begin{aligned} p^V(y_i|{\mathbf {x}}_i) = \frac{\exp \left( {\mathbf {x}}_i^T\text{ SV }({\mathbf {c}}(y_i))\right) }{\sum \nolimits _{y\in {\mathcal {Y}}^S}\exp \left( {\mathbf {x}}_i^T\text{ SV }({\mathbf {c}}(y))\right) }. \end{aligned}$$
(7)

Then the visual cross-entropy (VCE) loss is:

$$\begin{aligned} {\mathcal {L}}_{VCE}=-\sum \limits _{i}\log p^V(y_i|{\mathbf {x}}_i). \end{aligned}$$
(8)

The total cross-entropy (CE) loss is as follow:

$$\begin{aligned} {\mathcal {L}}_{CE}={\mathcal {L}}_{SCE}+{\mathcal {L}}_{VCE}. \end{aligned}$$
(9)

In order to learn an accurate latent bidirectional embedding, we perform center calibration for each category. Such a structured objective requires the center embedding of \({\mathbf {x}}_i\) being closer to the latent embedding of its groundtruth \({\mathbf {c}}(y_i)\) than other classes, the Center Calibration (CC) is defined as:

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{CC}=\sum \limits _{i,y}[\varDelta -d(E_{v\rightarrow l,\phi _1}({\mathbf {x}}_i),E_{a\rightarrow l}({\mathbf {c}}(y))) +d(E_{v\rightarrow l,\phi _1}({\mathbf {x}}_i),E_{a\rightarrow l}({\mathbf {c}}(y_i)))]_{+}, \end{aligned} \end{aligned}$$
(10)

where \(d(\cdot ,\cdot )\) denotes a certain distance metric. Here, we utilize the Euclidean distance in the experiments. \(\varDelta > 0\) is a margin to make \({\mathcal {L}}_{CC}\) more robust.

3.3 Feature generation with noise supervision

For each category y, there could be many visual samples \({\mathbf {x}}\), but the semantic description \({\mathbf {c}}\) of each category is unique. Thus, this unique semantic attribute \({\mathbf {c}}\) is insufficient to fully describe the variety of visual samples. Therefore, we assume the latent semantic distribution similar to the Gaussian distribution of latent visual features in Eq. (1). To adapt to this task, we use two sets of encoder-decoder structures. \(E_{v\rightarrow l}\) encodes the visual features \({\mathbf {x}}_i\) to a Gaussian distribution \({\mathcal {N}}(\varvec{\mu }_i^{(v)}, \varvec{\sigma }_i^{2})\) in the latent space, and \(E_{a\rightarrow l}\) encodes the semantic feature \({\mathbf {c}}(y_i)\) to the center \(\varvec{\mu }_i^{(a)}\) of category \(y_i\). Since the latent semantic Gaussian distribution \({\mathcal {N}}(\varvec{\mu }_i^{(a)}, \varvec{\sigma }_i^{2})\) should be consistent with the latent visual distribution, we design a bias passing mechanism to share the noise (bias) from the visual diversity for the latent semantic distribution. Then we use the decoders \(D_{l\rightarrow a}\) to decode \(\varvec{\mu }_i^{(v)}\) or \(\varvec{\mu }_i^{(a)}\) to semantic feature \(\widetilde{{\mathbf {c}}}(y_i)\), and use \(D_{l\rightarrow v}\) to decode \({\mathbf {s}}_i^{(v)}\sim {\mathcal {N}}(\varvec{\mu }_i^{(v)}, \varvec{\sigma }_i^{2})\) or \({\mathbf {s}}_i^{(a)}\sim {\mathcal {N}}(\varvec{\mu }_i^{(a)}, \varvec{\sigma }_i^{2})\) to visual feature \(\widetilde{{\mathbf {x}}}_i\). Finally, the loss with noise supervision for the modified VAE can be written as:

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{VAE}&={\mathbb {E}}_{q_\phi ({\mathbf {s}}^{(v)}|{\mathbf {x}})}\left[ \log p_{\theta _1}({\mathbf {c}}|{\mathbf {s}}^{(v)})\right] +{\mathbb {E}}_{q_\phi ({\mathbf {s}}^{(a)}|{\mathbf {c}})}\left[ \log p_{\theta _1}({\mathbf {c}}|{\mathbf {s}}^{(a)})\right] \\&\quad + {\mathbb {E}}_{q_\phi ({\mathbf {s}}^{(v)}|{\mathbf {x}})}\left[ \log p_{\theta _2}({\mathbf {c}}|{\mathbf {s}}^{(v)})\right] +{\mathbb {E}}_{q_\phi ({\mathbf {s}}^{(a)}|{\mathbf {c}})}\left[ \log p_{\theta _2}({\mathbf {c}}|{\mathbf {s}}^{(a)})\right] \\&\quad -\beta D_{KL}\left( q_\phi ({\mathbf {s}}^{(v)}|{\mathbf {x}})||{\mathcal {N}}(\varvec{\mu }^{(a)}, {\mathbf {I}})\right) , \end{aligned} \end{aligned}$$
(11)

where \(\varvec{\phi }\) refers to the parameters of \(E_{v\rightarrow l}\) and \(E_{a\rightarrow l}\), \(\varvec{\theta }_1\) and \(\varvec{\theta }_2\) refer to the parameters of \(D_{l\rightarrow a}\) and \(D_{l\rightarrow v}\), respectively. The hyperparameter \(\beta \) is set following CADA-VAE (Schonfeld et al., 2019).

3.4 Information bottleneck constraint

In our method, information is gradually disentangled from the visual space through the latent space to the semantic space. The semantic feature \({\mathbf {c}}\) is related disentangled attribute information while the visual feature \({\mathbf {x}}\) has high-dimensional entangled non-semantic information. Therefore, we hope that the latent feature \({\mathbf {s}}\) should contain as much semantic information of \({\mathbf {c}}\) as possible while discarding the redundant non-semantic information of \({\mathbf {x}}\). In information theory, the dependence between two random variables could be measured by mutual information \(I(\cdot ;\cdot )\). As illustrated in Fig. 1, we maximize the mutual information between the semantic space and the latent space (\(I({\mathbf {s}}; {\mathbf {c}})\)) and minimize the mutual information between the visual space and the latent space (\(I({\mathbf {s}}; {\mathbf {x}})\)). We define the information bottleneck (IB) (Tishby et al., 2000) to constrain the information relationship between spaces:

$$\begin{aligned} \max I({\mathbf {s}}; {\mathbf {c}})-\eta I({\mathbf {s}}; {\mathbf {x}}). \end{aligned}$$
(12)

Since \({\mathbf {s}}\) may be sampled from different distributions like \({\mathcal {N}}(\varvec{\mu }^{(v)}, \varvec{\sigma }^{2})\) or \({\mathcal {N}}(\varvec{\mu }^{(a)}, \varvec{\sigma }^{2})\) in our model, we resample \({\mathbf {s}}^*\sim {\mathcal {N}}(\varvec{\mu }^*, \varvec{\sigma }^{2})\), where \(\varvec{\mu }^*=\alpha \varvec{\mu }^{(v)}+(1-\alpha )\varvec{\mu }^{(a)}\) with a uniform distribution \(\alpha \sim U(0,1)\).

Fig. 4
figure 4

The wake-sleep procedure. The features represented as dotted lines are generated. In the wake phase, we use real visual data \({\mathbf {x}}\) to train the model representation capability. In the sleep phase, the generated visual data \(\widetilde{{\mathbf {x}}}\) is used to train the model abstraction capability

Full size image

Since the VAE model does not utilize the generated samples for training, the latent features generated are largely randomized and uncontrollable. Inspired by the work of Hu et al. (2018), we train the modified VAE model in a wake-sleep procedure, using real data and generated data for joint training. The extended wake-sleep procedure is shown in Fig. 4. In the wake phase, we use real visual data \({\mathbf {x}}\) to train the feature representation capability of the model. In the sleep phase, we use generated data \(\widetilde{{\mathbf {x}}}\) to train the abstraction capability of the model. So that the model can generate disentangled latent features. The wake-sleep information bottleneck constraint is as follow:

$$\begin{aligned} \max \big [I({\mathbf {s}}^*;{\mathbf {c}})-\eta I({\mathbf {s}}^*;{\mathbf {x}})\big ]+\lambda \big [I(\widetilde{{\mathbf {s}}}^*;{\mathbf {c}})-\eta I(\widetilde{{\mathbf {s}}}^*;\widetilde{{\mathbf {x}}})\big ], \end{aligned}$$
(13)

where \(\widetilde{{\mathbf {s}}}^*\) is the latent embedding representation of \(\widetilde{{\mathbf {x}}}\). The weighting factor \(\lambda \) is obtained by grid search on the validation set.

Since the information bottleneck with high dimension is intractable to calculate, we follow the strategy proposed by Alemi et al. (2016). The information bottleneck (IB) loss is shown as follow:

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{IB} = \frac{1}{N}\sum _{i=1}^N{\mathbb {E}}[-\log q({\mathbf {c}}(y_i)|D_{l\rightarrow a}({\mathbf {s}}_i^*))] +\eta D_{KL}[q_\phi ({\mathbf {s}}^*|{\mathbf {x}}_i,{\mathbf {c}}(y_i))||r({\mathbf {z}})], \end{aligned} \end{aligned}$$
(14)

where \(r({\mathbf {z}})\) is a standard normal distribution in the experiments. \(\eta \) is initialized to \(10^{-5}\) and changed with the IB loss.

Finally, the overall loss of the proposed model is defined as:

$$\begin{aligned} {\mathcal {L}}={\mathcal {L}}_{VAE}+\gamma {\mathcal {L}}_{CE}+\delta {\mathcal {L}}_{CC}+ \tau {\mathcal {L}}_{IB}, \end{aligned}$$
(15)

where \(\gamma \), \(\delta \), and \(\tau \) are the weighting factors of the cross entropy loss, center calibration, and information bottleneck loss, respectively. We empirically choose their values to balance the effect of different loss terms in the experiments.

For more intuitive understanding, we summarize our proposed method in Algorithm 1. When the encoder \(E_{v\rightarrow l}\) is finished training, we utilize this encoder to learn latent representations of both the seen and unseen image features. Then the latent representations are used for classification.

figure a

3.5 Transductive ZSL

Our proposed model can be easily extended to the TZSL setting. We adopt the self-training manner to exploit the unseen images. We first generate the pseudo labels \(\widetilde{{\mathcal {Y}}}^U\) for unseen image features \({\mathcal {X}}^U\) through calculating classification score \({{\textbf {O}}}\) of the learned classifier in the IZSL stage. There will inevitably be noise in \(\widetilde{{\mathcal {Y}}}^U\). We use classification score peakiness based filtering strategy (Li et al., 2019b) to mitigate the influence of noisy labels. The classification score of \({{\textbf {x}}}^u_i \in {\mathcal {X}}^U\) is \({{\textbf {o}}}^i \in {\mathbb {R}}^{N^U}\) according to all the \(N^U\) classes. The sum of each dimension of \({{\textbf {o}}}^i\) is 1. Let \({{\textbf {o}}}^i\) be the soft label of \({{\textbf {x}}}^u_i\). We assume the maximum and second maximum score of \({{\textbf {o}}}^i\) are \({{\textbf {o}}}^i_{u_m}\) and \({{\textbf {o}}}^i_{u_n}\). The pseudo label of \({{\textbf {x}}}^u_i\) should be \(u_m\). If an unseen sample satisfies \(\dfrac{{{\textbf {o}}}^i_{u_m}}{{{\textbf {o}}}^i_{u_n}}\>r\), we assign soft label and pseudo label to it. r is the threshold, we set \(r=1.4\) in our experiment. Then we have a high confident training set \(\widetilde{{\mathcal {U}}}=\{({{\textbf {x}}}^u, {\widetilde{u}},{\mathbf {c}}({\widetilde{u}}),{{\textbf {o}}})|{{\textbf {x}}}^u \in {\mathcal {X}}^U, {\widetilde{u}} \in {\mathcal {Y}}^U,{\mathbf {c}}({\widetilde{u}})\in {\mathcal {C}}^U, {{\textbf {o}}} \in {{\textbf {O}}}\}\).

For the seen classes, we still utilize the proposed method for the IZSL to train the model with training set \({\mathcal {S}}\). The difference is that we use the cross entropy loss to replace center calibration of Eq. (10). Simultaneously, we exploit the training set \(\widetilde{{\mathcal {U}}}\) to calibrate the model with unseen classes. Unlike Eqs. (5) and (8), we use the soft label and soft cross-entropy loss to replace hard label and cross-entropy loss in the visual and semantic space. In the latent space, the probability \({{\textbf {x}}}^u_i\) belong to the \({\widetilde{u}}_i\)-th category is calculated as:

$$\begin{aligned} p({\widetilde{u}}_i|{{\textbf {x}}}^u_i) = \dfrac{\exp \left( E_{v\rightarrow l,\phi _1}({{\textbf {x}}}^u_i)^TE_{a\rightarrow l}({\mathbf {c}}({\widetilde{u}}_i))\right) }{\sum \nolimits _{u\in {\mathcal {Y}}^U}\exp \left( E_{v\rightarrow l,\phi _1}({{\textbf {x}}}^u_i)^TE_{a\rightarrow l}({\mathbf {c}}(u))\right) }. \end{aligned}$$
(16)

Then, we introduce the generalized cross-entropy (GCE) loss (Zhang & Sabuncu, 2018) to alleviate the influence of noisy pseudo labels:

$$\begin{aligned} {\mathcal {L}}_{GCE} = \sum \limits _{i}\dfrac{1-p\big ({\widetilde{u}}_i|{{\textbf {x}}}^u_i\big )^q}{q}, \end{aligned}$$
(17)

where \(q \in (0,1]\) is a hyper-parameter of which a higher value is preferred when the noise level is high. The GCE loss is a generalization of the categorical cross-entropy (CCE) loss and the mean absolute error (MAE). It is equivalent to the CCE loss when q infinitely approaches 0 and turns to MAE loss when \(q=1\). We set \(q=0.4\) in our experiment. The CCE loss is powerful for classification tasks but will be overfitting on the label noise dataset. The MAE loss is worse for label clean classification task but is robust to noisy labels. The hyper-parameter q can be tuned between 0 and 1 to fit different noise levels. In the TZSL experiments, we have also tried the other label-noise learning methods T-Revision (Xia et al., 2019) and SIGUA (Han et al., 2020a) to combat noisy labels. Results show that these different label-noise learning methods achieve similar performance. This may be because that the soft label assignment procedure is more important for the transductive ZSL. Therefore, we choose the compact GCE loss to handle the label-noise problem in our method.

4 Experiments

In this section, we first give the experimental settings. Then the comparison with state-of-the-art methods and ablation studies are conducted to demonstrate the effectiveness of our method. Finally, more analyzing experiments show the superiority of our method.

4.1 Experimental settings

Datasets We evaluated our framework on four widely used benchmark datasets including CUB-200-2011 (CUB) (Welinder et al., 2010), SUN attribute (SUN) (Patterson & Hays, 2012), attributes Pascal and Yahoo (aPY) (Farhadi et al., 2009) and Animals with Attributes 2 (AwA2) (Xian et al., 2018a) for the GZSL. We extracted a 2048-dimensional CNN features for images using ResNet-101 (He et al., 2016) as the visual features. The pre-defined attributes on each dataset were used as the semantic descriptors. Moreover, we adopted the proposed split (PS) (Xian et al., 2018a) to divide all classes into seen and unseen classes on each dataset. The dataset details are listed in Table 1.

Table 1 The details for SUN, CUB, AwA2 and APY. \({\mathcal {Y}}^S\) and \({\mathcal {Y}}^U\) are seen class number and unseen class number
Full size table

Implementation details In our modified VAE model, we utilized multilayer perceptrons to implement the encoders (\(E_{v\rightarrow l}\) and \(E_{a\rightarrow l}\)) and decoders (\(D_{l\rightarrow a}\) and \(D_{l\rightarrow v}\)). The encoders \(E_{v\rightarrow l}\) and \(E_{a\rightarrow l}\) had 1560 and 1450 hidden units, respectively. The hidden units of \(D_{l\rightarrow a}\) and \(D_{l\rightarrow v}\) were 665 and 1660, respectively. The latent embedding dimensions were 64 for AwA2 and aPY and 256 for CUB and SUN. \(\beta \), \(\lambda \), \(\gamma \), \(\delta \) and \(\tau \) were set to 0.5, 0.1, 1.0, 0.1 and 1.0. The margin \(\varDelta \) was set to \(10^{-3}\). Adam optimizer (Kingma & Ba, 2014) was used for training, the epoch size was 120 and the batch size was 64. After the model training, the encoders \(E_{v\rightarrow l}\) and \(E_{a\rightarrow l}\) transformed the visual features of seen classes and attribute features of unseen classes into the unified latent space. Finally, we trained a softmax classifier to classify latent features.

Evaluation metrics The performance of our method is evaluated by per-class Top-1 accuracy. In conventional ZSL, the Top-1 accuracy is evaluated on seen classes, denoted as T1. In GZSL, since the test set is composed of seen and unseen images, the Top-1 accuracy is evaluated respectively on seen classes, denoted as S, and unseen classes, denoted as U. Their harmonic mean (H) (Xian et al., 2018a) is used to evaluate the performance of GZSL, which is defined as:

$$\begin{aligned} {{\textbf {H}}} = \dfrac{2 \times {{\textbf {S}}} \times {{\textbf {U}}}}{{{\textbf {S}}} + {{\textbf {U}}}}. \end{aligned}$$
(18)
Table 2 Results of the state-of-the-arts generalized zero-shot learning for inductive setting
Full size table
Table 3 Results of conventional zero-shot learning for inductive setting
Full size table
Table 4 Results of the generalized zero-shot learning for transductive setting
Full size table
Table 5 Results of conventional zero-shot learning for transductive setting
Full size table

4.2 Comparison with baseline methods

Table 6 Ablation study of the proposed modules
Full size table
Table 7 Ablation study of wake-sleep IB constraint on CUB dataset
Full size table

Inductive ZSL We selected several state-of-the-art GZSL methods for comparison, which include non-feature generation methods ALE (Akata et al., 2013), DeViSE (Frome et al., 2013), ESZSL (Romera-Paredes & Torr, 2015), SJE (Akata et al., 2015b), LATEM (Xian et al., 2016), SYNC (Changpinyo et al., 2016), SAE (Kodirov et al., 2017), SP-AEN (Chen et al., 2018), TCN (Jiang et al., 2019), TripletLoss (Cacheux et al., 2019) and feature generation based methods SE-GZSL (Kumar Verma et al., 2018), CVAE-ZSL (Mishra et al., 2018), f-CLSWGAN (Xian et al., 2018b), LisGAN (Li et al., 2019a), GDAN (Huang et al., 2019) CADA-VAE (Schonfeld et al., 2019), ABP (Zhu et al., 2019), OCD-CVAE (Keshari et al., 2020), LsrGAN (Vyas et al., 2020). Table 2 shows the results of different methods on four datasets. It can be seen that our proposed method outperforms all the ten compared non-feature generation methods with a large margin for the harmonic mean results. Moreover, our method significantly improves the Top-1 accuracy on unseen classes benefited from the generated unseen class samples. Compared with the feature generation based methods, our method can also achieve the best harmonic mean results on CUB, AwA2, and aPY. Since the IB constrained bidirectional embedding between the visual space and the semantic space can preserve essential attribute information and discard the non-semantic information. To further demonstrate the effectiveness of our method. We also compared our method under the conventional ZSL setting that the test image only belongs to unseen classes. As shown in Table 3, our proposed method achieves the best for three out of the four datasets.

Transductive ZSL Under the TZSL setting, we mitigated the deviation of the model on seen classes by utilizing the data of unseen classes during training. We compared our method with recent state-of-the-art TZSL methods, which include ALE-trans (Akata et al., 2015b), GFZSL-trans (Verma et al., 2017), QFSL (Song et al., 2018), GXE (Li et al., 2019b), GMN (Sariyildiz & Cinbis, 2019), f-VAEGAN (Xian et al., 2019), WDVSc (Wan et al., 2019). Tables 4 and 5 show the results of different methods on CUB, AwA2, and SUN for GZSL and conventional ZSL, respectively. We can see that our method outperforms all the compared methods for both GZSL and conventional ZSL on the CUB dataset. For AwA2 and SUN datasets, our method achieves the second highest performance. This validates that our method can be readily adapted to the TZSL setting.

4.3 Further analyses for inductive setting

Ablation study We conducted ablation experiments to verify the effectiveness of the proposed modules. Table 6 shows the influence of different losses. We can see that our proposed method achieves the best harmonic mean results with all the losses. Specifically, the proposed IB loss can significantly improve the performance. For the proposed wake-sleep IB constraint, we also performed ablation study with different conditions on CUB, as shown in Table 7. It can be seen that the IB loss constrained on the generated seen classes features (Sleep(S)) has significantly improved the classification accuracy of the seen classes and conventional ZSL. Accordingly, the IB loss constrained on the generated unseen classes features (Sleep(U)) also improves the result of unseen classes. Our method achieves the highest harmonic mean result under the wake-sleep IB constraint. In addition, we use \({\mathcal {N}}(\varvec{\mu }^{(a)}, {\mathbf {I}})\) to replace the latent semantic distribution \({\mathcal {N}}(\varvec{\mu }^{(a)}, \varvec{\sigma }^{2})\) to verify the effectiveness of the proposed bias passing mechanism. The results on four datasets are CUB(H = 53.6), AwA2(H = 64.3), SUN(H = 39.5) and aPY(H = 45.5). It shows that the bias passing mechanism can alleviate the visual and semantic noises problem.

Fig. 5
figure 5

The influence of the dimension of latent features generated by our model. We measured the Top-1 accuracy on seen classes and unseen classes and the harmonic mean accuracy on CUB, AwA2, SUN, and aPY datasets

Full size image
Fig. 6
figure 6

The influence of different numbers of generated features per seen and unseen classes

Full size image
Fig. 7
figure 7

Visualization of the latent feature distributions. The top is the latent visual embedding features and the bottom is the latent semantic embedding features

Full size image
Fig. 8
figure 8

The influence of q and r on GZSL (H) and conventional ZSL (T1) results (\(\%\)) for transductive setting

Full size image

The influence of latent dimensions We first evaluated our method with different dimensions of latent features, as shown in Fig. 5. The harmonic mean results have less fluctuation with different latent feature dimensions on four datasets. Our method achieves the best performance on CUB and SUN with the latent feature dimensions equal to 256. The best results are reached for AwA2 and aPY when the latent feature dimension is 64. We speculate the reason is that on the one hand the CUB and SUN are fine-grained datasets which need more information to distinguish. On the other hand, excessive dimensions lead to redundant information.

The influence of latent features We show the influence of different numbers of generated features per seen and unseen classes on CUB. Figure 6 reports the harmonic mean results (H) of GZSL (a) and T1 results of conventional ZSL (b). We can see that the GZSL performance of our method increases with more generated unseen features in most cases and when the number of generated unseen features is twice the generated seen features, our method achieves the best result. The results of conventional ZSL show a similar conclusion with GZSL, which validates the soundness of our method.

Visualization Result We use the t-SNE (Maaten & Hinton, 2008) to visualize our latent features used for the final GZSL classification. Figure 7 shows the distributions of the latent features of 50 classes on the CUB dataset. The top is the latent visual embedding features and the bottom is the latent semantic embedding features. From the almost consistent distribution, we can see our latent features can well align visual and semantic distributions.

4.4 Further analyses for transductive setting

The influence of q and r Figure 8a shows the results of H and T1 for CUB dataset when varying hyper-parameter q of generalized cross-entropy loss from 0.1 to 0.9. Our method achieves the best performance when \(q=0.4\). GCE loss combines the advantages of CCE loss and MAE loss, which makes our model more robust in the classification task with noisy labels. The influence of r on the results of H and T1 for CUB dataset is shown in Fig. 8b. We varied the value of r from 1.0 to 3.0 in steps of 0.2. At the first, the accuracy of H and T1 increases with the change of r. Our model achieves the best result when \(r=1.4\). Then the performance of the model declines with the change of r. Since the quality of the noisy unseen data used for training becomes higher, the performance of the model is increasing. While the higher the quality of the unseen data, the less noise in the label, and the smaller the amount of training data. Therefore the performance of the model decreases.

Quality analysis of noisy data We further analyzed the influence of the threshold r on the generated pseudo labels for CUB, AwA2, and SUN. As shown in Table 8, Acc is the accuracy of the generated pseudo label with different thresholds r. It shows that a larger threshold r can make the confidence of the pseudo label higher. Num illustrates the relationship between the number of noisy data for training and threshold r. Although the confidence of pseudo labels has become higher, the number of samples used for training is decreasing. Max, Min and Avg respectively show the maximum, minimum and average number of training samples for different categories with different r. We can see that when r is large, the sample numbers for different categories will be unbalanced, especially the number of training samples in some categories will even become 0. Therefore, we set \(r=1.4\) to achieve a good tradeoff in sample quality, sample training number, and sample balance in the experiment.

Table 8 Quality analysis of noisy data. Acc is the accuracy of the pseudo label
Full size table

5 Conclusion

In this paper, we introduced a novel bidirectional embedding based generative model for zero-shot learning. This method learns a unified latent space to align the feature distributions of both visual domain and semantic domain. A novel information bottleneck (IB) constrained latent bidirectional embedding allows the latent features to contain more essential attributes related information while discarding non-semantic information flowed from the visual features. In addition, data uncertainty estimation and wake-sleep procedure are introduced to facilitate latent distributions alignment. The proposed method has outperformed several state-of-the-art methods in different ZSL settings in experimental comparison, showing the advantages of our approach. Furthermore, our method can be lightly extended to adapt the transductive ZSL task and also achieves competitive performance.

For the future works, further investigation on information bottleneck and uncertainty estimation theories for the cross-domain alignment problem is significant. For example, the other feature generation based ZSL models, such as GANs and generative flows, can also utilize information bottleneck and uncertainty estimation methods to generate more reliable unseen samples.