Towards Task Sampler Learning for Meta-Learning

Abstract

Meta-learning aims to learn general knowledge with diverse training tasks conducted from limited data, and then transfer it to new tasks. It is commonly believed that increasing task diversity will enhance the generalization ability of meta-learning models. However, this paper challenges this view through empirical and theoretical analysis. We obtain three conclusions: (i) there is no universal task sampling strategy that can guarantee the optimal performance of meta-learning models; (ii) over-constraining task diversity may incur the risk of under-fitting or over-fitting during training; and (iii) the generalization performance of meta-learning models are affected by task diversity, task entropy, and task difficulty. Based on this insight, we design a novel task sampler, called Adaptive Sampler (ASr). ASr is a plug-and-play module that can be integrated into any meta-learning framework. It dynamically adjusts task weights according to task diversity, task entropy, and task difficulty, thereby obtaining the optimal probability distribution for meta-training tasks. Finally, we conduct experiments on a series of benchmark datasets across various scenarios, and the results demonstrate that ASr has clear advantages. The code is publicly available at https://github.com/WangJingyao07/Adaptive-Sampler.

Appendices

Appendix A: Proofs

This appendix first provides the theoretical proofs of the theorems in Sect. 6. Next, we introduce the details and experimental settings of the meta-learning models.

In this section, we provide the proofs of Theorems 1, 2, and 3 in Appendix A.1, Appendix A.2, and Appendix A.3, respectively.

Notations Throughout this section, we use $Z_i$ to denote the representation of task $\mathcal {T}_i$, use ${\textbf {Z}}_i$ to denote the representation of the optimal $\mathcal {T}_i$, use $\mathbb {A}_+^n$, $\mathbb {R}_+$, and $\mathbb {Z}_+$ to denote the collection of $n \times n$ symmetric positive definite matrices, non-negative real numbers, and positive integers, respectively. The task $\mathcal {T}_i$ contains n samples and k classes, and class j contains $n_j$ samples. The dimension of representation $Z_i$ is d, $Z_i\in \mathbb {R}^d$.

1.1 A.1 Proof of Theorem 1

Theorem 1, also the Theorem 4 mentioned below, gives the upper bound of task diversity. The condition for the upper bound being tight is consistent with Maximally Feature Space in Corollary 1.

Theorem 4

Let $Z_i=\left[ Z_i^1,\ldots ,,Z_i^k \right] \in \mathbb {R}^{d\times n}$ be the representation of task $\mathcal {T}_i$, which has k classes and $n= {\textstyle \sum _{j=1}^{k}n_j}$ samples. For any representations $Z_i^j\in \mathbb {R}^{d \times n_j}$ of class j and any $\sigma >0$, we have:

$$\begin{aligned}&\frac{n}{2}\log \textrm{det}(\mathcal {I} +\frac{d}{n\sigma ^2} Z_i^*Z_i) \nonumber \\&\quad \le \sum _{j=1}^{k}\frac{n}{2}\log \textrm{det} \left( \mathcal {I} +\frac{d}{n\sigma ^2} {(Z_i^j)}^*{(Z_i^j)}\right) \end{aligned}$$

(14)

the equality holds if and only if:

$$\begin{aligned} {(Z_i^{j_1})}^*(Z_i^{j_2})=0,\quad s.t.\quad 1\le j_1 \le j_2 \le k \end{aligned}$$

(15)

Proof

According to Chan (2022), $\log \textrm{det} (\cdot ):\mathbb {A}_+^n\rightarrow \mathbb {R}$ is strictly concave. For any $\beta \in (0,1)$ and $\left\{ Z_{j_1},Z_{j_2} \right\} \in \mathbb {A}_+^n$:

$$\begin{aligned}{} & {} \log \textrm{det}((1-\beta )Z_{j_1}+\beta Z_{j_2}) \ge (1-\beta )\nonumber \\{} & {} \quad \log \textrm{det}(Z_{j_1})+\beta \log \textrm{det} (Z_{j_2}) \end{aligned}$$

(16)

with equality holds if and only if $Z_{j_1}=Z_{j_2}$. Then for all $\left\{ A_a, A_b \right\} \in \mathbb {A}_+^n$, we have:

$$\begin{aligned} \log (A_a)\le \log (A_b)+\left\langle \bigtriangledown \log (A_b),A_a-A_b \right\rangle \end{aligned}$$

(17)

According to Boyd and Vandenberghe (2004), let $A_b^{-1}=\bigtriangledown \log (A_b)$ and $A_b^{-1}=(A_b^{-1})^*$, we have:

$$\begin{aligned} \log (A_a)\le \log (A_b)+\textrm{tr}(A_b^{-1}A_a)-n \end{aligned}$$

(18)

we now let:

$$\begin{aligned} A_a&= \mathcal {I}+\frac{d}{n\sigma ^2}(Z_i)^*Z_i\nonumber \\&=\mathcal {I}+\frac{d}{n\sigma ^2}\begin{bmatrix} (Z_i^1)^*Z_i^1 &{} (Z_i^1)^*Z_i^2 &{} \dots &{} (Z_i^1)^*Z_i^k \\ (Z_i^2)^*Z_i^1 &{} (Z_i^2)^*Z_i^2 &{} \dots &{} (Z_i^2)^*Z_i^k \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ (Z_i^k)^*Z_i^1 &{} (Z_i^k)^*Z_i^2 &{} \dots &{} (Z_i^k)^*Z_i^k \end{bmatrix} \end{aligned}$$

(19)

From the property of determinant for block diagonal matrix (Lu et al., 2018), we let:

$$\begin{aligned} A_b&= \mathcal {I}+\frac{d}{n\sigma ^2}(Z_i^j)^*Z_i^j\nonumber \\&=\mathcal {I}+\frac{d}{n\sigma ^2}\begin{bmatrix} (Z_i^1)^*Z_i^1 &{} 0 &{} \dots &{} 0 \\ 0 &{} (Z_i^2)^*Z_i^2 &{} \dots &{} 0\\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ 0 &{} 0 &{} \dots &{} (Z_i^k)^*Z_i^k \end{bmatrix} \end{aligned}$$

(20)

Then, for $\textrm{tr}(A_b^{-1}A_a)$:

$$\begin{aligned} \textrm{tr}(A_b^{-1}A_a) = \textrm{tr}\begin{bmatrix} \mathcal {I} &{} \dots &{} \chi \\ \vdots &{} \ddots &{} \vdots \\ \chi &{} \dots &{} \mathcal {I} \end{bmatrix} =n \end{aligned}$$

(21)

bring Eqs. (19), (20), and (21) back to Eq. (18), we can get:

$$\begin{aligned}&\le \frac{n}{2}\log \textrm{det}(\mathcal {I} +\frac{d}{n\sigma ^2} Z_i^*Z_i) \nonumber \\&\le \sum _{j=1}^{k}\frac{n}{2}\log \textrm{det} \left( \mathcal {I} +\frac{d}{n\sigma ^2} {(Z_i^j)}^*{(Z_i^j)}\right) \end{aligned}$$

(22)

where the equality holds if and only if $A_a=A_b$, i.e., ${(Z_i^{j_1})}^*(Z_i^{j_2})=0,\quad s.t.\quad 1\le j_1 \le j_2 \le k$. $\square$

1.2 A.2 Proof of Theorem 2

Theorem 2, also the Theorem 5 mentioned below, gives that task entropy is maximized by the representations that are maximally discriminative between different classes and tight in each class. This is consistent with Maximally Discriminability in Corollary 1, demonstrating that task entropy can well reflect intra-class compaction and inter-class separability.

Theorem 5

Let $Z_i=\left[ Z_i^1,\ldots , Z_i^k \right]$ be the representation of task $\mathcal {T}_i$, $\varsigma _j:= \left[ \varsigma _{1,j},\ldots , \varsigma _{min(n_j,d),j} \right]$ be the singular values of the representation $Z_i^j$ of class j, $\textrm{C}_i=\left[ \textrm{C}_i^1,\ldots ,\textrm{C}_i^k \right]$ is a collection of diagonal matrices, where the diagonal elements encode the n samples into the k classes. Given any $\epsilon >0$ and $d \ge d_j>0$, consider the optimization problem of task entropy:

$$\begin{aligned}&\underset{Z_i\in \mathbb {R}^{d\times n} }{\arg \max } (t_{et}^i)\nonumber \\&s.t.\quad \left\| Z_i\textrm{C}_i^j \right\| ^2=\textrm{tr}(\textrm{C}_i^j ),\nonumber \\&\textrm{rank}(Z_i)\le d_j, \forall j\in \left\{ 1,\ldots ,k \right\} \end{aligned}$$

(23)

Under the conditions where the error upper limit $\epsilon ^4< \underset{j}{\min }\left\{ \frac{n_j}{n}\frac{d ^2}{d_j^2} \right\}$, and the dimension $d\ge {\textstyle \sum _{j=1}^{k}d_j}$, the optimal solution ${\textbf {Z}}_i$ satisfies:

• Between-class: The representation ${\textbf {Z}}_i^{j_1}$ and ${\textbf {Z}}_i^{j_2}$ lie in orthogonal subspaces, i.e., $({\textbf {Z}}_i^{j_1})^*({\textbf {Z}}_i^{j_2})=0$, where $1\le j_1 \le j_2 \le k$.

• Within-class: each class j achieves its maximal dimension $d_j$, i.e., $\textrm{rank}({\textbf {Z}}_i^j)= d_j$, and either $\left[ \varsigma _{1,j},\ldots , \varsigma _{d_j,j} \right]$ equal to $\frac{\textrm{tr}(\textrm{C}_i^j )}{d_j}$, or $\left[ \varsigma _{1,j},\ldots , \varsigma _{d_j-1,j} \right]$ equal to and have values larger than $\frac{\textrm{tr}(\textrm{C}_i^j )}{d_j}$.

Proof

Use singular value decomposition (SVD) to decompose $Z_i$ into $Z_i=U_i\Sigma _i V_i^*$, where $U_i$ and $V_i$ are unitary matrices, $\Sigma _i$ is a diagonal matrix, and its diagonal elements are singular values of $Z_i$. Since $\textrm{rank}(Z_i)\le d_j, \forall j\in \left\{ 1,\ldots ,k \right\}$, assume the first $d_j$ diagonal elements of $\Sigma _i$ is not zero, and the subsequent diagonal elements are all zero. Therefore, $\Sigma _i$ will be:

$$\begin{aligned} \Sigma _i=\left[ \begin{array}{cc} \Sigma _{i,1} &{} 0 \\ 0 &{} 0 \end{array}\right] \end{aligned}$$

(24)

where $\Sigma _{i,1}$ is a diagonal matrix of $d_j\times d_j$, and its diagonal elements are $\varsigma _{1,j},\ldots , \varsigma _{d_j,j}$. Similarly, for $U_i$ and $V_i$:

$$\begin{aligned} U_i=\left[ \begin{array}{cc} U_{i,1} &{} U_{i,2} \\ \end{array}\right] , \quad V_i=\left[ \begin{array}{cc} V_{i,1} &{} V_{i,2} \\ \end{array}\right] \end{aligned}$$

(25)

Among them, $U_{i,1}$ and $V_{i,1}$ are both matrices of $d\times d_j$, and $U_{i,2}$ and $V_{i,2}$ are both $The matrix of d\times (n-d_j)$. Then, we get:

$$\begin{aligned} Z_i&=U_i\Sigma _i V_i^* \nonumber \\&=\left[ \begin{array}{cc} U_{i,1} &{} U_{i,2} \\ \end{array}\right] \left[ \begin{array}{cc} \Sigma _{i,1} &{} 0 \\ 0 &{} 0 \end{array}\right] \left[ \begin{array}{c} V_{i,1}^* \\ V_{i,2}^* \end{array}\right] \nonumber \\&=U_{i,1}\Sigma _{i,1} V_{i,1}^* \end{aligned}$$

(26)

Since $\textrm{C}_i^j$ is a diagonal matrix, and only $n_j$ of its diagonal elements are 1, and the rest are 0, we have:

$$\begin{aligned} \textrm{tr}(\textrm{C}_i^j )=n_j,\quad Z_i\textrm{C}_i^j =U_{i,1}\Sigma _{i,1} V_{i,1}^ *\textrm{C}_i^j \end{aligned}$$

(27)

Therefore, the constraint $\left\| Z_i\textrm{C}_i^j \right\| ^2 =\textrm{tr}(\textrm{C}_i^j )$ can be equivalently written as:

$$\begin{aligned} \left\| U_{i,1}\Sigma _{i,1} V_{i,1}^*\textrm{C}_i^j\right\| ^2=n_j, \quad \forall j\in \left\{ 1,\ldots ,k \right\} \end{aligned}$$

(28)

Without loss of generality, let ${\textbf {Z}}_i=\left[ {\textbf {Z}}_i^1,\ldots , {\textbf {Z}}_i^k \right]$ is the feature representation of the optimal task $\mathcal {T}_i$. To show that ${\textbf {Z}}_i^j,j\in \left\{ 1,\ldots ,k \right\}$, are pairwise orthogonal, suppose for the purpose of arriving at a contradiction that $({\textbf {Z}}_i^{j_1})^*({\textbf {Z}}_i^{j_2})=0$ for some $1\le j_1 \le j_2 \le k$. That is:

$$\begin{aligned}&\underset{Z_i\in \mathbb {R}^{d\times n} }{\arg \max } (t_{et}^i)\nonumber \\&\quad \le \frac{\textrm{tr}(\textrm{C}_i^j ) }{2n})\log \det \left( \mathcal {I} +\frac{d}{\textrm{tr}(\textrm{C}_i^j ) \epsilon ^2}{{{\textbf {Z}}}_i^*} \textrm{C}_i^j{{{\textbf {Z}}}_i}\right) \nonumber \\&s.t.\quad \left\| {\textbf {Z}}_i\textrm{C}_i^j \right\| ^2 =\textrm{tr}(\textrm{C}_i^j ),\nonumber \\&\textrm{rank}({\textbf {Z}}_i)= d_j, \forall j\in \left\{ 1,\ldots ,k \right\} \end{aligned}$$

(29)

According to the proof of Theorem 4, the strict inequality in Between-class of Eq. (23) holds for the optimal solution ${\textbf {Z}}_i$. On the other hand, since $\sum _{j=1}^{k} d_j\le n$, there exists $\left\{ Q_i^j\in \mathbb {R}^{d\times d_j} \right\} _{j=1}^k$ such that the columns of the matrix $\mathcal {Q}$ are orthonormal.

Since $Z_i=U_i\Sigma _i V_i^*$, $\Sigma _i\Sigma _i^*$ is a diagonal matrix and its diagonal element is $\varsigma _{l,j}^2$, $\Sigma _{i,2}$ is a diagonal matrix of $d\times d$, and its diagonal element is $\varsigma _{l,j}^2$, we have:

$$\begin{aligned} Z_iZ_i^*&=U_i\Sigma _i\Sigma _i^* U_i^*\nonumber \\&=\left[ \begin{array}{cc} U_{i,1} &{} U_{i,2} \\ \end{array}\right] \left[ \begin{array}{cc} \Sigma _{i,2} &{} 0 \\ 0 &{} 0 \end{array}\right] \left[ \begin{array}{c} U_{i,1}^* \\ U_{i,2}^* \end{array}\right] \nonumber \\&=U_{i,1}\Sigma _{i,2} U_{i,1}^* \end{aligned}$$

(30)

where the rank of $Z_iZ_i^*$ is equal to the rank of $\Sigma _{i,2}$, that is, $\textrm{rank}(Z_iZ_i^*)=\textrm{rank}(\Sigma _{i,2})=d_j$. This means that only $d_j$ of the eigenvalues of $Z_iZ_i^*$ are non-zero, and the rest are zero. Since $Z_iZ_i^*$ is a symmetric matrix, we can diagonalize it as:

$$\begin{aligned} Z_iZ_i^*=Q_i\Lambda _i Q_i^* \end{aligned}$$

(31)

Since $(Z_i^{j_1})^* Z_i^{j_2}=V_{i,1}^{j_1*}\Sigma _{i,1}^{j_1*} U_{i,1}^{j_1* } U_{i,1}^{j_2}\Sigma _{i,1}^{j_2} V_{i,1}^{j_2}$, then:

$$\begin{aligned}&Z_i^{j_1}Z_i^{j_2*}\nonumber \\&\quad =Q_{i,1}^{j_1}\Lambda _{i,1}^{j_1} Q_{i,1}^{j_1*} Q_{i,1}^{j_2}\Lambda _{i,1}^{j_2} Q_{i,1}^{j_2*}\nonumber \\&\quad =Q_{i,1}^{j_1}\Lambda _{i,1}^{j_1}(Q_{i,1}^{j_1*} Q_{i,1}^{j_2})\Lambda _{i,1}^{j_2} Q_{i,1}^{j_2*}\nonumber \\&\quad =0 \end{aligned}$$

(32)

That is, the matrices are pairwise orthogonal, i.e., $({\textbf {Z}}_i^{j_1})^*({\textbf {Z}}_i^{j_2})$ $=0$, where $1\le j_1 \le j_2 \le k$.

Since $\det (Z_iZ_i^*)=\det (\Sigma _i\Sigma _i^*)=\det (\Sigma _{i,1}\Sigma _{i,1}^*)$ $=\prod _{l=1} ^{d_j}\varsigma _{l,j}^2$, we have:

$$\begin{aligned} t_{et}^i=\sum _{j=1}^{k}\frac{n_j}{n}\log _2\left( \frac{n}{n_j}\right) -\frac{n}{d}\log _2\left( \frac{n}{d}\right) +\frac{2n}{d}\log _2\left( \prod _{l=1}^{d_j}\varsigma _{l,j}\right) \end{aligned}$$

(33)

In order to maximize $t_{et}^i$, we need to maximize $\prod _{l=1}^{d_j}\varsigma _{l,j}$, subject to satisfying the constraints. Since $\left\| U_{i,1}\Sigma _{i,1} V_{i,1}^*\textrm{C}_i^j \right\| ^2=n_j$, we have:

$$\begin{aligned}&\left\| U_{i,1}\Sigma _{i,1} V_{i,1}^*\textrm{C}_i^j \right\| ^2\nonumber \\&\quad =\textrm{tr}(U_{i,1}\Sigma _{i,1} V_{i,1}^*\textrm{C}_i^j V_{i,1} \Sigma _{i,1}^* U_{i,1}^*)\nonumber \\&\quad =\textrm{tr}(\Sigma _{i,1} V_{i,1}^*\textrm{C}_i^j V_{i,1} \Sigma _{i,1}^*)\nonumber \\&\quad =\sum _{l=1}^{d_j}\varsigma _{l,j}^2\left\| V_{i,l}^* \textrm{C}_i^j V_{i,l} \right\| ^2\nonumber \\&\quad =n_j \end{aligned}$$

(34)

where $V_{i,l}$ represents the lth column of $V_{i,1}$. Since $\left\| V_{i,l}^*\textrm{C}_i^j V_{i,l} \right\| ^2\le 1$, we can get:

$$\begin{aligned} \sum _{l=1}^{d_j}\varsigma _{l,j}^2\le n_j \end{aligned}$$

(35)

the equal sign holds true if and only if $\left\| V_{i,l}^*\textrm{C}_i^j V_{i,l} \right\| ^2= 1$. This means that $V_{i,l}$ must be an eigenvector of $\textrm{C}_i^j$, and the corresponding eigenvalue is 1. Since only $n_j$ eigenvalues of $\textrm{C}_i^j$ are 1 and the rest are 0. To maximize $\prod _{l=1}^{d_j}\varsigma _{l,j}$, we need to make $\varsigma _{l,j}$ as equal as possible, that is:

$$\begin{aligned} \varsigma _{l,j}=n_j/d_j,\quad \forall l\in \left\{ 1,\ldots ,d_j \right\} \end{aligned}$$

(36)

Then, according to Chan (2022), the optimization problem in Eq. (23) depends on $Z_i^j$ only through its singular values. We have:

$$\begin{aligned}&\left\| Z_i^j \right\| _{F}^2 =\sum _{l=1}^{\min \left\{ n_j,d \right\} } \varsigma _{l,j}^2\nonumber \\&\textrm{rank}(Z_i^j)=\left\| \varsigma _j \right\| _0 \end{aligned}$$

(37)

Let $\varsigma _j^*:= \left[ \varsigma _{1,j}^*,\ldots , \varsigma _{min(n_j,d),j}^* \right]$ be an optimal solution to Eq. (23). Without loss of generality we assume that the entries of $\varsigma _j^*$ are sorted in descending order. We define:

$$\begin{aligned} f(z;d,\epsilon ,n_j,n)=\log \left( \frac{1 +\frac{d}{n \epsilon ^2}{\varsigma _i}}{1 +\frac{d}{n_j \epsilon ^2}{\varsigma _i^j}}\right) \end{aligned}$$

(38)

Then apply the Lemma 13 in Chan (2022) and conclude that the unique optimal solution to Eq. (23), we get:

• $\varsigma _i^*=\left[ \frac{n_j}{d_j},\ldots ,\frac{n_j}{d_j} \right]$ or

• $\varsigma _i^*=\left[ \frac{n_j}{d_j},\ldots ,\frac{n_j}{{d_j}-1}, \varsigma _i^L \right] , \varsigma _i^L>0$

$\square$

1.3 A.3 Proof of Theorem 3

Theorem 3, also the Theorem 6 mentioned below, gives the lower bound of task difficulty. The condition for the lower bound being tight is consistent with Minimally Effect Gap in Corollary 1. This shows that task difficulty can well reflect causal invariance.

Theorem 6

The support set $\mathcal {D}_i^s$ and query set $\mathcal {D}_i^q$ are two different datasets of $\mathcal {T}_i$. For any $\mathcal {T}_i$ and f, we have:

$$\begin{aligned} \sum \limits _{i,j} {\Vert {{\nabla _{x_{i,j}^s}}\mathcal {L} (\mathcal {D}_{i}^s,f) - {\nabla _{x_{i,j}^q}}\mathcal {L} (\mathcal {D}_{i}^q,f}) \Vert _2^2}\ge 0 \end{aligned}$$

(39)

the equality holds if and only if the gradients of the support set $\mathcal {D}_i^s$ and query set $\mathcal {D}_i^q$ are consistent:

$$\begin{aligned} \sum \limits _{j}{{\nabla _{x_{i,j}^s}}\mathcal {L}(\mathcal {D}_{i}^s,f) = \sum \limits _{j}{\nabla _{x_{i,j}^q}}\mathcal {L}(\mathcal {D}_{i}^q,f}) \end{aligned}$$

(40)

Proof

we can deduce it directly from Eq. (39):

$$\begin{aligned}&\sum \limits _{i,j} {\Vert {{\nabla _{x_{i,j}^s}}\mathcal {L} (\mathcal {D}_{i}^s,f) - {\nabla _{x_{i,j}^q}}\mathcal {L} (\mathcal {D}_{i}^q,f}) \Vert _2^2}\nonumber \\&\quad =\sum \limits _{i} {\Vert \sum \limits _{j}{{\nabla _{x_{i,j}^s}} \mathcal {L}(\mathcal {D}_{i}^s,f) -\sum \limits _{j}{\nabla _{x_{i,j}^q}}\mathcal {L} (\mathcal {D}_{i}^q,f)} \Vert _2^2} \end{aligned}$$

(41)

where $\left\| \cdot \right\|$ represents absolute value, always greater than zero, then:

$$\begin{aligned} \sum \limits _{i} {\Vert \sum \limits _{j}{{\nabla _{x_{i,j}^s}} \mathcal {L}(\mathcal {D}_{i}^s,f) -\sum \limits _{j}{\nabla _{x_{i,j}^q}} \mathcal {L}(\mathcal {D}_{i}^q,f}) \Vert _2^2}\ge 0 \end{aligned}$$

(42)

where the equality holds if and only if:

$$\begin{aligned} \sum \limits _{j}{{\nabla _{x_{i,j}^s}}\mathcal {L} (\mathcal {D}_{i}^s,f) = \sum \limits _{j}{\nabla _{x_{i,j}^q}} \mathcal {L}(\mathcal {D}_{i}^q,f)} \end{aligned}$$

(43)

In summary, the measurements we proposed can well evaluate the quality of meta-learning tasks.

Appendix B: Meta-learning Models

In this section, we describe the details and experimental settings of the three types of meta-learning models mentioned in Sect. 8.2.

1.1 B.1 Overview

To conduct a more comprehensive analysis of task diversity, we incorporate various meta-learning models. According to Hospedales et al. (2021), we classify them into three categories: 1) Optimization-based (i.e., MAML (Finn et al., 2017), Reptile (Nichol and Schulman, 2018), and MetaOptNet (Lee et al., 2019)). These methods aim to learn a set of optimal initialization parameters that guide the model to quickly converge when learning new tasks. 2) Metric-based (i.e., ProtoNet (Snell et al., 2017), MatchingNet (Vinyals et al., 2016), and RelationNet (Sung et al., 2018)). These non-parametric methods are based on metric learning and are similar to nearest neighbor algorithms and kernel density estimation. 3) Bayesian-based models (i.e., CNAPs (Requeima et al., 2019), SCNAP (Bateni et al., 2020)). These methods use conditional probability as the core of meta-learning computations and modify the classifier to pick up new classes using pre-trained networks.

With the development of meta-learning, many novel models and variants have emerged in recent years. Besides the above three types of frameworks, there are methods that particularly meta-learn features designed for few-shot learning and/or update features during meta-testing (i.e., SimpleShot (Wang et al., 2019b), SUR (Mangla et al., 2020), and PPA (Qiao et al., 2018)). We consider these models to be non-direct frameworks for the study of task sampling. For cross-domain analysis, we use Baseline++ (Chen et al., 2019) and S2M2 (Mangla et al., 2020) that use linear classifiers, and MetaQDA (Zhang et al., 2021b) which is a Bayesian meta-learning generalization of the classic quadratic discriminant analysis as base frameworks.

1.2 B.2 Optimization-based

1.2.1 B.2.1 MAML

MAML (Finn et al., 2017) is a meta-learning approach that is agnostic to specific models, making it compatible with any model trained using gradient descent. It can be applied to a variety of learning problems, with the explicit goal of training parameters that will generalize well to new tasks with only a small amount of training data and a few gradient steps.

For meta-learning with MAML, the method first initializes meta-parameters $\theta$ and samples tasks $\mathcal {T}_{i}$ from a task distribution $p(\mathcal {T})$. In the inner loop, the adaptive parameters are calculated for each task $\mathcal {T}_{i}$ using gradient descent, as follows:

$$\begin{aligned} \theta _{i}^{'}=\theta -\alpha \bigtriangledown _{\theta } \mathcal {L}_{\mathcal {T}_{i} }(f_{\theta }) \end{aligned}$$

(44)

In the outer loop, the meta-parameter $\theta$ is updated based on the accumulated gradient, as follows:

$$\begin{aligned} \theta \leftarrow \theta -\beta \bigtriangledown _{\theta } {\textstyle \sum _{\mathcal {T}_{i}\sim p(\mathcal {T})}} \mathcal {L}_{\mathcal {T}_{i} }(f_{\theta _{i}^{'}}) \end{aligned}$$

(45)

where $\alpha$ and $\beta$ are the step size hyperparameters of the inner loop and outer loop, respectively.

In the experiments, we set the parameters as follows: the size of the running epoch is set to 150; batch size is 32 or 16; the meta-learning rate of Adam optimizer (Kingma and Ba, 2014) is 0.001; and the internal adaptation number is 1 with a step size of 0.4.

1.2.2 B.2.2 Reptile

Reptile (Nichol and Schulman, 2018) is a meta-learning approach that extends MAML by learning the initialization of neural network model parameters. It operates by repeatedly sampling a task, training it, and moving the initialization of the model parameters toward the trained weights for that task.

For meta-learning with Reptile, the method first initializes the meta-parameter $\theta$. For each iteration, a task $\mathcal {T}$ is sampled, corresponding to loss $\mathcal {L}_{\mathcal {T}}$ and a set of trained weights $\tilde{\theta }$. For a specific task, Reptile computes $\tilde{\theta }=U_{\mathcal {T}^{k} }(\theta )$ denoting k steps of SGD or Adam. The meta-parameter $\theta$ is then updated using the following equation:

$$\begin{aligned} \theta \leftarrow \theta +\epsilon (\tilde{\theta }-\theta ) \end{aligned}$$

(46)

In the last step, instead of simply updating $\theta$, Reptile treats $\tilde{\theta }-\theta$ as a gradient and plugs it into an adaptive algorithm, such as Adam (Kingma and Ba, 2014).

In the experiments, we set the parameters as follows: For miniImageNet, the running epoch size is set to 150, the batch size is 32, the learning rate is 0.01, the meta-learning rate is 0.001, and the internal adaptation number is 5. The inner loop uses the SGD optimizer, and the outer loop uses the Adam optimizer. For tieredImageNet, we increase the number of internal adaptations to 10. For Omniglot, the meta-learning rate is set to 0.0005, and the number of internal adaptations is the same as that on tieredImageNet, while only running for 100 epochs.

1.2.3 B.2.3 MetaOptNet

MetaOptNet (Lee et al., 2019) is a meta-learning model proposed for few-shot learning, which aims to learn an embedding model that generalizes well for novel categories under a linear classification rule. To achieve this, it utilizes the implicit differentiation of the optimality conditions of the convex problem and the dual formulation of the optimization problem.

The learning objective of MetaOptNet is to minimize the generalization error across tasks given a base learner $\mathcal {A}$ and an embedding model $\phi$. The generalization error is estimated on a set of held-out tasks. The choice of the base learner has a significant impact on the objective as it has to be efficient since the expectation has to be computed over a distribution of tasks.

Formally, the learning objective is:

$$\begin{aligned} \underset{\phi }{\min }\ \mathbb {E}_{\mathcal {T}} \left[ \mathcal {L}^{meta}(\mathcal {D}^{test};\theta ;\phi )\right] \end{aligned}$$

(47)

Once the embedding model $f_{\phi }$ is learned, its generalization is estimated on a set of held-out tasks (often referred to as a meta-test set):

$$\begin{aligned} \mathbb {E}_{S}[\mathcal {L}^{meta}(\mathcal {D}^{test};\theta ,\phi ) ], \quad where \quad \theta =\mathcal {A}(\mathcal {D}^{train};\theta ) \end{aligned}$$

(48)

The above equation is greatly affected by the selection of the base learner $\mathcal {A}$. The chosen base learner must be efficient as the expectation is calculated across a task distribution. In this study, we explore base learners that rely on multi-class linear classifiers, which can be expressed in a simplified form as follows:

$$\begin{aligned}&\theta = \mathcal {A}(\mathcal {D}^{train};\phi )=\arg \underset{\left\{ w_{k} \right\} }{\min } \underset{\left\{ \zeta _{i} \right\} }{\min } \frac{1}{2}\sum _{k}\left\| w_{k} \right\| _{2}^{2} +C\sum _{n}\zeta _{n} \end{aligned}$$

(49)

$$\begin{aligned}&w_{y_{n}} \cdot f_{\phi }(x_{n})-w_{k}\cdot f_{\phi }(x_{n}) \ge 1-\delta _{y_{n,k}}-\zeta _{n},\forall n,k \end{aligned}$$

(50)

where C is the regularization parameter, and $\delta _{.,.}$ denotes the Kronecker delta function. The official repository trains the model using a 5-way 15-shot approach and evaluates it using a 5-way 1-shot approach. However, to ensure a fair and accurate comparison with other models as outlined in Kumar et al. (2022), we train and test the model using a 5-way 1-shot approach in this study. It is worth noting that our focus is on comparing the performance of different samplers for a given model, and the aforementioned difference in training and testing approaches would not affect our examination of task diversity in any way.

In the experiments, we set the parameters as follows: the size of the running epoch is set to 60, the batch size is 32 or 16, the learning rate is 0.01, and the meta-learning rate is 0.001. We use an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001 to make gradient steps.

1.3 B.3 Metric-based Models

1.3.1 B.3.1 ProtoNet

Prototypical networks (ProtoNet) (Snell et al., 2017) is a method proposed for few-shot classification tasks. This involves a classifier that must generalize to new classes not seen in the training set, with only a few examples available for each new class. ProtoNet addresses this problem by learning a metric space, where classification is performed by calculating distances to prototype representations of each class. Compared to other few-shot learning approaches, ProtoNet’s simpler inductive bias is advantageous in the limited-data regime and achieves outstanding results.

To generate M-dimensional prototype representations $c_{k} \in \mathbb {R}^{M}$ for each class, ProtoNet employs an embedding function $f_{\phi }:\mathbb {R}^{D}\rightarrow \mathbb {R}^{M}$ with learnable parameters $\phi$. Each prototype is computed as the mean vector of the embedded support points belonging to its corresponding class:

$$\begin{aligned} c_k=\frac{1}{\left| S \right| }\sum _{(x_i,y_i)\in S_{k}}f_{\phi }(x_i) \end{aligned}$$

(51)

Once a prototype is constructed for each class, ProtoNet classifies query examples by determining the nearest prototype to them in the metric space using Euclidean distance. Specifically, the probability that a query example $x^*$ belongs to class k is calculated as follows:

$$\begin{aligned} p(y^*=k|x^*,S)=\frac{\textrm{exp}(-\left\| g(x^*)-c_{k} \right\| _2^2 ) }{ {\textstyle \sum _{k^{'}\in \left\{ 1,\ldots ,N \right\} }\textrm{exp}(-\left\| g(x^*) -c_{k^{'}} \right\| _2^2 )}} \end{aligned}$$

(52)

In our experiments, we set the parameters as follows: For miniImageNet, Omniglot, and tieredImageNet, we use a batch size of 32 and run for 100 epochs in a 5-way-1-shot setting. However, we use a batch size of 16 rather than 32 in a 20-way-1-shot setting to accommodate the longer training time and memory constraints. We set the meta-learning rate to 0.001, use an Adam optimizer for gradient steps, and set the step size of the StepLR scheduler to 0.4 with a gamma value of 0.5.

1.3.2 B.3.2 MatchingNet

Matching Networks (MatchingNet), as described in Vinyals et al. (2016), leverages the concepts of metric learning based on deep neural features and the latest developments that enhance neural networks with external memories. This approach trains a network that maps a small labeled support set and an unlabelled example to its label, eliminating the need for fine-tuning to adapt to new class types.

The crucial point is that once trained, MatchingNet can generate sensible test labels for unseen classes without modifying the network. More precisely, MatchingNet aims to map a support set of k image-label pairs, denoted as $S=\left\{ (x_i,y_i) \right\} ^k_{i=1}$, to a classifier $c_{S}(x^* )$. Given a test example $x^*$, the classifier produces a probability distribution over possible outputs $y^*$. The parametric neural network defined by p is used to predict the appropriate label $\hat{y}$ for each test example $x^*$. MatchingNet assigns labels to each query example based on a cosine distance-weighted linear combination of the support labels:

$$\begin{aligned} p(y^*=k|x^*,S)=\sum _{i=1}^{\left| S \right| }a(x^*,x_i) \Psi _{y_{i}=k} \end{aligned}$$

(53)

where $a(\cdot ,\cdot )$ denotes cosine similarity, $\Psi$ is the indicator function, and the output is softmax normalized over all support examples $x_i$.

In the experiments, we set the model parameters as follows: For standard few-shot learning under 5-way-1-shot settings, we run the epoch for 100 times with a batch size of 32. To make gradient steps, we use an Adam optimizer with a meta-learning rate of 0.001 and a weight decay of 0.0001. For training on CUB and Meta-Dataset under a 5-way 1-shot setting, we use the same parameters as the miniImageNet, except for the batch size and learning rate, which were set to 16 and 0.005, respectively.

1.3.3 B.3.3 RelationNet

Relation Network (RelationNet), presented in Sung et al. (2018), is a flexible and general framework for few-shot learning that is conceptually simple. The framework involves learning a deep distance metric to compare a small number of images within episodes, which is trained end-to-end from scratch.

The RelationNet framework comprises two modules: an embedding module, $f_{\varphi }$, and a relation module, $g_{\phi }$. The embedding module produces feature maps, $f_{\varphi }(x_{i})$ and $f_{\varphi }(x_{j})$, where $x_{i}$ and $x_{j}$ are samples in the support set S, and query set Q. These feature maps are combined using the operator $C(f_{\varphi }(x_{i}),f_{\varphi }(x_{j}))$ and fed into the relation module, $g_{\phi }$, for the next stage. The relation module produces a scalar value between 0 and 1 that represents the similarity between $x_{i}$ and $x_{j}$. The relation scores $r_{i,j}$ in C-way-1-shot settings (C relation scores) are generated using the following equation:

$$\begin{aligned} r_{i,j}=g_{\phi }(C(f_{\varphi }(x_i),f_{\varphi }(x_j))),i=1,2,\ldots ,C \end{aligned}$$

(54)

For K-shot settings, where $K>1$, we sum the embedding module outputs of all samples from each training class element-wise to form the feature map for that class. The model is trained using Mean Square Error (MSE) loss:

$$\begin{aligned} \varphi ,\phi =\underset{\varphi ,\phi }{\arg \min }\sum _{i=1}^{m} \sum _{j=1}^{n}(r_{i,j}-1(y_i==y_j))^2 \end{aligned}$$

(55)

Conceptually, this framework predicts relation scores, which can be considered a regression problem.

In the experiments, we set the model parameters as follows: For Omniglot, miniImageNet, and tieredImageNet under a 5-way-1-shot setting, the method is run for 100 epochs with a batch size of 32. An Adam optimizer is used to make gradient steps with a meta-learning rate of 0.001 and a weight decay of 0.0005. The same hyperparameters are used for training the model on Omniglot under a 20-way-1-shot setting.

In our experiments on Omniglot and miniImageNet, and tieredImageNet under a 5-way-1-shot setting, we run the epoch 100 times with a batch size of 32. We use an Adam optimizer to make gradient steps with a meta-learning rate of 0.001 and a weight decay of 0.0005. The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.

1.4 B.4 Bayesian-based Models

1.4.1 B.4.1 CNAPs

The Conditional Neural Adaptive Processes (CNAPs) (Requeima et al., 2019) approach is designed to handle multi-task classification problems. It is based on a conditional neural process that employs an adaptation network to modulate the classifier’s parameters based on the current task’s dataset, without requiring additional tuning. This feature enables the model to handle a variety of input distributions.

The data for task $\tau$ includes a context set $D^\tau =\left\{ (x_{n}^\tau ,y_{n}^\tau ) \right\} _{n=1}^{N_{\tau }}$ and a target set $\left\{ (x_{m}^\tau ,y_{m}^\tau ) \right\} _{n=1}^{M_{\tau }}$. The former is with inputs and outputs observed while the latter is used to make predictions ($y^{\tau *}$ are only observed during training). CNAPs construct predictive distributions given $x^*$ as:

$$\begin{aligned} p(y^*|x^*,\theta , D^\tau )=p(y^*|x^*,\theta ,\psi ^\tau =\psi _\phi (D^\tau )) \end{aligned}$$

(56)

where $\theta$ are global classifier parameters shared across tasks, $\phi$ are adaptation network parameters used in the function $\psi _\phi (\cdot )$ that acts on $D^\tau$, and $\psi ^\tau$ are local task-specific parameters produced by $\psi _\phi (\cdot )$.

In the experiments, we set the model parameters as follows: In the standard few-shot learning setting, we run the epoch ten times with a batch size of 16 and a meta-learning rate of 0.005. In multi-domain few-shot learning, the meta-learning rate is set to 0.01.

1.4.2 B.4.2 SCNAP

Simple CNAPS (SCNAP) (Bateni et al., 2020)) is an architecture that performs better than CNAPs with up to 9.2% fewer trainable parameters. It hypothesizes that a class-covariance-based distance metric, specifically the Mahalanobis distance, can be adopted into CNAPs. In contrast to CNAPs, SCNAP directly computes the conditional probability $p(\cdot )$ of a sample belonging to a class using a deterministic, fixed distance metric $d_k$, as follows:

$$\begin{aligned}&p(y_i^*=k|f_{\theta }^{\tau }(x_i^*),S^\tau ) =\textrm{softmax}(-d_k(f_{\theta }^{\tau }(x_i^*)),\mu _k) \nonumber \\&d_k(x,y)=\frac{1}{2}(x-y)^T(Q_k^\tau )^{-1}(x-y) \qquad \quad \end{aligned}$$

(57)

where $Q_k^\tau$ is a covariance matrix specific to the task and class.

The parameters of this model are consistent with CNAPs.