Information preservation with wasserstein autoencoders: generation consistency and adversarial robustness

Abstract

Amongst the numerous variants Variational Autoencoder (VAE) has inspired, the Wasserstein Autoencoder (WAE) stands out due to its heightened generative quality and intriguing theoretical properties. WAEs consist of an encoding and a decoding network— forming a bottleneck— with the prime objective of generating new samples resembling the ones it was catered to. In the process, they aim to achieve a target latent representation of the encoded data. Our work offers a comprehensive theoretical understanding of the machinery behind WAEs. From a statistical viewpoint, we pose the problem as concurrent density estimation tasks based on neural network-induced transformations. This allows us to establish deterministic upper bounds on the realized errors WAEs commit, supported by simulations on real and synthetic data sets. We also analyze the propagation of these stochastic errors in the presence of adversaries. As a result, both the large sample properties of the reconstructed distribution and the resilience of WAE models are explored.

: Experiments and Simulations

Table 1 Two-sample tests of equality on latent and encoded distributions

Fig. 17

Concentration of bin estimates corresponding to Five-Gaussian data under ReLU encoders (yellow) against target latent Beta(0.5, 0.8) copula (blue), over epochs (200, 800, 1400, 2000) (left to right from top) in a WAE-GAN setup

Fig. 18

Concentration of bin estimates corresponding to Five-Gaussian under ReLU encoders (yellow), given latent bivariate Gaussian distribution (blue), over epochs (200, 800, 1400, 1800) (left to right from top) in a WAE-MMD setup with regularization $\lambda =0.1$

Fig. 19

Concentration of bin estimates (yellow) against latent Gaussian distribution (blue) and corresponding QQ plots of marginals (upper), for epochs 500 (top row) and 4000 (bottom row) for Swiss roll data. Evidently, the encoded distribution preserves information from the input data and matches the target marginals simultaneously

Fig. 20

Actual Swiss roll data (top left) vs reconstructed samples ($n=10000$) after epochs (1000, 4000, 8000) (clockwise from top right) under MMD latent loss

Fig. 21

Evolution of information preservation over epochs (0, 200, 1000, 1800) (clockwise) based on the propagation of quantile-quantile (QQ) plots of marginals corresponding to encoded (blue) vs latent distribution (red) under ReLU encoders given Five-Gaussian input data, in a WAE-MMD setup with regularization $\lambda =0.1$

1.1 Encoded vs. Latent Distributions

To check the efficacy of a WAE-encoding statistically, we perform two-sample non-parametric tests of equality on target latent and encoded observations. Peacock (1983) suggested a multi-dimensional generalization to the well-known Kolmogorov–Smirnov test, which, however, has high computational complexity. In our study, we identify the test suggested by Fasano and Franceschini (1987) (FF) as a suitable alternative based on its manageable complexity without sacrificing the power and consistency⁶. Many suggestions have been made to improve the reliability of two-sample tests in higher dimensions since. However, given the ease of implementation, we use FF’s version of the KS test. To ascertain our findings, we additionally carry out a referral test of equality of distributions, based on kernelized distances between pairs of observations, namely the Cramér test (Baringhaus and Franz 2004). Unlike FF, the test statistic corresponding to Cramér⁷ is not distribution-free, and requires bootstrapping to obtain the p-value. We utilize two of such methods, namely the usual Monte-Carlo (MC) and calculating the approximate eigenvalues (EV) as the weights to the limiting distribution (of the statistic). Test results on the Five-Gaussian data at $5\%$ level of significance, given $\lambda = 0.8$, are as follows:

The test results corroborate our theoretical findings. Though we only establish an upper bound to the latent loss, it is apparent that there lies an optimization error due to the minimum distance estimate. As such, under a metrizing measure of discrepancy, the target latent law and the encoded estimate must be distinct in distribution. The rejection of the null hypothesis reiterates the same observation.

It becomes even more clear looking at the histograms corresponding to samples from the two, overlaid. The interesting observation from Fig 17, 18 is the visual manifestation of information preservation. Semantic information, in the form of cluster structures originally present in the data set, remains intact in encoded distributions while trying to maximize similarity with their target counterparts. The evolution of this ‘maximization’ is clear from the histograms obtained over epochs. Another viewpoint that attests to this finding is the quantile-quantile plot [Fig 21] of the marginals.

Fig. 22

Sample corrected ($\times n^{\frac{1}{3}}$) Wasserstein reconstruction error corresponding to WAE-MMD for Five-Gaussian input data using (a) a decoder that follows the architecture of Theorem 5.2, and (b) one that violates the width criteria therein, having a comparable number of parameters. (c) Regenerated sample from the latter model after 4000 epochs. The second model does not exhibit accurate reconstruction and the associated errors follow a much slower convergence rate in the process

Fig. 23

Reconstruction error of Five-Gaussian data under (a) JS and (b), (c) sample corrected ($\times n^{\frac{1}{2}}$) MMD latent loss, using GroupSort encoders (grouping 2)

Fig. 24

Reconstructed samples ($n=10000$) from Five-Gaussian dataset with $10\%$ observations contaminated at level 0.2, under MMD latent loss. The corrupting distribution remains standard tri-variate Cauchy

Fig. 25

Reconstructed samples ($n=2000$) from Swiss roll dataset with $10\%$ observations contaminated at level 0.01 & 0.1 (left to right) and $20\%$ observations contaminated at level 0.1, under MMD latent loss. The corrupting distribution is taken to be standard tri-variate Cauchy

Fig. 26

Reconstructed samples ($n=10,000$) from Five-Gaussian dataset with $10\%$ observations contaminated at level 0.2, under MMD latent loss. The corrupting distribution is taken to be standard tri-variate Cauchy