这是indexloc提供的服务，不要输入任何密码
Skip to main content
Springer Nature Link
Log in

Breaking the Limits of Reliable Prediction via Generated Data

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In open-world recognition of safety-critical applications, providing reliable prediction for deep neural networks has become a critical requirement. Many methods have been proposed for reliable prediction related tasks such as confidence calibration, misclassification detection, and out-of-distribution detection. Recently, pre-training has been shown to be one of the most effective methods for improving reliable prediction, particularly for modern networks like ViT, which require a large amount of training data. However, collecting data manually is time-consuming. In this paper, taking advantage of the breakthrough of generative models, we investigate whether and how expanding the training set using generated data can improve reliable prediction. Our experiments reveal that training with a large quantity of generated data can eliminate overfitting in reliable prediction, leading to significantly improved performance. Surprisingly, classical networks like ResNet-18, when trained on a notably extensive volume of generated data, can sometimes exhibit performance competitive to pre-training ViT with a substantial real dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  • Bai, Y., Mei, S., Wang, H., & Xiong, C. (2021). Don’t just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In International conference on machine learning (pp. 566–576).

  • Bao, F., Li, C., Zhu, J., & Zhang, B. (2022). Analytic-DPM: An analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International conference on learning representations.

  • Besnier, V., Jain, H., Bursuc, A., Cord, M., & Pérez, P. (2020). This dataset does not exist: training models from generated images. In IEEE international conference on acoustics speech and signal processing.

  • Bitterwolf, J., Meinke, A., Augustin, M., & Hein, M. (2022). Breaking down out-of-distribution detection: Many methods based on ood training data estimate a combination of the same core quantities. In International conference on machine learning (pp. 2041–2074).

  • Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., & Zhang, X. (2016). End to end learning for self-driving cars. arxiv preprint arxiv:1604.07316

  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review.

  • Brock, A., Donahue, J., & Simonyan, K. (2019). Large scale GAN training for high fidelity natural image synthesis. In International conference on learning representations.

  • Cheng, Z., Zhu, F., Zhang, X.-Y., & Liu, C.-L. (2023). Average of pruning: Improving performance and stability of out-of-distribution detection. arxiv preprint arxiv:2303.01201

  • Chow, C. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16, 41–46.

    Article  MATH  Google Scholar 

  • Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014). Describing textures in the wild. In IEEE conference on computer vision and pattern recognition.

  • Corbière, C., Thome, N., Bar-Hen, A., Cord, M., & Pérez, P. (2019). Addressing failure prediction by learning model confidence. In Advances in neural information processing systems.

  • Corbière, C., Thome, N., Saporta, A., Vu, T.-H., Cord, M., & Pérez, P. (2021). Confidence estimation via auxiliary models. In IEEE: IEEE transactions on pattern analysis and machine intelligence.

  • Cortes, C., DeSalvo, G., & Mohri, M. (2016). Learning with rejection. In Alt.

  • Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition.

  • Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., & Houlsby, N. (2021). An image is worth \(16\times 16\) words: Transformers for image recognition at scale. In International conference on learning representations.

  • Dubuisson, B., & Masson, M. (1993). A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1), 155–165.

    Article  MATH  Google Scholar 

  • El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 1605–1641.

    MathSciNet  MATH  Google Scholar 

  • Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature: Nature Publishing Group.

    Book  Google Scholar 

  • Fang, Z., Li, Y., Lu, J., Dong, J., Han, B., & Liu, F. (2022). Is out-of-distribution detection learnable? Advances in Neural Information Processing Systems, 35, 37199–37213.

    MATH  Google Scholar 

  • Fang, Z., Lu, J., Liu, A., Liu, F., & Zhang, G. (2021). Learning bounds for open-set learning. In International conference on machine learning (pp. 3122–3132).

  • Fort, S., Ren, J., & Lakshminarayanan, B. (2021). Exploring the limits of out-of-distribution detection. In Advances in neural information processing systems.

  • Gao, Y., Parameswaran, A., & Peng, J. (2017). On the interpretability of conditional probability estimates in the agnostic setting. In International conference on artificial intelligence and statistics (pp. 1367–1374). PMLR.

  • Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in neural information processing systems.

  • Geifman, Y., Uziel, G., & El-Yaniv, R. (2019). Bias-reduced uncertainty estimation for deep neural classifiers. In International conference on learning representations.

  • Gowal, S., Rebuffi, S.-A., Wiles, O., Stimberg, F., Calian, D. A., & Mann, T. A. (2021). Improving robustness using generated data. In Advances in neural information processing systems.

  • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In International conference on machine learning.

  • Gupta, K., Rahimi, A., Ajanthan, T., Mensink, T., Sminchisescu, C., & Hartley, R. (2021). Calibration of neural networks using splines. In International conference on learning representations.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition.

  • He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., & Qi, X. (2023). Is synthetic data from generative models ready for image recognition? In International conference on learning representations.

  • Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., & Song, D. (2022a). Scaling out-of-distribution detection for real-world settings. In International conference on machine learning.

  • Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In International conference on learning representations.

  • Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International conference on learning representations.

  • Hendrycks, D., Lee, K., & Mazeika, M. (2019a). Using pre-training can improve model robustness and uncertainty. In International conference on machine learning.

  • Hendrycks, D., Mazeika, M., & Dietterich, T. (2019b). Deep anomaly detection with outlier exposure. In International conference on learning representations.

  • Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., & Steinhardt, J. (2022b). Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16783–16792).

  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in neural information processing systems.

  • Huang, R., & Li, Y. (2021). Mos: Towards scaling out-of-distribution detection for large semantic space. In IEEE conference on computer vision and pattern recognition.

  • Jaeger, P. F., Lüth, C. T., Klein, L., & Bungert, T. J. (2023). A call to reflect on evaluation practices for failure detection in image classification. In International conference on learning representations.

  • Janai, J., Güney, F., Behl, A., Geiger, A. (2020). Computer vision for autonomous vehicles: Problems, datasets and state of the art. Foundations and trends® in computer graphics and vision. Now Publishers, Inc.

  • Joo, T., & Chung, U. (2020). Revisiting explicit regularization in neural networks for well-calibrated predictive uncertainty. arxiv preprint arxiv:2006.06399

  • Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. In Advances in neural information processing systems.

  • Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images.

  • Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in neural information processing systems.

  • Li, S., Xia, X., Ge, S., & Liu, T. (2022). Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 316–325).

  • Liang, S., Li, Y., & Srikant, R. (2018). Enhancing the reliability of out-of-distribution image detection in neural networks. In International conference on learning representations.

  • Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. In Advances in neural information processing systems.

  • Luo, Y., Zhang, Y., Cai, X., & Yuan, X. (2019). E2gan: End-to-end generative adversarial network for multivariate time series imputation. In International joint conference on artificial intelligence.

  • Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., & Lucic, M. (2021). Revisiting the calibration of modern neural networks. In Advances in neural information processing systems.

  • Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., & Li, Y. (2022a). Delving into out-of-distribution detection with vision-language representations. In Advances in neural information processing systems.

  • Ming, Y., Fan, Y., & Li, Y. (2022b). POEM: Out-of-distribution detection with posterior sampling. In International conference on machine learning.

  • Ming, Y., & Li, Y. (2023). How does fine-tuning impact out-of-distribution detection for vision-language models? International Journal of Computer Vision, 132, 596–609.

    Article  MATH  Google Scholar 

  • Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018). Foundations of machine learning. Cambridge: MIT Press.

    MATH  Google Scholar 

  • Moon, J., Kim, J., Shin, Y., & Hwang, S. (2020). Confidence-aware learning for deep neural networks. In International conference on machine learning.

  • Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., & Dokania, P. (2020). Calibrating deep neural networks using focal loss. In Advances in neural information processing systems.

  • Müller, R., Kornblith, S., & Hinton, G. E. (2019). When does label smoothing help? In Advances in neural information processing systems.

  • Naeini, M. P., Cooper, G., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI conference on artificial intelligence.

  • Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A.Y. (2011). Reading digits in natural images with unsupervised feature learning.

  • Ng, A., & Jordan, M. (2001). On discriminative vs. generative classifiers: A comparison of logistic regression and Naive Bayes. In Advances in neural information processing systems, vol. 14.

  • Nguyen, A. M., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE conference on computer vision and pattern recognition.

  • Ni, C., Charoenphakdee, N., Honda, J., & Sugiyama, M. (2019). On the calibration of multiclass classification with rejection. In Advances in neural information processing systems.

  • Nichol, A. Q., & Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. In International conference on machine learning.

  • Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., & Chen, M. (2022). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International conference on machine learning.

  • Patel, K., Beluch, W. H., Yang, B., Pfeiffer, M., & Zhang, D. (2021). Multi-class uncertainty calibration via mutual information maximization-based binning. In International conference on learning representations.

  • Pinto, F., Yang, H., Lim, S.-N., Torr, P., & Dokania, P. K. (2022). Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness. In Advances in neural information processing systems.

  • Ramaswamy, H. G., Tewari, A., & Agarwal, S. (2015). Consistent algorithms for multiclass classification with a reject option. arxiv preprint arxiv:1505.04137

  • Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., & Lakshminarayanan, B. (2019). Likelihood ratios for out-of-distribution detection. In Advances in neural information processing systems.

  • Roth, K., Milbich, T., Sinha, S., Gupta, P., Ommer, B., & Cohen, J. P. (2020). Revisiting training strategies and generalization performance in deep metric learning. In International conference on machine learning (pp. 8242–8252).

  • Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M. H., & Sabokrou, M. (2022). A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. In TMLR.

  • Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., & Madry, A. (2018). Adversarially robust generalization requires more data. In Advances in neural information processing systems.

  • Scimeca, L., Rubinstein, A., Teney, D., Oh, S. J., Nicolicioiu, A. M., & Bengio, Y. (2023). Shortcut bias mitigation via ensemble diversity using diffusion probabilistic models. arxiv preprint arXiv:2311.16176

  • Sehwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., & Mittal, P. (2022). Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In International conference on learning representations.

  • Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding machine learning: From theory to algorithms. Cambridge: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Shipard, J., Wiliem, A., Thanh, K. N., Xiang, W., & Fookes, C. (2023). Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 769–778).

  • Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. In Advances in neural information processing systems.

  • Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. In International conference on learning representations.

  • Sun, Y., Guo, C., & Li, Y. (2021). React: Out-of-distribution detection with rectified activations. In Advances in neural information processing systems.

  • Sun, Y., Ming, Y., Zhu, X., & Li, Y. (2022). Out-of-distribution detection with deep nearest neighbors. In International conference on machine learning.

  • Terjék, D. (2020). Adversarial lipschitz regularization. In International conference on learning representations.

  • Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2579–2605.

    MATH  Google Scholar 

  • Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., & Belongie, S. (2018). The inaturalist species classification and detection dataset. In IEEE conference on computer vision and pattern recognition.

  • Vaze, S., Han, K., Vedaldi, A., & Zisserman, A. (2022). Open-set recognition: A good closed-set classifier is all you need. In International conference on learning representations.

  • Wang, H., Li, Z., Feng, L., & Zhang, W. (2022). Vim: Out-of-distribution with virtual-logit matching. In IEEE conference on computer vision and pattern recognition.

  • Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li, Y., & Han, B. (2024). Learning to augment distributions for out-of-distribution detection. In Advances in neural information processing systems, vol. 36.

  • Wang, Z., Pang, T., Du, C., Lin, M., Liu, W., & Yan, S. (2023). Better diffusion models further improve adversarial training. In International conference on machine learning.

  • Wei, H., Xie, R., Cheng, H., Feng, L., An, B., & Li, Y. (2022). Mitigating neural network overconfidence with logit normalization. In International conference on machine learning.

  • Wen, Y., Jerfel, G., Muller, R., Dusenberry, M. W., Snoek, J., Lakshminarayanan, B., & Tran, D. (2021). Combining ensembles and data augmentation can harm your calibration. In International conference on learning representations.

  • Xia, X., Liu, T., Han, B., Gong, C., Wang, N., Ge, Z., & Chang, Y. (2021). Robust early-learning: Hindering the memorization of noisy labels. In International conference on learning representations.

  • Xia, X., Liu, T., Wang, N., Han, B., Gong, C., Niu, G., & Sugiyama, M. (2019). Are anchor points really indispensable in label-noise learning? In Advances in neural information processing systems, vol. 32.

  • Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., & Xiao, J. (2015). Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arxiv preprint arxiv:1504.06755

  • Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun. Y., & Du, X. (2022). Openood: Benchmarking generalized out-of-distribution detection. arxiv preprint arxiv:2210.07242

  • Yang, J., Zhou, K., & Liu, Z. (2023). Full-spectrum out-of-distribution detection. International Journal of Computer Vision, 131, 2607–2622.

    Article  MATH  Google Scholar 

  • Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., & Xiao, J. (2015). Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arxiv preprint arxiv:1506.03365

  • Yuan, J., Pinto, F., Davies, A., & Torr, P. (2022). Not just pretty pictures: Toward interventional data augmentation using text-to-image generators. arxiv preprint arXiv:2212.11237

  • Zagoruyko, S., & Komodakis, N. (2016). Wide residual networks. In Procedings of the British machine vision conference.

  • Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In International conference on learning representations.

  • Zheng, C., Wu, G., Bao, F., Cao, Y., Li, C., & Zhu, J. (2023a). Revisiting discriminative vs. generative classifiers: Theory and implications. In International conference on machine learning (pp. 42420–42477).

  • Zheng, C., Wu, G., & Li, C. (2023b). Toward understanding generative data augmentation. In Advances in neural information processing systems.

  • Zheng, H., Wang, Q., Fang, Z., Xia, X., Liu, F., Liu, T., & Han, B. (2023c). Out-of-distribution detection learning with unreliable out-of-distribution sources. In Advances in neural information processing systems (Vol. 36, pp. 72110–72123). Curran Associates, Inc.

  • Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2017). Places: A 10 million image database for scene recognition. In IEEE transactions on pattern analysis and machine intelligence. IEEE.

  • Zhu, F., Cheng, Z., Zhang, X.-Y., & Liu, C.-L. (2022). Rethinking confidence calibration for failure prediction. In European conference on computer vision.

  • Zhu, F., Cheng, Z., Zhang, X.-Y., & Liu, C.-L. (2023). Openmix: Exploring outlier samples for misclassification detection. In IEEE conference on computer vision and pattern recognition.

  • Zhu, F., Zhang, X.-Y., Cheng, Z., & Liu, C.-L. (2024). Revisiting confidence estimation: Towards reliable failure prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3370–3387.

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by “Scientific and Technological Innovation 2030” Program of China Ministry of Science and Technology (2021ZD0113803), National Natural Science Foundation of China (62222609, 62076236), CAS Project for Young Scientists in Basic Research (YSBR-083), and Key Research Program of Frontier Sciences of CAS (ZDBS-LY-7004).

Author information

Authors and Affiliations

  1. State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation of Chinese Academy of Sciences, Beijing, 100190, China

    Zhen Cheng, Xu-Yao Zhang & Cheng-Lin Liu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, 100049, China

    Zhen Cheng, Xu-Yao Zhang & Cheng-Lin Liu

  3. Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, 999077, China

    Fei Zhu

Authors
  1. Zhen Cheng
    View author publications

    Search author on:PubMed Google Scholar

  2. Fei Zhu
    View author publications

    Search author on:PubMed Google Scholar

  3. Xu-Yao Zhang
    View author publications

    Search author on:PubMed Google Scholar

  4. Cheng-Lin Liu
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Cheng-Lin Liu.

Additional information

Communicated by Hong Liu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, Z., Zhu, F., Zhang, XY. et al. Breaking the Limits of Reliable Prediction via Generated Data. Int J Comput Vis 133, 1195–1221 (2025). https://doi.org/10.1007/s11263-024-02221-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11263-024-02221-5

Keywords