这是indexloc提供的服务，不要输入任何密码
Skip to main content
Springer Nature Link
Log in

Delving into Identify-Emphasize Paradigm for Combating Unknown Bias

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Dataset biases are notoriously detrimental to model robustness and generalization. The identify-emphasize paradigm appears to be effective in dealing with unknown biases. However, we discover that it is still plagued by two challenges: A, the quality of the identified bias-conflicting samples is far from satisfactory; B, the emphasizing strategies only produce suboptimal performance. In this paper, for challenge A, we propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy, along with two practical strategies — peer-picking and epoch-ensemble. For challenge B, we point out that the gradient contribution statistics can be a reliable indicator to inspect whether the optimization is dominated by bias-aligned samples. Then, we propose gradient alignment (GA), which employs gradient statistics to balance the contributions of the mined bias-aligned and bias-conflicting samples dynamically throughout the learning process, forcing models to leverage intrinsic features to make fair decisions. Furthermore, we incorporate self-supervised (SS) pretext tasks into training, which enable models to exploit richer features rather than the simple shortcuts, resulting in more robust models. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases and achieve state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Notes

  1. Here, the method assumes the presence of bias in training data. In Sect. 5.5, we discussed whether this method is safe (does not cause severe degradation) when there is no bias in the dataset.

  2. Here, we use the output probability value (the maximum value among all classes) as the confidence.

  3. The data is available at https://nlp.stanford.edu/data/dro/waterbird_complete95_forest2water2.tar.gz.

  4. The data is available at http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html.

  5. As the auxiliary biased models used in ERew and PoE are designed for NLP tasks, here, we combine our ECS with them.

  6. As stated in the original papers, BiasCon and RNF-GT have variations that do not require real annotations assist with various auxiliary biased models. We only provide the upper bound of these strategies when combating unknown biases, as we found that the auxiliary models have a significant impact on the outcomes.

  7. LfF is implemented at https://github.com/alinlab/LfF and ERew is implemented at https://github.com/UKPLab/emnlp2020-debiasing-unknown.

  8. REBIAS is implemented at https://github.com/clovaai/rebias. DFA is implemented at https://github.com/kakaoenterprise/Learning-Debiased-Disentangled. BiasCon is implemented at https://github.com/grayhong/bias-contrastive-learning.

  9. We adopt the clustering methods utilized in GEORGE referring to https://github.com/HazyResearch/hidden-stratification.

  10. The method is implemented at https://github.com/UKPLab/emnlp2020-debiasing-unknown.

  11. It is implemented at https://github.com/mohammadpz/Gradient_Starvation.

References

  • Agarwal, V., Shetty, R., & Fritz, M. (2020). Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9690–9698.

  • Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.

  • Bahng, H., Chun, S., Yun, S., Choo, J., & Oh, S.J. (2020). Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528–539 PMLR.

  • Byrd, J., & Lipton, Z. (2019). What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 872–881. PMLR.

  • Cadene, R., Dancette, C., Cord, M., Parikh, D., et al. (2019). Rubi: Reducing unimodal biases for visual question answering. Advances in Neural Information Processing Systems, 32, 841–852.

    Google Scholar 

  • Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 9912–9924.

    Google Scholar 

  • Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597–1607. PMLR.

  • Chuang, C.-Y., & Mroueh, Y. (2021). Fair mixup: Fairness via interpolation. In International Conference on Learning Representations. URL https://openreview.net/forum?id=DNl5s5BXeBn.

  • Clark, C., Yatskar, M., & Zettlemoyer, L. (2019). Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4069–4082.

  • Clark, C., Yatskar, M., & Zettlemoyer, L. (2020). Learning to model and ignore dataset bias with mixed capacity ensembles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3031–3045.

  • Creager, E., Jacobsen, J.-H., & Zemel, R. (2021). Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189–2200. PMLR.

  • Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9268–9277.

  • Dhar, P., Gleason, J., Roy, A., Castillo, C. D., & Chellappa, R. (2021). Pass: protected attribute suppression system for mitigating bias in face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15087–15096.

  • Doersch, C., Gupta, A., & Efros, A. A. (2015). Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422–1430.

  • Donini, M., Oneto, M., Ben-David, S., Shawe-Taylor, J.S., & Pontil, M. (2018). Empirical risk minimization under fairness constraints. Advances in Neural Information Processing Systems, 31.

  • Du, M., Mukherjee, S., Wang, G., Tang, R., Awadallah, A., & Hu, X. (2021). Fairness via representation neutralization. Advances in Neural Information Processing Systems, 34.

  • Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., & Brendel, W. (2018). Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations.

  • Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R. S., Brendel, W., Bethge, M., & Wichmann, F. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2, 665–673.

    Article  Google Scholar 

  • Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728.

  • Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., & Cord, M. (2019). Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8059–8068.

  • Goel, K., Gu, A., Li, Y., & Re, C. (2021). Model patching: Closing the subgroup performance gap with data augmentation. In International Conference on Learning Representations. URL https://openreview.net/forum?id=9YlaeLfuhJF.

  • Goh, J., & Sim, M. (2010). Distributionally robust optimization and its tractable approximations. Operations Research, 58(4–part–1), 902–917.

    Article  MathSciNet  Google Scholar 

  • Gong, S., Liu, X., & Jain, A.K. (2021). Mitigating face recognition bias via group adaptive classifier. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3414–3424.

  • Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I.W., & Sugiyama, M. (2018). Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS.

  • Han, B., Niu, G., Yu, X., Yao, Q., Xu, M., Tsang, I., & Sugiyama, M. (2020). SIGUA: Forgetting may make learning with noisy labels more robust. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4006–4016. PMLR, 13–18 Jul. URL https://proceedings.mlr.press/v119/han20c.html.

  • Hashimoto, T., Srivastava, M., Namkoong, H., & Liang, P. (2018). Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929–1938. PMLR.

  • He, H., Zha, S., & Wang, H. (2019). Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 132–142.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.

  • He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738.

  • He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009.

  • Hong, Y., & Yang, E. (2021). Unbiased classification through bias-contrastive and bias-balanced learning. Advances in Neural Information Processing Systems, 34.

  • Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020). Supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 18661–18673.

    Google Scholar 

  • Kim, B., Kim, H., Kim, K., Kim, S., & Kim, J. (2019a). Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9012–9020.

  • Kim, E., Lee, J., & Choo, J. (2021a). Biaswap: Removing dataset bias with bias-tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14992–15001.

  • Kim, E., Lee, J., Lee, J., Lee, J., & Choo, J. (2021b). Learning debiased representation via disentangled feature augmentation. arXiv preprint arXiv:2107.01372.

  • Kim, M. P., Ghorbani, A., & Zou, J. (2019b). Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 247–254.

  • Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F., Thain, N., Wang, X., & Chi, E.H. (2020). Fairness without demographics through adversarially reweighted learning. arXiv preprint arXiv:2006.13114.

  • Le Bras, R., Swayamdipta, S., Bhagavatula, C., Zellers, R., Peters, M., Sabharwal, A., & Choi, Y. (2020). Adversarial filters of dataset biases. In International Conference on Machine Learning, pp. 1078–1088. PMLR.

  • Li, Y., & Vasconcelos, N. (2019). Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9572–9581.

  • Li, Y., Li, Y., & Vasconcelos, N. (2018). Resound: Towards action recognition without representation bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513–528.

  • Li, Z., Hoogs, A., & Xu, C. (2022). Discover and Mitigate Unknown Biases with Debiasing Alternate Networks. In The European Conference on Computer Vision (ECCV).

  • Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980–2988.

  • Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., & Finn, C. (2021). Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781–6792. PMLR.

  • Liu, Z., Luo, P., Wang, X., & Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738.

  • Moyer, D., Gao, S., Brekelmans, R., Galstyan, A., & Ver Steeg, G. (2018). Invariant representations without adversarial training. Advances in Neural Information Processing Systems, 31:9084–9093.

  • Nam, J., Cha, H., Ahn, S., Lee, J., & Shin, J. (2020). Learning from failure: Training debiased classifier from biased classifier. In Advances in Neural Information Processing Systems, 33, 20673–84.

    Google Scholar 

  • Noroozi, M., & Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69–84. Springer.

  • Park, T., Zhu, J.-Y., Wang, O., Lu, J., Shechtman, E., Efros, A.A., & Zhang, R. (2020). Swapping autoencoder for deep image manipulation. arXiv preprint arXiv:2007.00653.

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 8026–8037.

    Google Scholar 

  • Pezeshki, M., Kaba, S.-O., Bengio, Y., Courville, A., Precup, D., & Lajoie, G. (2020). Gradient starvation: A learning proclivity in neural networks. arXiv preprint arXiv:2011.09468.

  • Ragonesi, R., Volpi, R., Cavazza, J., & Murino, V. (2021). Learning unbiased representations via mutual information backpropagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2729–2738.

  • Reddy, C., Sharma, D., Mehri, S., Romero-Soriano, A., Shabanian, S., & Honari, S. (2021). Benchmarking bias mitigation algorithms in representation learning through fairness metrics. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).

  • Robinson, J., Sun, L., Yu, K., Batmanghelich, K., Jegelka, S., & Sra, S. (2021). Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 34, 4974–4986.

    Google Scholar 

  • Sagawa, S., Koh, P.W., Hashimoto, T.B., & Liang, P. (2020a). Distributionally robust neural networks. In International Conference on Learning Representations. URL https://openreview.net/forum?id=ryxGuJrFvS.

  • Sagawa, S., Raghunathan, A., Koh, P.W., & Liang, P. (2020b). An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346–8356. PMLR.

  • Sanh, V., Wolf, T., Belinkov, Y., & Rush, A.M. (2020). Learning from others’ mistakes: Avoiding dataset biases without modeling them. In International Conference on Learning Representations.

  • Sohoni, N., Dunnmon, J., Angus, G., Gu, A., & Ré, C. (2020). No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems, 33, 19339.

    Google Scholar 

  • Tan, J., Lu, X., Zhang, G., Yin, C., & Li, Q. (2021). Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1685–1694.

  • Tartaglione, E., Barbano, C. A., & Grangetto, M. (2021). End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13508–13517.

  • Tong, Z., Song, Y., Wang, J., & Wang, L. (2022). Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602.

  • Utama, P.A., Moosavi, N.S., & Gurevych, I. (2020a). Towards debiasing nlu models from unknown biases. In EMNLP.

  • Utama, P.A., Moosavi, N.S., & Gurevych, I. (2020b). Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, July.

  • Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., & Van Gool, L. (2020). Scan: Learning to classify images without labels. In European Conference on Computer Vision, pp. 268–285. Springer.

  • Wang, H., He, Z., Lipton, Z.L., & Xing, E.P. (2019). Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations. URL https://openreview.net/forum?id=rJEjjoR9K7.

  • Wang, Q.-W., Zhao, B., Zhu, M., Li, T., Liu, Z., & Xia, S.-T. (2022). Towards mitigating the problem of insufficient and ambiguous supervision in online crowdsourcing annotation. arXiv preprint arXiv:2210.11194.

  • Wang, X., Zhang, R., Shen, C., Kong, T., & Li, L. (2021). Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024–3033.

  • Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020). Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8919–8928.

  • Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., & Perona, P. (2010). Caltech-ucsd birds 200.

  • Yaghoobzadeh, Y., Mehri, S., des Combes, R.T., Hazen, T.J., & Sordoni, A. (2021). Increasing robustness to spurious correlations using forgettable examples. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3319–3332.

  • Zeng, Y., Zhao, B., Qiu, S., Dai, T., & Xia, S.-T. (2022). Towards effective image manipulation detection with proposal contrastive learning. arXiv preprint arXiv:2210.08529.

  • Zhai, X., Oliver, A., Kolesnikov, A., & Beyer, L. (2019). S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476–1485.

  • Zhang, Z., & Sabuncu, M.R. (2018). Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS).

  • Zhao, B., Xiao, X., Gan, G., Zhang, B., & Xia, S.-T. (2020). Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208–13217.

  • Zhao, B., Chen, C., Wang, Q.-W., He, A., & Xia, S.-T. (2023). Combating unknown bias with effective bias-conflicting scoring and gradient alignment. In Proceedings of the AAAI Conference on Artificial Intelligence. URL https://arxiv.org/abs/2111.13108.

  • Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pp. 2921–2929.

  • Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2017). Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6), 1452–1464.

    Article  Google Scholar 

  • Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232.

  • Zhu, W., Zheng, H., Liao, H., Li, W., & Luo, J. (2021). Learning bias-invariant representation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15002–15012.

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under Grant 62171248, the R &D Program of Shenzhen under Grant JCYJ20220818101012025, the PCNL KEY project (PCL2021A07), and Shenzhen Science and Technology Innovation Commission (Research Center for Computer Network (Shenzhen) Ministry of Education).

Author information

Authors and Affiliations

  1. Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

    Bowen Zhao, Qian-Wei Wang & Shu-Tao Xia

  2. OPPO research institute, Shenzhen, China

    Chen Chen

  3. Research Center of Artificial Intelligence, Peng Cheng Laboratory, Shenzhen, China

    Qian-Wei Wang & Shu-Tao Xia

  4. Tencent, Shenzhen, China

    Anfeng He

Authors
  1. Bowen Zhao
    View author publications

    Search author on:PubMed Google Scholar

  2. Chen Chen
    View author publications

    Search author on:PubMed Google Scholar

  3. Qian-Wei Wang
    View author publications

    Search author on:PubMed Google Scholar

  4. Anfeng He
    View author publications

    Search author on:PubMed Google Scholar

  5. Shu-Tao Xia
    View author publications

    Search author on:PubMed Google Scholar

Corresponding authors

Correspondence to Chen Chen or Shu-Tao Xia.

Additional information

Communicated by Bernhard Egger.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, B., Chen, C., Wang, QW. et al. Delving into Identify-Emphasize Paradigm for Combating Unknown Bias. Int J Comput Vis 132, 2310–2330 (2024). https://doi.org/10.1007/s11263-023-01969-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11263-023-01969-6

Keywords