Abstract
Domain Incremental Learning (DIL) focuses on handling complex domain shifts of a continuous data stream for visual tasks such as image classification and image segmentation. In real life, severe domain gaps in DIL are generated from various sources such as data style shifts, data quality degradation, environment changes, and so on. The well-known catastrophic forgetting issue in DIL becomes even more critical when simultaneously considering multiple sources of domain shifts. In this paper, we propose a unified and effective paradigm named Compositional Prompting (C-Prompt) to mitigate the critical forgetting challenge in DIL for image classification tasks. Unlike a popular type of conventional DIL approaches that need to retain abundant exemplars from the old domains, our exemplar-free C-Prompt leverages a prompt-guided Batch-wise Exponential Moving Average (BEMA) strategy to adaptively consolidate learned knowledge without retaining any exemplars. A set of prompts shared across different domains is designed to estimate the knowledge shifts for automatically balancing knowledge acquisition and forgetting. To enhance the learning ability, our proposed C-Prompt explores a domain-specific pool of learnable prompts for each domain, and all the prompt pools are further exploited in a cross-domain compositional manner to facilitate inference. Since the latest prompting-based DIL methods aim to learn one individual prompt for each domain, they always suffer from critical performance degradation caused by the incorrect prediction of domain index during inference and the limited learning capacity by using a single prompt per domain. Instead, our C-Prompt can not only readily acquire domain-specific knowledge but also exploit domain-shared knowledge. Extensive experiments on various large-scale multi-domain benchmarks have demonstrated the superiority of our proposed C-Prompt compared with state-of-the-art methods. Code is available at https://github.com/zhoujiahuan1991/IJCV2024-C-Prompt.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The datasets that support the results and analysis of the current study are available in the DomainNet http://ai.bu.edu/M3SDA/, ImageNet-R https://github.com/hendrycks/imagenet-r, ImageNet-C https://github.com/hendrycks/robustness and CORe50 https://vlomonaco.github.io/core50/ repositories.
References
Agarwal, P., Paudel, D. P., Zaech, J. -N., & Van Gool, L. (2022). Unsupervised robust domain adaptation without source data. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, (pp. 2009–2018).
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022). Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274.
Boschini, M., Bonicelli, L., Buzzega, P., Porrello, A., & Calderara, S. (2022). Class-incremental continual learning into the extended der-verse. arXiv preprint arXiv:2201.00766.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020). Dark experience for general continual learning: A strong, simple baseline. Advances in Neural Information Processing Systems, 33, 15920–15930.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, (pp. 9650–9660).
Cha, H., Lee, J., & Shin, J. (2021). Co2l: Contrastive continual learning. In ICCV.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., & Ranzato, M. (2019). On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486.
Chen, C., Fu, Z., Chen, Z., Jin, S., Cheng, Z., Jin, X., & Hua, X.-S. (2020). Homm: Higher-order moment matching for unsupervised domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 3422–3429.
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2021). A continual learning survey: Defying forgetting in classification tasks. In PAMI.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., & Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
Douillard, A., Ramé, A., Couairon, G., & Cord, M. (2022). Dytox: Transformers for continual learning with dynamic token expansion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9285–9295.
Fini, E., Da Costa, V.G.T., Alameda-Pineda, X., Ricci, E., Alahari, K., & Mairal, J. (2022). Self-supervised models are continual learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 9621–9630).
Garg, P., Saluja, R., Balasubramanian, V.N., Arora, C., Subramanian, A., & Jawahar, C. (2022). Multi-domain incremental learning for semantic segmentation. In WACV.
Hao, S., Zhou, Y., & Guo, Y. (2020). A brief survey on semantic segmentation with deep learning. Neurocomputing, 406, 302–321.
Hayes, T. L., Cahill, N. D., & Kanan, C. (2019). Memory efficient experience replay for streaming learning. In ICRA.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 16000–16009).
Hendrycks, D., & Dietterich, T. G. (2018). Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.01697.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., & Guo, M., et al. (2021). The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV.
Hou, S., Pan, X., Loy, C.C., Wang, Z., & Lin, D. (2019). Learning a unified classifier incrementally via rebalancing. In CVPR.
Hoyer, L., Dai, D., & Van Gool, L. (2022). Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In CVPR.
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Huang, Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., & Yu, N. (2023). Diversity-aware meta visual prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 10878–10887).
Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Gool, L.V. (2020). Reparameterizing convolutions for incremental multi-task learning without task interference. In ECCV.
Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., & Grabska-Barwinska, A., Hassabis, D. (2017). Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences.
Kundu, J. N., Venkat, N., & Babu, R. V., et al. (2020). Universal source-free domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 4544–4553).
Kundu, J. N., Venkatesh, R.M., Venkat, N., Revanur, A., & Babu, R.V. (2020). Class-incremental domain adaptation. In ECCV.
Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
Li, Z., & Hoiem, D. (2017). Learning without forgetting. PAMI.
Li, X.L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Li, Y., Mao, H., Girshick, R., & He, K. (2022). Exploring plain vision transformer backbones for object detection. In European conference on computer vision, (pp. 280–296). Springer.
Lian, Q., Lv, F., Duan, L., & Gong, B. (2019). Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: A non-adversarial approach. In ICCV.
Liang, J., Hu, D., & Feng, J. (2020). Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, (pp. 6028–6039). PMLR.
Li, S., Liu, C., Lin, Q., Xie, B., Ding, Z., Huang, G., & Tang, J. (2020). Domain conditioned adaptation network. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 11386–11393.
Lin, H., Zhang, Y., Qiu, Z., Niu, S., Gan, C., Liu, Y., & Tan, M. (2022). Prototype-guided continual adaptation for class-incremental unsupervised domain adaptation. In ECCV.
Lomonaco, V., & Maltoni, D. (2017). Core50: A new dataset and benchmark for continuous object recognition. In Conference on robot learning, (pp. 17–26). PMLR.
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing System.
Oren, G., & Wolf, L. (2021). In defense of the learning without forgetting for task incremental learning. In ICCV
Pan, S. J., Tsang, I. W., Kwok, J. T., & Yang, Q. (2010). Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22, 199–210.
Patel, V. M., Gopalan, R., Li, R., & Chellappa, R. (2015). Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32, 53–69.
Pellegrini, L., Graffieti, G., Lomonaco, V., & Maltoni, D. (2020). Latent replay for real-time continual learning. In IROS.
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., & Wang, B. (2019). Moment matching for multi-source domain adaptation. In ICCV.
Prabhu, A., Torr, P.H., & Dokania, P.K. (2020). Gdumb: A simple approach that questions our progress in continual learning. In Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, Proceedings, Part II 16, (pp. 524–540). Springer.
Price, W., & Cohen, I. (2019). Privacy in the age of medical big data. Nature Medicine, 25, 37–43. https://doi.org/10.1038/s41591-018-0272-7
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., & Clark, J. (2021). Learning transferable visual models from natural language supervision. In ICML.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C.H. (2017). icarl: Incremental classifier and representation learning. In CVPR.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv:1606.04671.
Schick, T., & Schütze, H. (2020). Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
Simon, C., Faraki, M., Tsai, Y.-H., Yu, X., Schulter, S., Suh, Y., Harandi, M., & Chandraker, M. (2022). On generalizing beyond domains in cross-domain continual learning. In CVPR.
Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., & Kira, Z. (2023). Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 11909–11919).
Tang, S., Su, P., Chen, D., & Ouyang, W. (2021). Gradient regularized contrastive learning for continual domain adaptation. In AAAI.
Tao, X., Hong, X., Chang, X., & Gong, Y. (2020). Bi-objective continual learning: Learning ‘new’ while consolidating ‘known’. In AAAI.
Ven, G.M., & Tolias, A.S. (2019). Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.
Volpi, R., Larlus, D., & Rogez, G. (2021). Continual adaptation of visual representations via domain randomization and meta-learning. In CVPR.
Wang, Q., Fink, O., Van Gool, L., & Dai, D. (2022). Continual test-time domain adaptation. In CVPR.
Wang, Y., Huang, Z., & Hong, X. (2022). S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. arXiv preprint arXiv:2207.12819.
Wang, Z., Jian, T., Chowdhury, K., Wang, Y., Dy, J., & Ioannidis, S. (2020). Learn-prune-share for lifelong learning. In ICDM.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., & Dy, J., Pfister, T. (2022). Dualprompt: Complementary prompting for rehearsal-free continual learning.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., & Pfister, T. (2022). Learning to prompt for continual learning. In CVPR.
Wang, Q., & Breckon, T. (2020). Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 6243–6250.
Wang, Y., Ma, Z., Huang, Z., Wang, Y., Su, Z., & Hong, X. (2023). Isolation and impartial aggregation: A paradigm of incremental learning without interference. Proceedings of the AAAI Conference on Artificial Intelligence, 37, 10209–10217.
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., & Fu, Y. (2019). Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (pp. 374–382).
Xie, J., Yan, S., & He, X. (2022). General incremental learning with domain-aware categorical representations. In CVPR.
Yang, S., Wang, Y., Van De Weijer, J., Herranz, L., & Jui, S. (2021). Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, (pp. 8978–8987).
Yang, C., Wu, Z., Zhou, B., & Lin, S. (2021). Instance localization for self-supervised detection pretraining. In CVPR.
Zaken, E.B., Ravfogel, S., & Goldberg, Y. (2021). Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.
Zhou, K., Yang, J., Loy, C.C., & Liu, Z. (2022). Conditional prompt learning for vision-language models. In CVPR.
Zhu, Y., Zhang, Z., Wu, C., Zhang, Z., He, T., Zhang, H., Manmatha, R., Li, M., & Smola, A. (2021). Improving semantic segmentation via efficient self-training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(3), 1589–1602.
Zou, Y., Yu, Z., Kumar, B., & Wang, J. (2018). Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62376011, 61925201, 62132001).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Gunhee Kim.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liu, Z., Peng, Y. & Zhou, J. Compositional Prompting for Anti-Forgetting in Domain Incremental Learning. Int J Comput Vis 132, 5783–5800 (2024). https://doi.org/10.1007/s11263-024-02134-3
Received:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1007/s11263-024-02134-3