这是indexloc提供的服务，不要输入任何密码
Skip to main content
Springer Nature Link
Log in

Anti-Fake Vaccine: Safeguarding Privacy Against Face Swapping via Visual-Semantic Dual Degradation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Deepfake techniques pose a significant threat to personal privacy and social security. To mitigate these risks, various defensive techniques have been introduced, including passive methods through fake detection and proactive methods through adding invisible perturbations. Recent proactive methods mainly focus on face manipulation but perform poorly against face swapping, as face swapping involves the more complex process of identity information transfer. To address this issue, we develop a novel privacy-preserving framework, named Anti-Fake Vaccine, to protect the facial images against the malicious face swapping. This new proactive technique dynamically fuses visual corruption and content misdirection, significantly enhancing protection performance. Specifically, we first formulate constraints from two distinct perspectives: visual quality and identity semantics. The visual perceptual constraint targets image quality degradation in the visual space, while the identity similarity constraint induces erroneous alterations in the semantic space. We then introduce a multi-objective optimization solution to effectively balance the allocation of adversarial perturbations generated according to these constraints. To further improving performance, we develop an additive perturbation strategy to discover the shared adversarial perturbations across diverse face swapping models. Extensive experiments on the CelebA-HQ and FFHQ datasets demonstrate that our method exhibits superior generalization capabilities across diverse face swapping models, including commercial ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data Availibility

The data that support the findings of this study are available from the authors upon reasonable request.

Notes

  1. https://vision.aliyun.com/facebody.

  2. https://ai.baidu.com/tech/face/merge.

  3. https://cloud.tencent.com/product/facefusion.

  4. https://github.com/Oldpan/Faceswap-Deepfake-Pytorch.

References

  • Chen, R., Chen, X., Ni, B., & Ge, Y., (2020) Simswap: An efficient framework for high fidelity face swapping. In Proceedings of the 28th ACM International conference on multimedia, pp. 2003–2011.

  • Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L. (2022). Voice-face homogeneity tells deepfake. arXiv preprint arXiv:2203.02195

  • Deb, D., Zhang, J., & Jain, A. K. (2020). Advfaces: Adversarial face synthesis. In 2020 IEEE international joint conference on biometrics (IJCB), pp. 1–10. IEEE.

  • Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019). Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), June.

  • Frank, J., & Holz, T. (2021). [RE] CNN-generated images are surprisingly easy to spot... for now. arXiv preprint arXiv:2104.02984

  • Gao, G., Huang, H., Fu, C., Li, Z., & He, R. (2021). Information bottleneck disentanglement for identity swapping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3404–3413.

  • Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. Statistics, 1050, 20.

    MATH  Google Scholar 

  • He, Z., Zuo, W., Kan, M., Shan, S., & Chen, X. (2019). Attgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(11), 5464–5478.

    MathSciNet  Google Scholar 

  • Huang, H., Wang, Y., Chen, Z., Zhang, Y., Li, Y., Tang, Z., Chu, W., Chen, J., Lin, W., & Ma, K-K. (2022). Cmua-watermark: A cross-model universal adversarial watermark for combating deepfakes. In Proceedings of the AAAI conference on artificial intelligence, vol. 36, pp. 989–997.

  • Huang, Q., Zhang, J., Zhou, W., Zhang, W., & Nenghai, Y. (2021). Initiative defense against facial manipulation. In Proceedings of the AAAI conference on artificial intelligence, vol. 35, pp. 1619–1627.

  • Ilyas, H., Javed, A., & Malik, K. M. (2023). Avfakenet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection. Applied Soft Computing, 136, 110124.

    MATH  Google Scholar 

  • Jiang, D., Song, D., Tong, R., & Tang, M. (2023). Styleipsb: Identity-preserving semantic basis of stylegan for high fidelity face swapping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 352–361.

  • Juefei-Xu, F., Wang, R., Huang, Y., Guo, Q., Ma, L., & Liu, Y. (2022). Countering malicious deepfakes: Survey, battleground, and horizon. International Journal of Computer Vision, 130(7), 1678–1734.

    MATH  Google Scholar 

  • Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401–4410.

  • Kim, J., Lee, J., & Zhang, B-T., (2022). Smooth-swap: A simple enhancement for face-swapping with smoothness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10779–10788.

  • Kim, M., Tariq, S., & Woo, S. S. (2021). Fretal: Generalizing deepfake detection using knowledge distillation and representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1001–1012.

  • Kuhn, H. W., & Tucker, A. W. (2013). Nonlinear programming. In Traces and emergence of nonlinear programming, pp. 247–258. Springer.

  • Lee, J., Hyung, J., Jeong, S., & Choo, J. (2024). Selfswapper: Self-supervised face swapping via shape agnostic masked autoencoder. arXiv preprint arXiv:2402.07370

  • Lee, C. H., Liu, Z., Wu, L., & Luo, P. (2020). Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5549–5558.

  • Li, Y., Bai, S., Xie, C., Liao, Z., Shen, X., & Yuille, A. (2020). Regional homogeneity: Towards learning transferable universal adversarial perturbations against defenses. In Computer Vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pp. 795–813. Springer.

  • Li, L., Bao, J., Yang, H., Chen, D., & Wen, F. (2020). Advancing high fidelity identity swapping for forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5074–5083.

  • Li, Z., Yu, N., Salem, A., Backes, M., Fritz, M., & Zhang, Y. (2022). Unganable: Defending against gan-based face manipulation. arXiv preprint arXiv:2210.00957

  • Li, X., Zhang, S., Hu, J., Cao, L., Hong, X., Mao, X., Huang, F., Wu, Y., & Ji, R. (2021). Image-to-image translation via hierarchical style disentanglement. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8639–8648.

  • Lin, X., Chen, H., Pei, C., Sun, F., Xiao, X., Sun, H., Zhang, Y., Ou, W., & Jiang, P. (2019). A pareto-efficient algorithm for multiple objective optimization in e-commerce recommendation. In Proceedings of the 13th ACM conference on recommender systems, pp. 20–28.

  • Liu, Z., Li, M., Zhang, Y., Wang, C., Zhang, Q., Wang, J., & Nie, Y. (2023a). Fine-grained face swapping via regional gan inversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8578–8587.

  • Liu, K., Perov, I., Gao, D., Chervoniy, N., Zhou, W., & Zhang, W. (2023b). Deepfacelab: Integrated, flexible and extensible face-swapping framework. Pattern Recognition, 141, 109628.

  • Li, J., Xie, H., Lingyun, Y., Gao, X., & Zhang, Y. (2021). Discriminative feature mining based on frequency information and metric learning for face forgery detection. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12167–12180.

    MATH  Google Scholar 

  • Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In International conference on learning representations.

  • Nakka, K.K., & Salzmann, M. (2021). Learning transferable adversarial perturbations. In NeurIPS.

  • Naruniec, J., Helminger, L., Schroers, C., & Weber, R. M. (2020). High-resolution neural face swapping for visual effects. Computer Graphics Forum, 39, 173–184.

    Google Scholar 

  • Nirkin, Y., Keller, Y., & Hassner, T. (2019). Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7184–7193.

  • Otto, C., Naruniec, J., Helminger, L., Etterlin, T., Mignone, G., Chandran, P., Zoss, G., Schroers, C., Gross, M., Gotardo, P., et al. (2022). Learning dynamic 3d geometry and texture for video face swapping. Computer Graphics Forum, 41, 611–622.

    Google Scholar 

  • Poursaeed, O., Katsman, I., Gao, B., & Belongie, S.(2018). Generative adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4422–4431.

  • Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020). Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European conference on computer vision, pp. 86–103. Springer.

  • Rosberg, F., Aksoy, E. E., Alonso-Fernandez, F., & Englund, C. (2023) Facedancer: Pose- and occlusion-aware high fidelity face swapping. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (WACV), pp. 3454–3463.

  • Ruiz, N., Bargal, S. A., & Sclaroff, S. (2020). Disrupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems. In Computer Vision–ECCV 2020 workshops: Glasgow, UK, August 23–28, 2020, proceedings, Part IV 16, pp. 236–251. Springer.

  • Ruiz, N., Bargal, S. A., Xie, C., & Sclaroff, S. (2023). Practical disruption of image translation deepfake networks. In Proceedings of the AAAI conference on artificial intelligence, vol. 37, pp. 14478–14486.

  • Sener, O., & Koltun, V. (2018). Multi-task learning as multi-objective optimization. Advances in Neural Information Processing Systems, 31.

  • Shao, R., Lan, X., & Yuen, P. C. (2020). Regularized fine-grained meta face anti-spoofing. In Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 11974–11981.

  • Shen, Y., Yang, C., Tang, X., & Zhou, B. (2022). Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4), 2004–2018.

    MATH  Google Scholar 

  • Shiohara, K., Yang, X., & Taketomi, T. (2023). Blendface: Re-designing identity encoders for face-swapping. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7634–7644.

  • Sun, Y., Yu, L., Xie, H., Li, J., & Zhang, Y. (2024). Diffam: Diffusion-based adversarial makeup transfer for facial privacy protection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24584–24594.

  • Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017). Synthesizing obama: Learning lip sync from audio. ACM Transactions on Graphics (ToG), 36(4), 1–13.

    Google Scholar 

  • Teotia, D., Lapedriza, A., & Ostadabbas, S. (2022). Interpreting face inference models using hierarchical network dissection. International Journal of Computer Vision, 130(5), 1277–1292.

    Google Scholar 

  • Thambawita, V., Isaksen, J. L., Hicks, S. A., Ghouse, J., Ahlberg, G., Linneberg, A., Grarup, N., Ellervik, C., Olesen, M. S., Hansen, T., et al. (2021). Deepfake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine. Scientific Reports, 11(1), 21896.

    Google Scholar 

  • Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016). Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2387–2395.

  • Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., & Cohen-Or, D. (2021). Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4), 1–14.

    Google Scholar 

  • Tripathy, S., Kannala, J., & Rahtu, E. (2020). Icface: Interpretable and controllable face reenactment using gans. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3385–3394.

  • Ververas, E., & Zafeiriou, S. (2020). Slidergan: Synthesizing expressive face images by sliding 3d blendshape parameters. International Journal of Computer Vision, 128(10–11), 2629–2650.

    Google Scholar 

  • Wang, R., Huang, Z., Chen, Z., Liu, L., Chen, J., & Wang, L. (2022). Anti-forgery: Towards a stealthy and robust deepfake disruption attack via adversarial perceptual-aware perturbations. arXiv preprint arXiv:2206.00477

  • Wang, R., Juefei-Xu, F., Luo, M., Liu, Y., & Wang, L. (2021). Faketagger: Robust safeguards against deepfake dissemination via provenance tracking. In Proceedings of the 29th ACM international conference on multimedia, pp. 3546–3555.

  • Wiles, O., Koepke, A., & Zisserman, A. (2018). X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pp. 670–686.

  • Wu, P-W., Lin, Y-J., Chang, C-H., Chang, E. Y , Liao, S-W. (2019). Relgan: Multi-domain image-to-image translation via relative attributes. In Proceedings of the IEEE international conference on computer vision, pp. 5914–5922.

  • Xu, Y., Deng, B., Wang, J., Jing, Y., Pan, J., & He, S. (2022). High-resolution face swapping via latent semantics disentanglement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7642–7651.

  • Yang, C., Ding, L., Chen, Y., & Li, H. (2021). Defending against gan-based deepfake attacks via transformation-aware adversarial faces. In 2021 international joint conference on neural networks (IJCNN), pp. 1–8. IEEE.

  • Yang, X., Dong, Y., Pang, T., Su, H., Zhu, J., Chen, Y., & Xue, H. (2021). Towards face encryption by generating adversarial identity masks. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp. 3897–3907, October.

  • Yeh, C. Y., Chen, H.-W., Tsai, S.-L., & Wang, S.-D. (2020). Disrupting image-translation-based deepfake algorithms with adversarial attacks. In Proceedings of the IEEE/CVF winter conference on applications of computer vision workshops, pp. 53–62.

  • Yin, F., Zhang, Y., Wu, B., Feng, Y., Zhang, J., Fan, Y., & Yang, Y. (2023). Generalizable black-box adversarial attack with meta learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(3), 1804–1818.

    MATH  Google Scholar 

  • Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595.

  • Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021). Learning self-consistency for deepfake detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15023–15033.

  • Zhao, H., Zhou, W., Chen, D., Wei, T., Zhang, W., & Yu, N. (2021). Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2185–2194.

  • Zhu, Y., Li, Q., Wang, J., Xu, C-Z., & Sun, Z. (2021). One shot face swapping on megapixels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4834–4844.

Download references

Acknowledgements

This work was supported in part by the National Key R &D Program of China under (Grant No. 2021YFB3100800); in part by the National Natural Science Foundation of China (No. 62306308, 62025604, U22A2030, 62372448), in part by Hunan Provincial Funds for Distinguished Young Scholars (Grant No. 2024JJ2025), in part by the Foundation of Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education (No. EIN2024B004).

Author information

Authors and Affiliations

  1. State Key Laboratory of Information Security, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

    Jingzhi Li, Changjiang Luo & Hua Zhang

  2. Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan

    Yang Cao

  3. College of Computer Science and Electronic Engineering, Hunan University, Changsha, China

    Xin Liao

  4. School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China

    Xiaochun Cao

  5. Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education, Kunming, China

    Jingzhi Li

Authors
  1. Jingzhi Li
    View author publications

    Search author on:PubMed Google Scholar

  2. Changjiang Luo
    View author publications

    Search author on:PubMed Google Scholar

  3. Hua Zhang
    View author publications

    Search author on:PubMed Google Scholar

  4. Yang Cao
    View author publications

    Search author on:PubMed Google Scholar

  5. Xin Liao
    View author publications

    Search author on:PubMed Google Scholar

  6. Xiaochun Cao
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Jingzhi Li.

Additional information

Communicated by Segio Escalera.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Luo, C., Zhang, H. et al. Anti-Fake Vaccine: Safeguarding Privacy Against Face Swapping via Visual-Semantic Dual Degradation. Int J Comput Vis 133, 2025–2043 (2025). https://doi.org/10.1007/s11263-024-02259-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11263-024-02259-5

Keywords