😎Awesome Foundation Models and Federated Learning🧠➕🌍

This repository is primarily based on our survey paper 📚🔍:

Synergizing Foundation Models and Federated Learning: A Survey

Shenghui Li, Fanghua Ye, Meng Fang, Jiaxu Zhao, Yun-Hin Chan, Edith C. H. Ngai, Thiemo Voigt

Unlike smaller models, Foundation Models (FMs) 🧠, such as LLMs and VLMs, are built upon vast amounts of training data 📊. While general FMs can use public data, domain-specific FMs require proprietary data for pre-training and fine-tuning, raising privacy concerns 🔒. Federated Learning (FL) 🤝💻, a compelling privacy-preserving approach, enables collaborative learning across distributed datasets while maintaining data privacy🛡️. Synergizing FM and FL 🧠➕🌍 offers a promising way to address data availability and privacy challenges in FM development, potentially revolutionizing large-scale machine learning in sensitive domains.

🙏If you find this survey useful for your research, please consider citing:

@misc{li2024synergizing,
      title={Synergizing Foundation Models and Federated Learning: A Survey},
      author={Shenghui Li and Fanghua Ye and Meng Fang and Jiaxu Zhao and Yun-Hin Chan and Edith C. -H. Ngai and Thiemo Voigt},
      year={2024},
      eprint={2406.12844},
      archivePrefix={arXiv}
}

Table of Contents

Efficiency

Knowledge Distillation (5)

Title	Venue	Year
Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions	arXiv	2025-01
FedPFT: Federated Proxy Fine-Tuning of Foundation Models	IJCAI	2024-08
FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models	arXiv	2024-06
Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data	arXiv	2024-05
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning	AAAI	2024-03

Selective Tuning (6)

Title	Venue	Year
F\textasciicircum3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics	CVPR	2025-June
Efficient Federated Fine-Tuning of Large Language Models with Layer Dropout	arXiv	2025-03
FedPFT: Federated Proxy Fine-Tuning of Foundation Models	IJCAI	2024-08
FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning	CVPR	2024-06
Federated LoRA with Sparse Communication	arXiv	2024-06
Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation	WWW	2024-05

Additive Tuning

Adapter Tuning (9)

Title	Venue	Year
Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning	ICLR	2025-04
DoFIT: Domain-aware Federated Instruction Tuning with Alleviated Catastrophic Forgetting	NeurIPS	2024-09
Federated Adaptation for Foundation Model-based Recommendations	IJCAI	2024-08
Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization	TASLP	2024-06
Learning Task-Specific Initialization for Effective Federated Continual Fine-Tuning of Foundation Model Adapters	IEEE CAI	2024-06
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning	AAAI	2024-03
Joint Federated Learning and Personalization for on-Device ASR	ASRU	2023-12
Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter	ACL	2023-07
FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning	IEEE DEB	2023-03

Prompt Tuning

Textual Prompt Tuning (6)

Title	Venue	Year
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization	EMNLP	2023-12
Tunable Soft Prompts are Messengers in Federated Learning	EMNLP	2023-12
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10
FedPrompt: Communication-Efficient and Privacy-Preserving Prompt Tuning in Federated Learning	ICASSP	2023-05

Visual Prompt Tuning (4)

Title	Venue	Year
Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning	CVPR	2024-06
Visual Prompt Based Personalized Federated Learning	TMLR	2024-02
Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation	ICCV	2023-10
Learning Federated Visual Prompt in Null Space for MRI Reconstruction	CVPR	2023-06

Textual-Visual Prompt Tuning (8)

Title	Venue	Year
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models	ICLR	2025-04
Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models	ICLR	2025-04
DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning	CVPR	2024-06
Global and Local Prompts Cooperation via Optimal Transport for Federated Learning	CVPR	2024-06
Federated Text-driven Prompt Generation for Vision-Language Models	ICLR	2024-05
Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning	AAAI	2024-03
Dual Prompt Tuning for Domain-Aware Federated Learning	arXiv	2023-10
PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models - Federated Learning in Age of Foundation Model	TMC	2023-08

Reparameterization-Based (26)

Title	Venue	Year
Selective Aggregation for Low-Rank Adaptation in Federated Learning	ICLR	2025-04
Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning	ICLR	2025-04
Federated Residual Low-Rank Adaption of Large Language Models	ICLR	2025-04
Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks	IEEE TWC	2025-01
FedFMSL: Federated Learning of Foundation Models With Sparsely Activated LoRA	IEEE TMC	2024-12
Data Reconstruction and Protection in Federated Learning for Fine-Tuning Large Language Models	IEEE TBD	2024-12
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models	EMNLP	2024-11
Promoting Data and Model Privacy in Federated Learning through Quantized LoRA	EMNLP	2024-11
Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures	arXiv	2024-11
FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients	ECCV	2024-10
Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning	arXiv	2024-10
Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources	NeurIPS	2024-09
Dual-Personalizing Adapter for Federated Foundation Models	NeurIPS	2024-09
Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning	ACM TMIS	2024-08
FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model	KDD	2024-08
Federated LoRA with Sparse Communication	arXiv	2024-06
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering	NAACL	2024-06
Improving LoRA in Privacy-preserving Federated Learning	ICLR	2024-05
FedHLT: Efficient Federated Low-Rank Adaption with Hierarchical Language Tree for Multilingual Modeling	WWW	2024-05
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering	LLMAgents@ICLR	2024-05
FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning	arXiv	2024-04
Towards Building The Federatedgpt: Federated Instruction Tuning	ICASSP	2024-03
Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks	ICASSP	2024-03
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models	FL@FM-NeurIPS	2023-12
pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning	arXiv	2023-10
Low-Parameter Federated Learning with Large Language Models	arXiv	2023-07

Heterogeneous Resource

Lora (7)

Title	Venue	Year
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models	EMNLP	2024-11
Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures	arXiv	2024-11
FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients	ECCV	2024-10
Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning	arXiv	2024-10
Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources	NeurIPS	2024-09
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering	NAACL	2024-06
pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning	arXiv	2023-10

Split Learning (9)

Title	Venue	Year
Split Fine-Tuning for Large Language Models in Wireless Networks	arXiv	2025-01
Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions	arXiv	2025-01
Safely Learning with Private Data: A Federated Learning Framework for Large Language Model	EMNLP	2024-11
SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models	arXiv	2024-07
Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models	ICML	2024-07
Safely Learning with Private Data: A Federated Learning Framework for Large Language Model	arXiv	2024-06
Heterogeneous Federated Learning with Splited Language Model	arXiv	2024-03
Privacy-Preserving Split Learning for Large-Scaled Vision Pre-Training	TIFS	2023-02
Federated Split BERT for Heterogeneous Text Classification	IJCNN	2022

Sparsification (6)

Title	Venue	Year
Data Reconstruction and Protection in Federated Learning for Fine-Tuning Large Language Models	IEEE TBD	2024-12
Promoting Data and Model Privacy in Federated Learning through Quantized LoRA	EMNLP	2024-11
Federated LoRA with Sparse Communication	arXiv	2024-06
Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent	arXiv	2024-06
Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation	WWW	2024-05
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models	FL@FM-NeurIPS	2023-12

Zeroth-Order Optimization (9)

Title	Venue	Year
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization	ICLR	2025-04
FeedSign: Robust Full-parameter Federated Fine-tuning of Large Models with Extremely Low Communication Overhead of One Bit	arXiv	2025-01
On the Convergence of Zeroth-Order Federated Tuning for Large Language Models	KDD	2024-08
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes	ICML	2024-07
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences	ATC	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning	arXiv	2023-10
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10

Adaptability

Domain-Centric Adaptation

Multi-Domain Adaptation (6)

Title	Venue	Year
Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning	ICLR	2025-04
FedCSR: A Federated Framework for Multi-Platform Cross-Domain Sequential Recommendation with Dual Contrastive Learning	COLING	2025-01
DoFIT: Domain-aware Federated Instruction Tuning with Alleviated Catastrophic Forgetting	NeurIPS	2024-09
Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization	TASLP	2024-06
DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning	CVPR	2024-06
Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning	AAAI	2024-03

Client-Centric Adaptation

Preference-Aware Adaptation (3)

Title	Venue	Year
FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF	AAMAS	2025-05
Towards Federated RLHF with Aggregated Client Preference for LLMs	ICLR	2025-04
Federated Fine-Tuning of Large Language Models: Kahneman-Tversky vs. Direct Preference Optimization	arXiv	2025-02

Personalization (4)

Title	Venue	Year
FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF	AAMAS	2025-05
FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning	CVPR	2024-06
pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning	arXiv	2023-10
FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning	IEEE DEB	2023-03

Clustering (3)

Title	Venue	Year
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering	NAACL	2024-06
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering	LLMAgents@ICLR	2024-05
FedKC: Federated Knowledge Composition for Multilingual Natural Language Understanding	WWW	2022-04

Trustworthiness

IP Protection

Black-Box Tuning (4)

Title	Venue	Year
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning	arXiv	2023-10
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10

Privacy Preservation

Privacy Attack (6)

Title	Venue	Year
Data Reconstruction and Protection in Federated Learning for Fine-Tuning Large Language Models	IEEE TBD	2024-12
Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models	NeurIPS	2024-09
Revisiting Data Reconstruction Attacks on Real-world Dataset for Federated Natural Language Understanding	LREC-COLING	2024-05
Analysis of Privacy Leakage in Federated Large Language Models	AISTATS	2024-05
Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation	ICLR	2023-03
Recovering Private Text in Federated Learning of Language Models	NeurIPS	2022-11

Privacy-Preserving Techniques (7)

Title	Venue	Year
Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models	ICLR	2025-04
Data Reconstruction and Protection in Federated Learning for Fine-Tuning Large Language Models	IEEE TBD	2024-12
Promoting Data and Model Privacy in Federated Learning through Quantized LoRA	EMNLP	2024-11
Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning	ACM TMIS	2024-08
PrE-Text: training language models on private federated data in the age of LLMs	ICML	2024-07
Improving LoRA in Privacy-preserving Federated Learning	ICLR	2024-05
Federated Learning of Gboard Language Models with Differential Privacy	ACL	2023-07

Attack Robustness

Poisoning Attack (8)

Title	Venue	Year
Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models	ICLR	2025-01
Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation	arXiv	2025-01
SecFFT: Safeguarding Federated Fine-Tuning for Large Vision Language Models against Covert Backdoor Attacks in IoRT Networks	IEEE IoTJ	2024-12
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning	arXiv	2024-11
Securing Federated Learning Against Novel and Classic Backdoor Threats During Foundation Model Integration	arXiv	2024-10
SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning	arXiv	2024-09
Unveiling Backdoor Risks Brought by Foundation Models in Heterogeneous Federated Learning	PAKDD	2024-04
Backdoor Threats from Compromised Foundation Models to Federated Learning	FL@FM-NeurIPS	2023-10

Application

Multilingualism (9)

Title	Venue	Year
Real world federated learning with a knowledge distilled transformer for cardiac CT imaging	NPJ Digital Medicine	2025-02
Generalizable Multilingual Hate Speech Detection on Low Resource Indian Languages using Fair Selection in Federated Learning	NAACL	2024-06
Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages	ICLR	2024-05
Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation	WWW	2024-05
FedHLT: Efficient Federated Low-Rank Adaption with Hierarchical Language Tree for Multilingual Modeling	WWW	2024-05
Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter	ACL	2023-07
Federated Multilingual Models for Medical Transcript Analysis	CHIL	2023-06
Pretrained Models for Multilingual Federated Learning	NAACL	2022-07
FedKC: Federated Knowledge Composition for Multilingual Natural Language Understanding	WWW	2022-04

Speech (4)

Title	Venue	Year
Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks	ICASSP	2024-03
Joint Federated Learning and Personalization for on-Device ASR	ASRU	2023-12
Importance of Smoothness Induced by Optimizers in Fl4Asr: Towards Understanding Federated Learning for End-To-End ASR	ASRU	2023-12
Federated Learning for Speech Recognition: Revisiting Current Trends Towards Large-Scale ASR	FL@FM-NeurIPS	2023-12

Recommendation Systems (7)

Title	Venue	Year
FedCSR: A Federated Framework for Multi-Platform Cross-Domain Sequential Recommendation with Dual Contrastive Learning	COLING	2025-01
Federated Adaptation for Foundation Model-based Recommendations	IJCAI	2024-08
Navigating the Future of Federated Recommendation Systems with Foundation Models	arXiv	2024-06
Prompt-enhanced Federated Content Representation Learning for Cross-domain Recommendation	WWW	2024-05
Federated Recommendation via Hybrid Retrieval Augmented Generation	arXiv	2024-03
LLM-based Federated Recommendation	arXiv	2024-02
TransFR: Transferable Federated Recommendation with Pre-trained Language Models	arXiv	2024-02

Domain Specific (10)

Title	Venue	Year
Real world federated learning with a knowledge distilled transformer for cardiac CT imaging	NPJ Digital Medicine	2025-02
Federated Prompt Learning for Weather Foundation Models on Devices	IJCAI	2024-08
Foundation models matter: federated learning for multi-center tuberculosis diagnosis via adaptive regularization and model-contrastive learning	WWW	2024-05
FedFMS: Exploring Federated Foundation Models for Medical Image Segmentation	MICCAI	2024
Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Data	IJCAI	2023-8
FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning	EMNLP	2023-12
FedJudge: Federated Legal Large Language Model	arXiv	2023-09
FEDLEGAL: The First Real-World Federated Learning Benchmark for Legal NLP	ACL	2023-07
Learning Federated Visual Prompt in Null Space for MRI Reconstruction	CVPR	2023-06
Federated Multilingual Models for Medical Transcript Analysis	CHIL	2023-06

Resources

Surveys (9)

Title	Venue	Year
Advances and Open Challenges in Federated Learning with Foundation Models	IEEE COMST	2025-01
Towards Federated Large Language Models: Motivations, Methods, and Future Directions	IEEE COMST	2024-10
Federated Large Language Model: Solutions, Challenges and Future Directions	IEEE WC	2024-10
Federated Large Language Models: Current Progress and Future Directions	arXiv	2024-09
A Survey on Efficient Federated Learning Methods for Foundation Model Training	IJCAI	2024-08
Synergizing Foundation Models and Federated Learning: A Survey	arXiv	2024-06
Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models	LREC-COLING	2024-05
The Role of Federated Learning in a Wireless World with Foundation Models	IEEE WC	2024
When foundation model meets federated learning: Motivations, challenges, and future directions	arXiv	2023-06

Frameworks (11)

Title	Venue	Year
FederatedScope: A Flexible Federated Learning Platform for Heterogeneity	VLDB	2023-09
FedLab: A Flexible Federated Learning Framework	JMLR	2023-01
OpenFL: the open federated learning library	PMB	2022-10
NVIDIA FLARE: Federated Learning from Simulation to Real-World	FL@NeurIPS	2022-07
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale	ICML	2022-07
Scalable federated machine learning with FEDn	CCGrid	2022-05
FLUTE: A Scalable Extensible Framework for High-Performance Federated Learning Simulations	FL@NeurIPS	2022-03
FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection	JMLR	2021-08
Pysyft: A library for easy federated learning	FLS	2021-06
Flower: A friendly federated learning research framework	Arxiv	2020-07
FedML: A Research Library and Benchmark for Federated Machine Learning	SpicyFL	2020-07