The scarcity of high-quality multimodal biomedical data limits the ability to effectively fine-tune pretrained Large Language Models (LLMs) for specialized biomedical tasks. To address this challenge, we introduce MINT (Multimodal Integrated kNowledge Transfer), a framework that aligns unimodal large models using multimodal knowledge transfer to improve rare disease prediction and tissue type classification.

Note:

• Due to privacy restrictions, the GMDB dataset used in this study cannot be publicly released.
• The training was performed on a SLURM-managed HPC cluster using 4×A100 GPUs.
• We will continue to update the GitHub repository to provide scripts and documentation for HPC-based deployment.
• Additionally, we provide pretrained model weights for immediate evaluation and reproducibility.

Arxiv Preprint: Wu, D., Wang, Z., Nguyen, Q., Xu, Z., Wang, K., Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications, arXiv: 2505.05736