This repository provides the code for the paper Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation
Create a Python virtual environment and install the dependencies:
conda create -n magical python=3.11 -y
conda activate magical
pip install -e .1.Collect Activations: collect activations of expert text and lay text with the following command:
cd scripts
python extract.py \
--model_name meta-llama/Llama-3.1-8B-Instruct \
--datasets "['elife', 'cochrane', 'genetics']"2.Semantic-Relevant Layer Identification: train probes to identify semantic-relevant layer with the following command:
python train_probes.py \
--data_dir ../data/probes/Llama-3.1-8B-Instruct3.Train Magical: train with the following command:
deepspeed train.py \
--model_name "meta-llama/Llama-3.1-8B-Instruct" \
--datasets "['elife', 'cochrane', 'genetics']" \
--max_seq_len 2048 \
--num_workers 16 \
--output_dir ../out \
--logging_steps 2 \
--save_steps 100 \
--wandb_enable \
--wandb_run_name "elife_cochrane_genetics" \
--num_train_epochs 5 \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 1 \
--learning_rate 1e-4 \
--weight_decay 0 \
--warmup_steps 100 \
--gradient_clipping 1.0 \
--lora_r 8 \
--num_loras 3 \
--target_modules "['up_proj','down_proj','gate_proj','q_proj','v_proj','o_proj','k_proj']" \
--contrastive_loss_weight 0.5 \
--contrastive_temp 0.5 \
--contrastive_targets_path "../data/probes/Llama-3.1-8B-Instruct/contrastive_targets.pkl" \
--num_contrastive_targets 32 \
--deepspeed_config ../configs/deepspeed_zero2_offload_bf16.jsonIf you find this repo useful, please kindly cite our paper.
@misc{liao2025magicalmedicallaylanguage,
title={Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation},
author={Weibin Liao and Tianlong Wang and Yinghao Zhu and Yasha Wang and Junyi Gao and Liantao Ma},
year={2025},
eprint={2508.08730},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2508.08730},
}