validated cpt, ephemeral_gpu_offloading and eva finetuning on XPU #2694
Conversation
Signed-off-by: Yao, Matrix <matrix.yao@intel.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
| ## Leveraging multiple accelerators | ||
|
|
||
| EVA initialization can be parallelized across multiple GPUs. In this case inputs from multiple GPUs are gathered before computing the SVD for the batch. This requires that the model is wrapped in a `torch.nn.DataParallel` or `torch.nn.DistributedDataParallel` class. An example of how to use this can be found in [eva_finetuning_multi_gpu.py](https://github.com/huggingface/peft/blob/main/examples/eva_finetuning/eva_finetuning_multi_gpu.py). | ||
| EVA initialization can be parallelized across multiple accelerators. In this case inputs from multiple accelerators are gathered before computing the SVD for the batch. This requires that the model is wrapped in a `torch.nn.DataParallel` or `torch.nn.DistributedDataParallel` class. An example of how to use this can be found in [eva_finetuning_multi_accelerator.py](https://github.com/huggingface/peft/blob/main/examples/eva_finetuning/eva_finetuning_multi_accelerator.py). |
There was a problem hiding this comment.
Just wondering: Are Intel accelerators not still technically "GPUs"? Is it really needed to replace "GPU" in the docs and comments with "accelerator"?
There was a problem hiding this comment.
Yes, Intel XPU are still technical GPUs architecture, which is general-purpose and programmable. I am using the term accelerator by following PyTorch's vocabulary(here), PyTorch are supporting more accelerators, so I am using accelerator here, so if some people validated another new device later, they don't need to change README anymore, and even not eva_finetuning.py. Pls let me know your opinions, thx.
There was a problem hiding this comment.
That makes sense, thanks for explaining.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
BenjaminBossan
left a comment
There was a problem hiding this comment.
Thanks for checking that CPT, EVA, and GPU offloading work with XPU
| ## Leveraging multiple accelerators | ||
|
|
||
| EVA initialization can be parallelized across multiple GPUs. In this case inputs from multiple GPUs are gathered before computing the SVD for the batch. This requires that the model is wrapped in a `torch.nn.DataParallel` or `torch.nn.DistributedDataParallel` class. An example of how to use this can be found in [eva_finetuning_multi_gpu.py](https://github.com/huggingface/peft/blob/main/examples/eva_finetuning/eva_finetuning_multi_gpu.py). | ||
| EVA initialization can be parallelized across multiple accelerators. In this case inputs from multiple accelerators are gathered before computing the SVD for the batch. This requires that the model is wrapped in a `torch.nn.DataParallel` or `torch.nn.DistributedDataParallel` class. An example of how to use this can be found in [eva_finetuning_multi_accelerator.py](https://github.com/huggingface/peft/blob/main/examples/eva_finetuning/eva_finetuning_multi_accelerator.py). |
There was a problem hiding this comment.
That makes sense, thanks for explaining.
* add eval loss logging during predition * make sure the train and eval logs aren't mixed * test grpo in eval * fix tests --------- Co-authored-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co>
@BenjaminBossan , pls help review, thx very much.