Feature request

RoAD is a Parameter-efficient Fine-Tuning technique that is especially well-suited for efficient inference with mixed adapters in a batch, while still providing high output quality with very small parameter count.

RoAD learns 2D rotation matrices that are applied to the layer output which can be written using only element-wise multiplication (rather than matrix multiplication), enabling very fast inference with adapters in unmerged state. It is somewhat related to Orthogonal Finetuning (OFT) method.

Paper: https://arxiv.org/pdf/2409.00119
PEFT implementation: https://github.com/ppetrushkov/peft/tree/road (not from paper authors)

Your contribution

The PEFT implementation should be ready and tested. I created this feature request following the contributing guidelines, if the maintainers are not against including this method, I can proceed with creating a pull request and add the missing documentation and examples.

I also have a fork of VLLM where I implemented and tested the efficiency of this method which shows significantly better performance than LoRA https://github.com/ppetrushkov/vllm/tree/v0.9.1-road