Feature request

This pull request aims to introduce support for quantization in the VeRA PEFT method. This enhancement is in response to the issue #1921, which highlighted the need for compatibility between VeRA and quantized models.

Motivation

The motivation behind this feature request is to extend the functionality of the VeRA method to support models that have been quantized using techniques such as bitsandbytes. Currently, users face issues with dimension mismatches and incorrect tensor shapes when applying VeRA to quantized models. By adding support for quantization, we can improve the usability and flexibility of VeRA, making it a more powerful tool for a broader range of use cases.

Your contribution

I am willing to submit a pull request with the implementation of this feature. The PR will include:

• Code changes to support quantized weights in the VeRA method.
• (Possible) Tests to validate the correct handling of quantized models.
• Updated documentation and examples demonstrating the new functionality.

Please let me know if there are any additional requirements or considerations for this feature. I look forward to contributing this enhancement to the project and am open to any feedback or suggestions. @BenjaminBossan