Thank you a lot for this PR to add RoAd to PEFT. I have to say, this is one of the most mature PRs to add a new PEFT method so far, really great work on understanding the PEFT code base and following the conventions.

I did a first review pass and didn't find any big flaws. There are a few smaller issues that I found, but I think they should be easily fixed. Please check my comments.

Something I'd be interested in is if you have checked the efficiency of mixed batch inference. If not, I think this it would be great to have a small script to check this. What I would imagine is to load a base model, add multiple RoAd/LoRA adapters (need not be trained), perform single adapter/batch inference and multi adapter/batch inference, then compare the relative overhead. If you don't have such a script, I'd be happy to help working on this.

Once you finish all your changes, don't forget to run make style before pushing and then ping me.

Regarding benchmarks: It makes sense that users who would be searching for the best performance would look into vLLM etc. Still, I think it's good to check if the advantages also manifest in PEFT.

I created a small toy benchmark:

import time

import torch
from transformers import AutoModelForCausalLM
from peft import get_peft_model, RoadConfig, LoraConfig

num_adapters = 10
num_iters = 50
model_id = "meta-llama/Llama-3.2-3B"

config0 = LoraConfig(init_lora_weights=False, r=4)  # matching params of RoadConfig
config1 = RoadConfig(init_weights=False)
inputs = [torch.randint(0, 100, (num_adapters, 50)).to(0) for _ in range(num_iters)]

for config in [config0, config1]:
    model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(0)
    model = get_peft_model(model, config, adapter_name="adapter0")
    for i in range(1, num_adapters):
        model.add_adapter(f"adapter{i}", config)
    model.eval()

    adapter_names = [f"adapter{i}" for i in range(num_adapters)]

    # warmup
    model(inputs[0])

    # Run the model w/o mixed adapter batch
    durations = {"no mixed batch": [], "   mixed batch": []}

    for i in range(num_iters):
        x = inputs[i]
        tic = time.perf_counter()
        model(x)
        toc = time.perf_counter()
        durations["no mixed batch"].append(toc - tic)

    # Run the model with mixed adapter batch
    for i in range(num_iters):
        x = inputs[i]
        tic = time.perf_counter()
        model(x, adapter_names=adapter_names)
        toc = time.perf_counter()
        durations["   mixed batch"].append(toc - tic)

    print(f"{config.peft_type.value} summary (over 10 runs)")
    for name, dur in durations.items():
        print(f"{name}: avg {sum(dur)/num_iters:.4f} sec, min {min(dur):.4f} sec, max {max(dur):.4f} sec, total: {sum(dur):.4f} sec")

    del model
    torch.cuda.empty_cache()

The results look quite promising so far:

LORA summary (over 10 runs)
no mixed batch: avg 0.0262 sec, min 0.0159 sec, max 0.0272 sec, total: 1.3117 sec
   mixed batch: avg 0.0878 sec, min 0.0848 sec, max 0.1182 sec, total: 4.3920 sec
ROAD summary (over 10 runs)
no mixed batch: avg 0.0263 sec, min 0.0179 sec, max 0.0269 sec, total: 1.3160 sec
   mixed batch: avg 0.0266 sec, min 0.0263 sec, max 0.0269 sec, total: 1.3297 sec

Thanks for addressing my comments, there is not much missing. There is an error in the included example, please check my comments. The of my comments are just smaller changes.

Before we can merge, since we have bnb support, let's also add some GPU tests for this. You can copy the RandLoRA examples and adjust them for RoAD:

PS: I also ran the MetaMathQA benchmark and although the numbers generally look good (memory similar to LoRA rank 32, runtime a bit slower), the test score was significantly lower, only reaching 39.4% accuracy. My first guess is that the learning rate could be a bit too high, but I'm not sure. For the PR, it's not required to improve the score, but maybe you have some ideas.

Thanks for all the changes, this PR looks pretty much ready for me. Please run make style to ensure that the linter is happy.

Thanks a lot for the final effort, the PR now LGTM and can be merged. Great work.