这是indexloc提供的服务，不要输入任何密码
Skip to content

Add RWKV LoRA defaults and opt-in test #2810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 16, 2025
Merged

Add RWKV LoRA defaults and opt-in test #2810

merged 2 commits into from
Oct 16, 2025

Conversation

@nirbo
Copy link
Contributor

@nirbo nirbo commented Oct 2, 2025

Summary

  • Add rwkv and rwkv7 entries to the LoRA target-module mapping so Transformers-based RWKV checkpoints automatically receive adapters.
  • Document the new behaviour in the README with a short RWKV snippet.
  • Introduce tests/test_rwkv_lora.py, a tiny forward/backward regression guarded by PEFT_RUN_RWKV_TESTS=1 so the Triton compile remains opt-in.

Testing

  make quality

  python -m pytest tests/test_rwkv_lora.py   # (skipped unless PEFT_RUN_RWKV_TESTS=1)

  # To exercise the RWKV-specific test:
  PEFT_RUN_RWKV_TESTS=1 python -m pytest tests/test_rwkv_lora.py

BenjaminBossan
BenjaminBossan requested changes Oct 2, 2025
View reviewed changes
Copy link
Member

@BenjaminBossan BenjaminBossan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding default target modules for RWKV. I have some comments, please take a look.

README.md Outdated
# prints something like: Preheat the oven to 350 degrees and place the cookie dough in a baking dish [...]
```

> [!NOTE]
Copy link
Member

@BenjaminBossan BenjaminBossan Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this README entry can be removed, as we generally don't highlight individual models here.

tests/test_rwkv_lora.py Outdated
@@ -0,0 +1,49 @@
# Copyright 2025-present the HuggingFace Inc. team.
Copy link
Member

@BenjaminBossan BenjaminBossan Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't really a need to add this test here, we don't have tests just to ensure that the default target modules are being set. I did, however, confirm that the test passes locally.

I could see an argument to add it to the general test suite, since RWKV has a different architecture:

peft/tests/test_decoder_models.py

Lines 56 to 66 in 815956b

PEFT_DECODER_MODELS_TO_TEST = [
"hf-internal-testing/tiny-random-OPTForCausalLM",
"hf-internal-testing/tiny-random-GPT2LMHeadModel",
"hf-internal-testing/tiny-random-BloomForCausalLM",
"hf-internal-testing/tiny-random-gpt_neo",
"hf-internal-testing/tiny-random-GPTJForCausalLM",
"hf-internal-testing/tiny-random-GPTBigCodeForCausalLM",
"trl-internal-testing/tiny-random-LlamaForCausalLM",
"peft-internal-testing/tiny-dummy-qwen2",
"hf-internal-testing/tiny-random-Gemma3ForCausalLM",
]

However, at the moment, the PEFT CI is already stressing the rate limit of HF Hub, so adding yet another model would not be a good idea. I think that if this situation relaxes and if we find that there is a big demand for RWKV finetuning with PEFT, we can consider that option.

@BenjaminBossan
Copy link
Member

BenjaminBossan commented Oct 14, 2025

gentle ping @nirbo

@nirbo
Copy link
Contributor Author

nirbo commented Oct 15, 2025

Sorry for the delay, been busy and this slipped my mind.
I've pushed the changes you requested.

@BenjaminBossan

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 16, 2025

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan
BenjaminBossan approved these changes Oct 16, 2025
View reviewed changes
Copy link
Member

@BenjaminBossan BenjaminBossan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding default target modules for RWKV, LGTM. The failing test is unrelated.

@BenjaminBossan BenjaminBossan merged commit 182f4c9 into huggingface:main Oct 16, 2025
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BenjaminBossan BenjaminBossan BenjaminBossan approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nirbo @BenjaminBossan @HuggingFaceDocBuilderDev