[FEAT] Add C3A Support #2577

Phoveran · 2025-06-09T07:51:33Z

currently, I fail to run the test by the command make test, and the output is:

Thank you for your time and guidance!

BenjaminBossan · 2025-06-10T14:14:43Z

Hi, thanks for the PR. I didn't have time to look at it yet or to read the paper, but regarding your question, I assume that you would like to add unit tests and that's where you encounter it. The reason is that you're missing the pytest-cov package in your environment.

Phoveran · 2025-06-16T05:04:29Z

Hi Benjamin,

I've passed the test cases and done make style. Would be kind to review the code? Thank you!

BenjaminBossan

Thanks a lot for adding C³A to PEFT. The method looks very interesting and promising. I'll try to find some time next week to do a bit of my own testing.

The implementation itself is already nearly perfect, I just added a few smaller comments, please check.

On top of that, before we merge, let's also work:

Add documentation about C³A
Add one example
Extend the tests a bit: Please add a test case to test_decoder_models.py, test_encoder_decoder_models.py, test_feature_extraction_models.py, test_seq_classifier.py, and test_stablediffusion.py. This should be straightforward as it only requires adding a test config similar to what you did in test_custom_models.py. However, due to the shape constraint of C³A, some of these models may not work. In that case, let's just add a comment that says something like "C³A could not be tested because the input and output shapes are ...".

Out of curiosity, is there any relation to FourierFT?

src/peft/tuners/c3a/__init__.py

src/peft/tuners/c3a/config.py

src/peft/tuners/c3a/layer.py

src/peft/tuners/c3a/config.py

src/peft/tuners/c3a/layer.py

src/peft/tuners/c3a/utils.py

review-notebook-app · 2025-06-18T03:28:16Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Phoveran · 2025-06-18T03:33:39Z

Hi Benjamin,

I have added more testing, a doc and an example according to your guidance. Besides, issues you listed should have been addressed. Let me know if you need further changes.

Actually, one of the original goal of this work is to make FourierFT mathematically elegant and efficient. :D

HuggingFaceDocBuilderDev · 2025-06-18T08:59:26Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan

Thanks for adding the tests, docs, example, and making the requested adjustments. Tests are passing (ignore the MacOS tests which fail for unrelated reasons).

I have a few small comments, but nothing major.

src/peft/tuners/c3a/config.py

src/peft/tuners/c3a/layer.py

tests/test_initialization.py

tests/test_decoder_models.py

src/peft/tuners/c3a/layer.py

examples/sequence_classification/C3A.ipynb

Phoveran · 2025-06-20T01:52:39Z

Hi Benjamin,

Thank you for these comments! I believe I have properly addressed them :D. Please feel free to leave messages for further concerns!

BenjaminBossan · 2025-06-20T10:43:06Z

@Phoveran With Python 3.9, there is the following error:

src/peft/tuners/c3a/layer.py:115: in C3ALinear
    init_weights: bool | Literal["gaussian", "kaiming_uniform", "xavier_uniform"],
E   TypeError: unsupported operand type(s) for |: 'type' and '_LiteralGenericAlias'

I think a from __future__ import annotations import should fix it.

BenjaminBossan

Thanks for fixing the last issue.

IMO the PR is ready to be merged, but before doing that, I wanted to discuss one matter. In PEFT, we have now added a benchmark for all PEFT method. I added C³A using the default config settings and started a run but then canceled after 2500 steps. The issue was that the loss was stagnating at around 1.3 and the valid accuracy stayed at 0. Other PEFT methods have much better scores. Probably the default settings are not working well. Do you have suggestions for how to adjust the hyper-parameters? Perhaps, it also means we need to update the default values.

(Btw. we have similarly bad scores for FourierFT, maybe you know some better settings for this model as well)

src/peft/tuners/c3a/config.py

Phoveran · 2025-06-24T01:47:46Z

Thanks for raising the benchmarking issue.

Indeed, we found that the usually used training config does not fit FourierFT nor C3A. The main difference should be the choice on LR. We refer to the Appendix part of these papers FourierFT & C3A for more LR details. In general, you might try 5e-1 for C3A and 3e-3 for FourierFT.

Thank you and please let me know if there's any further issue!

BenjaminBossan · 2025-06-24T13:24:43Z

The main difference should be the choice on LR. We refer to the Appendix part of these papers FourierFT & C3A for more LR details. In general, you might try 5e-1 for C3A and 3e-3 for FourierFT.

Thanks for the guidance. Could you please update the C³A docs to mention that a much higher learning rate than usual is needed? Otherwise, many users who try it might bounce because the model doesn't train well.

I ran an experiment locally with lr 0.5 and got much better results. That said, they're still not on par with LoRA (rank 32):

metric	LoRA rank 32	C³A w/ lr 0.5
total time	1847 sec	2102 sec
trainable params	9175040	1376256
cuda max reserved memo	22.3 GB	22.0 GB
train loss	0.607	0.659
test accuracy	0.478	0.412

It could be due to the number of parameters, but that's just my hunch.

Anyway, it is not critical for this PR, but I think going forward, it would be nice to provide good experimental settings and defaults so that users can get started as quickly as possible.

Phoveran · 2025-06-25T02:20:26Z

Thanks for the feedback! I have manually wrapped strings and updated the doc. I will work on a proper experimental setting ASAP.

Phoveran · 2025-06-27T05:34:55Z

Hi Benjamin,

I have done some experiments and found out one satisfying experimental setting. The setting and results are uploaded. (You might find the meta info part blank since I'm using a machine that is not allowed to connect to the public network and I'm using offline HF.) Besides, more instructions are added.

In my experiment, I found out that C3A excels in accuracy with only nearly half parameters compared with LoRA, and this results in a much smaller adapter file.

BenjaminBossan · 2025-06-27T08:43:11Z

Thanks a lot @Phoveran for running those experiments, discovering a good setting, and documenting how users can choose better hyper-parameters. As for the result file, could you please remove it? We will run the experiment on our machines to ensure that all experiments run under comparable conditions.

BenjaminBossan · 2025-06-27T11:53:30Z

Could you please merge with/rebase on the latest PEFT main, then the CI should pass and we can merge.

BenjaminBossan

Thanks for the great work, the PR LGTM. Hopefully it finds a lot of adoption.

Add new PEFT method C³A (Circular Convolution Adaptation). From "Parameter-Efficient Fine-Tuning via Circular Convolution": https://arxiv.org/abs/2407.19342

init code

167c483

Phoveran added 2 commits June 16, 2025 12:54

pass test

643c36a

make style

4916d86

BenjaminBossan requested changes Jun 17, 2025

View reviewed changes

BenjaminBossan mentioned this pull request Jun 17, 2025

WaveFT method added into tuners #2560

Merged

add more testing, doc, and example

314a6a7

BenjaminBossan requested changes Jun 18, 2025

View reviewed changes

Phoveran added 2 commits June 19, 2025 11:35

minor fix

c96c6d2

add minor comments

b2a3d51

make style

cb3432c

fix annotation bug

5b01cb1

BenjaminBossan reviewed Jun 23, 2025

View reviewed changes

src/peft/tuners/c3a/config.py Show resolved Hide resolved

format using docformatter

1d64fff

manual wrap, doc update for lr instruction

5eda9ad

add C3A benchmark; add instructions on C3A

1e9dff3

remove results

6b177de

Merge branch 'main' of https://github.com/huggingface/peft

89bd90e

BenjaminBossan approved these changes Jun 30, 2025

View reviewed changes

BenjaminBossan merged commit e657707 into huggingface:main Jun 30, 2025
21 of 27 checks passed

BenjaminBossan mentioned this pull request Aug 29, 2025

Integrate DeLoRA #2760

Closed

[FEAT] Add C3A Support #2577

[FEAT] Add C3A Support #2577

Uh oh!

Conversation

Phoveran commented Jun 9, 2025

Uh oh!

BenjaminBossan commented Jun 10, 2025

Uh oh!

Phoveran commented Jun 16, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

review-notebook-app bot commented Jun 18, 2025

Uh oh!

Phoveran commented Jun 18, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Jun 18, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Phoveran commented Jun 20, 2025

Uh oh!

BenjaminBossan commented Jun 20, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Phoveran commented Jun 24, 2025

Uh oh!

BenjaminBossan commented Jun 24, 2025

Uh oh!

Phoveran commented Jun 25, 2025

Uh oh!

Phoveran commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BenjaminBossan commented Jun 27, 2025

Uh oh!

BenjaminBossan commented Jun 27, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Phoveran commented Jun 27, 2025 •

edited

Loading