Hey @BenjaminBossan! I have raised a PR to fix this issue. Please take a look when you have a moment and let me know if anything needs to be updated further.

@

Thanks for working on this PR. Before we proceed, let's add some unit tests to ensure that everything works as expected. As a starting point, check these tests:

(

peft/tests/test_initialization.py

Lines 3857 to 3892 in 5d97453

	def test_scaling_simple(self, model):
	n_layers = 5
	rank, lora_alpha = 8, 16
	config = LoraConfig(
	r=rank,
	lora_alpha=lora_alpha,
	target_modules=["k_proj"],
	)
	model = get_peft_model(model, config)
	scalings = self.get_scalings(model)
	expected = [lora_alpha / rank] * n_layers
	assert scalings == expected
	# double
	self.scale_layer(model, 2)
	scalings = self.get_scalings(model)
	expected = [4.0] * n_layers
	assert scalings == expected
	# back to original
	self.unscale_layer(model, None)
	scalings = self.get_scalings(model)
	expected = [2.0] * n_layers
	assert scalings == expected
	# triple
	self.set_scale(model, "default", 3)
	scalings = self.get_scalings(model)
	expected = [6.0] * n_layers
	assert scalings == expected
	# back to original
	self.unscale_layer(model, 3)
	scalings = self.get_scalings(model)
	expected = [2.0] * n_layers
	assert scalings == expected

I would suggest that you create a copy of these tests with a new name, e.g. test_scaling_rslora, and then change the testing code to use rslora. Change the expected values accordingly. LMK if you have questions.

Thank you @BenjaminBossan, for reviewing it. I will add the test file as soon as possible.

@tanuj-rai When the PR is ready for another review, please ping me.

@BenjaminBossan I'm trying to run the test test_scaling_simple in tests/test_scaling_rslora.py, but I get the error: fixture 'model' not found. I'm unsure how to define or locate the model fixture required for this test.

 pytest tests/test_scaling_rslora.py
============================= test session starts =============================
platform win32 -- Python 3.10.11, pytest-8.4.1, pluggy-1.6.0
rootdir: C:\Users\1234\peft
configfile: pyproject.toml
plugins: anyio-4.4.0, cov-7.0.0
collected 1 item

tests\test_scaling_rslora.py E                                           [100%]

=================================== ERRORS ====================================
______________ ERROR at setup of TestScaling.test_scaling_simple ______________
file C:\Users\1234\peft\tests\test_scaling_rslora.py, line 5
      def test_scaling_simple(self, model):
E       fixture 'model' not found
>       available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, capteesys, cov, doctest_namespace, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

C:\Users\1234\peft\tests\test_scaling_rslora.py:5
=============================== tests coverage ================================
______________ coverage: platform win32, python 3.10.11-final-0 ___

`

@tanuj-rai The reason why you cannot run your test is because you put it in a separate file. If you place it below the other tests I referenced above, it will work. Why? Because in that test class, a so called pytest.fixture with the name model is defined (here), which will be automatically added to the test when it's called.

When you move your tests over there, you should give the function a unique name, e.g. test_scaling_with_rslora. Then, you want to run only this specific test, call pytest tests/test_initialization.py -k test_scaling_with_rslora.

@tanuj-rai The reason why you cannot run your test is because you put it in a separate file. If you place it below the other tests I referenced above, it will work. Why? Because in that test class, a so called pytest.fixture with the name model is defined (here), which will be automatically added to the test when it's called.

When you move your tests over there, you should give the function a unique name, e.g. test_scaling_with_rslora. Then, you want to run only this specific test, call pytest tests/test_initialization.py -k test_scaling_with_rslora.

Ahh, I see. Thank you @BenjaminBossan. That was my mistake. I’ve removed it now and added the new test function 'test_scaling_with_rslora' into test_initialization.py.

When I am running the pytest. It got deselected.

$ pytest tests/test_initialization.py -k test_scaling_with_rslora
============================= test session starts =============================
platform win32 -- Python 3.10.11, pytest-8.4.1, pluggy-1.6.0
rootdir: C:\Users\1234\peft
configfile: pyproject.toml
plugins: anyio-4.4.0, cov-7.0.0
collected 241 items / 241 deselected / 0 selected

=============================== tests coverage ================================
______________ coverage: platform win32, python 3.10.11-final-0 __
`

When I am running the pytest. It got deselected.

Strange, when I run the exact same command on your branch, the test runs as expected. It does, however, raise an error because math needs to be imported:

>       expected = [lora_alpha / math.sqrt(rank)] * n_layers
E       NameError: name 'math' is not defined. Did you forget to import 'math'

pytest tests/test_initialization.py -k test_scaling_with_rslora

@BenjaminBossan! As you said, it is running fine for you. But I don't know why it skips it when I run pytest. What should I do now to fix it? I have added import math.

Hi @BenjaminBossan, I’ve updated unscale_layer to correctly handle rslora by resetting to lora_alpha / sqrt(r) when scale=None. Please take a look and let me know if anything needs to be updated.

def unscale_layer(self, scale: Optional[float | int] = None) -> None:
    for active_adapter in self.active_adapters:
        if active_adapter not in self.lora_A:
            continue

        if scale is None:
            if self.use_rslora.get(active_adapter, False):
                self.scaling[active_adapter] = self.lora_alpha[active_adapter] / math.sqrt(self.r[active_adapter])
            else:
                self.scaling[active_adapter] = self.lora_alpha[active_adapter] / self.r[active_adapter]
        else:
            self.scaling[active_adapter] /= scale
`

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the update @tanuj-rai. The tests are passing now. If the PR is ready from your side, please convert it from draft to "ready for review".

Thanks for the update @tanuj-rai. The tests are passing now. If the PR is ready from your side, please convert it from draft to "ready for review".

Thanks, @BenjaminBossan for reviewing it. I have converted it from draft to "ready for review".