Switch testsuite to test with keycloak v2 login theme #30319

hmlnarik · 2024-06-11T06:36:17Z

⚠️ EDIT: Since the functional changes have been merged in #31402, this PR has changed to testsuite-only changes. ⚠️

The fixes (mostly selectors) needed for tests.

In the future, to switch the default login theme to keycloak.v2, do the following:

Update ThemeSelectorProvider: Uncomment the relevant lines
Update testsuite/integration-arquillian/tests/pom.xml: Revert the change in <login.theme.default> property
Update ThemeSelectorTest per comment
Update UncaughtErrorPageTest per comment

With the prior removal of alpine.js, htmlunit is used again, and the workflow run times are similar to those with original keycloak login theme

Fixes: #29009

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T07:49:44Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T13:16:38Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce

Keycloak CI - Store IT

java.lang.RuntimeException: Failed to deserialize token
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.toAccessToken(OidcClaimToUserSessionNoteMapperTest.java:154)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.login(OidcClaimToUserSessionNoteMapperTest.java:145)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce(OidcClaimToUserSessionNoteMapperTest.java:109)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T16:27:22Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnFirstLoginOnlyWhenNameMatchesAndSyncModeIsImport

Keycloak CI - Store IT

java.lang.RuntimeException: Failed to deserialize token
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.toAccessToken(OidcClaimToUserSessionNoteMapperTest.java:154)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.login(OidcClaimToUserSessionNoteMapperTest.java:145)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.claimIsPropagatedOnFirstLoginOnlyWhenNameMatchesAndSyncModeIsImport(OidcClaimToUserSessionNoteMapperTest.java:98)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
...

java.lang.RuntimeException: Failed to deserialize token
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.toAccessToken(OidcClaimToUserSessionNoteMapperTest.java:154)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.login(OidcClaimToUserSessionNoteMapperTest.java:145)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.claimIsPropagatedOnFirstLoginOnlyWhenNameMatchesAndSyncModeIsImport(OidcClaimToUserSessionNoteMapperTest.java:98)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T17:53:30Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T19:00:42Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-11T20:20:50Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce

Keycloak CI - Store IT

java.lang.RuntimeException: Failed to deserialize token
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.toAccessToken(OidcClaimToUserSessionNoteMapperTest.java:154)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.login(OidcClaimToUserSessionNoteMapperTest.java:145)
	at org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest.claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce(OidcClaimToUserSessionNoteMapperTest.java:119)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-06-12T09:09:50Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Keycloak CI - Base IT

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('principal=user-tenant1') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.5.0-1021-azure', java.version: '21.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver_ByGraphene
...

Report flaky test

hmlnarik · 2024-07-17T12:42:28Z

@stianst @vmuzikar The issue with the workflow run time has been addressed. Can you please revisit your review?

hmlnarik · 2024-07-18T10:19:11Z

This PR is being split into #31402 and the testsuite and GHA changes which will be present here.

The fixes (mostly selectors) are needed for tests. In the future, to switch the keycloak.v2 to the default theme, do the following: - Update `ThemeSelectorProvider`: Uncomment relevant lines - Update `testsuite/integration-arquillian/tests/pom.xml`: Revert the change in `<login.theme.default>` property - Update `ThemeSelectorTest` per comment Signed-off-by: Hynek Mlnarik <hmlnarik@redhat.com>

ssilvert

We need to merge this to make progress on Login V2. It changes most of the tests to use Login V2. It only affects how tests are run.

We plan to follow up with a PR that will fully test both Login V1 and Login V2 in the nightly build.

Meanwhile, Login V1 is still tested with FIPS tests and Docker tests, which provides us a nice sanity check on Login V1.

vmuzikar

The changes as such LGTM, nice work @hmlnarik with resolving the issue with htmlunit!

My only concern is that we'll be now testing with a different theme that we ship as the default. I understand it's only temporary which is ok. But I wonder if we'll be able to make v2 the default before the next major release, i.e. Keycloak 26, so that we'd test the default.

ssilvert · 2024-07-29T19:14:49Z

The changes as such LGTM, nice work @hmlnarik with resolving the issue with htmlunit!

My only concern is that we'll be now testing with a different theme that we ship as the default. I understand it's only temporary which is ok. But I wonder if we'll be able to make v2 the default before the next major release, i.e. Keycloak 26, so that we'd test the default.

@vmuzikar Regardless of which login theme is the default, we need to keep both around for awhile. Login V2 needs to be fully vetted in the community before we make it the default for new realms. However, V1 is pretty stable and should require less regression testing. So we want to start testing against V2 as soon as possible because that's where the new development is happening.

For performance reasons, we don't want to run the tests against both versions each time a PR is submitted. But we do plan to run both in the nightly build.

vmuzikar · 2024-07-30T06:58:14Z

@ssilvert Ok, makes sense. Thanks for clarifying.

hmlnarik self-assigned this Jun 11, 2024

keycloak-github-bot bot added the flaky-test label Jun 11, 2024

keycloak-github-bot bot reviewed Jun 11, 2024

View reviewed changes

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch 5 times, most recently from f27f246 to 082f0d2 Compare June 11, 2024 12:03

keycloak-github-bot bot reviewed Jun 11, 2024

View reviewed changes

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 173151e to 6bee030 Compare June 11, 2024 16:15

keycloak-github-bot bot reviewed Jun 11, 2024

View reviewed changes

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 9e7a645 to 2d55a1f Compare June 11, 2024 18:59

keycloak-github-bot bot mentioned this pull request Jun 11, 2024

Flaky test: org.keycloak.testsuite.model.user.UserModelTest#testAddRemoveUsersInTheSameGroupConcurrent #30264

Closed

keycloak-github-bot bot reviewed Jun 11, 2024

View reviewed changes

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 2d55a1f to 718c817 Compare June 12, 2024 07:48

keycloak-github-bot bot reviewed Jun 12, 2024

View reviewed changes

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 718c817 to dea8f2c Compare June 12, 2024 09:14

This was referenced Jul 10, 2024

removed use of alpine.js due to tests #30659

Closed

Login v2 without alpine #31243

Closed

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from c911eaf to 62103be Compare July 17, 2024 08:24

hmlnarik changed the title ~~Switch login theme to keycloak v2~~ Switch login theme to keycloak v2 (removed alpine.js) Jul 17, 2024

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch 2 times, most recently from 49ace1f to 36d9e32 Compare July 17, 2024 08:59

keycloak-github-bot bot mentioned this pull request Jul 17, 2024

Flaky test: org.keycloak.testsuite.broker.KcOidcBrokerTest#testPostBrokerLoginFlowWithOTP_bruteForceEnabled #30188

Closed

hmlnarik marked this pull request as ready for review July 17, 2024 13:02

edewit previously approved these changes Jul 18, 2024

View reviewed changes

hmlnarik mentioned this pull request Jul 18, 2024

Deprecate keycloak v1 and fix v2 login themes #31402

Merged

hmlnarik marked this pull request as draft July 18, 2024 10:19

hmlnarik dismissed edewit’s stale review via 259831b July 19, 2024 14:03

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 36d9e32 to 259831b Compare July 19, 2024 14:03

hmlnarik changed the title ~~Switch login theme to keycloak v2 (removed alpine.js)~~ Switch testsuite to test with keycloak v2 login theme Jul 19, 2024

hmlnarik force-pushed the 29009-Switch-login-theme-to-keycloak-v2-Stabilize-testsuite-after-the-switch-main branch from 259831b to 04041fa Compare July 19, 2024 19:34

hmlnarik marked this pull request as ready for review July 29, 2024 14:19

hmlnarik requested review from stianst and vmuzikar July 29, 2024 14:20

ssilvert approved these changes Jul 29, 2024

View reviewed changes

vmuzikar reviewed Jul 29, 2024

View reviewed changes

vmuzikar approved these changes Jul 30, 2024

View reviewed changes

miquelsi approved these changes Jul 30, 2024

View reviewed changes

mposolda approved these changes Jul 30, 2024

View reviewed changes

hmlnarik merged commit 183cd6c into keycloak:main Jul 30, 2024

hmlnarik mentioned this pull request Jul 30, 2024

Run tests with original keycloak login theme in nightly #31764

Closed

Switch testsuite to test with keycloak v2 login theme #30319

Switch testsuite to test with keycloak v2 login theme #30319

Uh oh!

Conversation

hmlnarik commented Jun 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnFirstLoginOnlyWhenNameMatchesAndSyncModeIsImport

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 11, 2024

Unreported flaky test detected

org.keycloak.testsuite.broker.OidcClaimToUserSessionNoteMapperTest#claimIsPropagatedOnAllLoginsWhenNameMatchesAndSyncModeIsForce

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jun 12, 2024

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#multiTenant1SamlTest

Uh oh!

hmlnarik commented Jul 17, 2024

Uh oh!

hmlnarik commented Jul 18, 2024

Uh oh!

ssilvert left a comment

Choose a reason for hiding this comment

Uh oh!

vmuzikar left a comment

Choose a reason for hiding this comment

Uh oh!

ssilvert commented Jul 29, 2024

Uh oh!

vmuzikar commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

hmlnarik commented Jun 11, 2024 •

edited

Loading