Error while implementing the distribution cache (ispn) in Keycloak Deployment #36380

nitindhiman314e · 2025-01-13T04:34:56Z

nitindhiman314e
Jan 13, 2025

Description

We are getting below error in our Keycloak deployment, causing failures in transaction operations. Below is the error log for reference:

2025-01-06 10:32:55,391 INFO  [org.hibernate.orm.jdbc.batch] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) HHH100503: On release of batch it still contained JDBC statements
2025-01-06 10:32:55,391 ERROR [org.hibernate.orm.jdbc.batch] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) HHH100501: Exception executing batch [org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update public.OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?], SQL: update public.OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
2025-01-06 10:32:55,398 WARN  [com.arjuna.ats.arjuna] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0af500a0:92fd:677baf85:1cc, org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization@5e5bfaa4 >: jakarta.persistence.OptimisticLockException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update public.OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
        at org.hibernate.internal.ExceptionConverterImpl.wrapStaleStateException(ExceptionConverterImpl.java:221)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:95)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:167)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:173)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1433)
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:491)
        at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:2354)
        at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:1978)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:439)
        at org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:336)
        at org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
        at org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:52)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:348)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:66)
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:135)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1290)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
        at io.quarkus.narayana.jta.runtime.NotifyingTransactionManager.commit(NotifyingTransactionManager.java:70)
        at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:100)
        at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
        at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:396)
        at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:361)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:439)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:312)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:302)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.lambda$process$3(PersistentSessionsWorker.java:89)
        at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:102)
        at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:93)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.process(PersistentSessionsWorker.java:86)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.run(PersistentSessionsWorker.java:69)
Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update public.OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
        at org.hibernate.jdbc.Expectations.checkBatched(Expectations.java:80)
        at org.hibernate.jdbc.Expectation$RowCount.verifyOutcome(Expectation.java:174)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.checkRowCounts(BatchImpl.java:321)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.lambda$performExecution$2(BatchImpl.java:285)
        at org.hibernate.engine.jdbc.mutation.internal.PreparedStatementGroupSingleTable.forEachStatement(PreparedStatementGroupSingleTable.java:67)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.performExecution(BatchImpl.java:264)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.execute(BatchImpl.java:242)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:188)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:674)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:511)

The issue seems to be related to the OFFLINE_USER_SESSION table and a Hibernate StaleStateException, indicating that batch updates are returning unexpected row counts. We have already reviewed our setup and implemented a distributed cache configuration to support multiple replicas. Below are the details of our environment and configuration:

Environment Details:

Keycloak Version: 26.0.7
Image Used: quay.io/keycloak/keycloak:26.0.7
Deployment Strategy: Custom Kubernetes deployment (not using the Keycloak operator)
Database: PostgreSQL
Replication Setup: Minimum of 2 replicas

Caching Configuration:
We followed the Keycloak documentation for distributed cache (Keycloak Server Caching) and made the following changes:

Modified cache-ispn.xml: Set owners=2 for all distributed-cache settings.
Arguments Passed to Start Keycloak:

args:
  - start
  - --cache=ispn
  - --cache-stack=kubernetes
  - -Djgroups.dns.query=keycloak-discovery.keycloak.svc.cluster.local
  - --spi-connections-infinispan-transport-stack=kubernetes
  - --features=persistent-user-sessions
  - --health-enabled=true
  - --metrics-enabled=true
  - --cache-metrics-histograms-enabled=true

Keycloak application starting logs:

Server configuration updated and persisted. Run the following command to review the configuration:

        kc.sh show-config

Next time you run the server, just run:

        kc.sh start --cache=ispn --cache-stack=kubernetes --spi-connections-infinispan-transport-stack=kubernetes --cache-metrics-histograms-enabled=true --optimized

2025-01-07 08:20:37,865 INFO  [org.keycloak.quarkus.runtime.storage.infinispan.CacheManagerFactory] (main) Starting Infinispan embedded cache manager
2025-01-07 08:20:39,045 INFO  [org.infinispan.CONTAINER] (ForkJoinPool.commonPool-worker-1) ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
2025-01-07 08:20:39,360 WARN  [org.jgroups.stack.Configurator] (ForkJoinPool.commonPool-worker-1) JGRP000014: ThreadPool.thread_dumps_threshold has been deprecated: ignored
2025-01-07 08:20:39,397 INFO  [org.infinispan.CLUSTER] (ForkJoinPool.commonPool-worker-1) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
2025-01-07 08:20:39,399 INFO  [org.jgroups.JChannel] (ForkJoinPool.commonPool-worker-1) local_addr: 88973f98-2cf5-48d0-9786-f9353c55be77, name: keycloak-68488d7ff7-9hhsp-34218
2025-01-07 08:20:39,415 INFO  [org.jgroups.protocols.FD_SOCK2] (ForkJoinPool.commonPool-worker-1) server listening on *.57800
2025-01-07 08:20:41,453 INFO  [org.jgroups.protocols.pbcast.GMS] (ForkJoinPool.commonPool-worker-1) keycloak-68488d7ff7-9hhsp-34218: no members discovered after 2028 ms: creating cluster as coordinator
2025-01-07 08:20:41,461 INFO  [org.infinispan.CLUSTER] (ForkJoinPool.commonPool-worker-1) ISPN000094: Received new cluster view for channel ISPN: [keycloak-68488d7ff7-9hhsp-34218|0] (1) [keycloak-68488d7ff7-9hhsp-34218]
2025-01-07 08:20:41,530 INFO  [org.infinispan.CLUSTER] (ForkJoinPool.commonPool-worker-1) ISPN000079: Channel `ISPN` local address is `keycloak-68488d7ff7-9hhsp-34218`, physical addresses are `[10.245.0.17:7800]`
2025-01-07 08:20:42,073 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-68488d7ff7-9hhsp-34218, Site name: null
2025-01-07 08:20:42,077 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2025-01-07 08:20:43,496 WARN  [io.agroal.pool] (main) Datasource '<default>': JDBC resources leaked: 3 ResultSet(s) and 0 Statement(s)
2025-01-07 08:20:43,681 INFO  [io.quarkus] (main) Keycloak 26.0.7 on JVM (powered by Quarkus 3.15.1) started in 14.350s. Listening on: http://0.0.0.0:8080/. Management interface listening on http://0.0.0.0:9000/.
2025-01-07 08:20:43,681 INFO  [io.quarkus] (main) Profile prod activated. 
2025-01-07 08:20:43,681 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, micrometer, narayana-jta, opentelemetry, reactive-routes, rest, rest-jackson, smallrye-context-propagation, smallrye-health, vertx]

Issue Summary:

The error appears to be related to the handling of the OFFLINE_USER_SESSION table. Despite implementing distributed caching for multi-replica support

Could you please assist in identifying the root cause of this issue and provide guidance on resolving it? Specifically:

Are there any additional configurations or optimizations required in our distributed caching setup?
Is there any known issue with the OFFLINE_USER_SESSION table when running multiple replicas of Keycloak?
Any recommendations for database-level settings to avoid table locks?

We look forward to your assistance and are happy to provide further details if required.
Thanks.

darius-m · 2025-01-13T11:25:57Z

darius-m
Jan 13, 2025

Are you going for a multi-site deployment, or single-site? The blog has some clarifications on which cache settings should be used in either case; perhaps that could help - https://www.keycloak.org/2024/12/storing-sessions-in-kc26.html. If you have an external Infinispan server for a multi-site deployment, you need to use the "cache-remote-store" settings and enable the "multi-site" feature.

From the logs, it seems that the Keycloak service does not see the other instances in the cluster (it creates a cluster by itself). This would imply that the Keycloak nodes do not synchronize using the caches. I think it could be normal when using an external Infinispan server, as the Keycloak instances can use that to sync, instead of clustering the embedded Infinispan instances.

16 replies

Nowheresly Jan 15, 2025

Here are some output of psql commands:

postgres=# \c keycloak
You are now connected to database "keycloak" as user "postgres".
keycloak=# \d

                     List of relations
 Schema |             Name              | Type  |   Owner   
--------+-------------------------------+-------+-----------
 public | admin_event_entity            | table | ukeycloak
 public | associated_policy             | table | ukeycloak
 public | authentication_execution      | table | ukeycloak
 public | authentication_flow           | table | ukeycloak
 public | authenticator_config          | table | ukeycloak
 public | authenticator_config_entry    | table | ukeycloak
 public | broker_link                   | table | ukeycloak
 public | client                        | table | ukeycloak
 public | client_attributes             | table | ukeycloak
 public | client_auth_flow_bindings     | table | ukeycloak
 public | client_initial_access         | table | ukeycloak
 public | client_node_registrations     | table | ukeycloak
 public | client_scope                  | table | ukeycloak
 public | client_scope_attributes       | table | ukeycloak
 public | client_scope_client           | table | ukeycloak
 public | client_scope_role_mapping     | table | ukeycloak
 public | component                     | table | ukeycloak
 public | component_config              | table | ukeycloak
 public | composite_role                | table | ukeycloak
 public | credential                    | table | ukeycloak
 public | databasechangelog             | table | ukeycloak
 public | databasechangeloglock         | table | ukeycloak
<other results ommitted>

keycloak=# SELECT 
    pid, 
    usename, 
    query, 
    state 
FROM 
    pg_stat_activity 
WHERE 
    usename='ukeycloak';
   pid   |  usename  |                                  query                                  | state 
---------+-----------+-------------------------------------------------------------------------+-------
 3073403 | ukeycloak | SELECT address, name, ip, coord FROM JGROUPS_PING WHERE cluster_name=$1 | idle
 3044048 | ukeycloak | COMMIT                                                                  | idle
 3060299 | ukeycloak | COMMIT                                                                  | idle
 3071405 | ukeycloak | COMMIT                                                                  | idle
 3071406 | ukeycloak |                                                                         | idle
 3073427 | ukeycloak | COMMIT                                                                  | idle

I hope this helps to establishing that our keycloak instances are connected to an external postgres db.

ahus1 Jan 15, 2025
Collaborator

The errors you see here occur when a user session is updated concurrently on two nodes. The loosing node will write those warnings, and then retry.

Unfortunately the warnings are always written by Hibernate. The Keycloak team asked the Hibernate team to remove that warning, but that will take another release of Hibernate, Quarkus and Keycloak.

Nowheresly Jan 16, 2025

Thank you very much @ahus1 for taking time to answer. So it means this error can be safely ignored? and you're working to reduce the verbosity of the logs with the Hibernate team? is that correct?

ahus1 Jan 16, 2025
Collaborator

A "StaleStateException" of the persistent session worker will lead to a retry. Once the retry succeeds, there is no user-facing error. So it is ok to continue. Please continue to watch out for other errors.

It would be good to understand how/why in your setup those exceptions occur. The only explanation to me is that for the same user session there are multiple concurrent requests going on on different nodes of Keycloak. It would be good to understand in what scenarios they are happening, so we could reduce the number of those exceptions. In the end, each retry increases the load of the system, and slows down the response.

nitindhiman314e Jan 23, 2025
Author

It seems version 26.1.0 changes the default discovery protocol to jdbc-ping, which should make things easier in cloud environments, according to the team - https://github.com/keycloak/keycloak/releases/tag/26.1.0. See if it works better for you.

Hi @darius-m - Thanks for your reply.

I tried running version 26.1.0 with multiple replicas (2) using the default arguments, but I'm encountering the same issue. For the database, I'm using a PostgreSQL database, and both replica instances are sharing the same database connection string. I'm not using keycloak.conf to manage the connection. Could you please help me resolve this issue?

- args:
            - start
            - --metrics-enabled=true

Nowheresly · 2025-01-15T13:21:43Z

Nowheresly
Jan 15, 2025

Hi,

we switched to keycloak 26.1.0 with jdbc-ping as cache stack. We have 2 instances of keycloak.

We are experiencing the same error from times to times:

2025-01-15 13:14:16,177 logLevel=INFO  [org.hibernate.orm.jdbc.batch] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) HHH100503: On release of batch it still contained JDBC statements
k2025-01-15 13:14:16,177 logLevel=ERROR [org.hibernate.orm.jdbc.batch] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) HHH100501: Exception executing batch [org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?], SQL: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
k2025-01-15 13:14:16,182 logLevel=WARN  [com.arjuna.ats.arjuna] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a013505:b541:6787aed0:31c6, org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization@685d69f5 >: jakarta.persistence.OptimisticLockException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
 	at org.hibernate.internal.ExceptionConverterImpl.wrapStaleStateException(ExceptionConverterImpl.java:221)
 	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:95)
 	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:167)
 	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:173)
 	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1433)
 	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:491)
 	at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:2354)
 	at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:1978)
 	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:439)
 	at org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:336)
 	at org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
 	at org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
 	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:52)
 	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:348)
 	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:66)
 	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:135)
 	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1290)
 	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
 	at io.quarkus.narayana.jta.runtime.NotifyingTransactionManager.commit(NotifyingTransactionManager.java:70)
 	at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:101)
 	at org.keycloak.services.DefaultKeycloakTransactionManager.lambda$commitWithTracing$0(DefaultKeycloakTransactionManager.java:169)
 	at org.keycloak.tracing.NoopTracingProvider.trace(NoopTracingProvider.java:59)
 	at org.keycloak.tracing.NoopTracingProvider.trace(NoopTracingProvider.java:69)
 	at org.keycloak.services.DefaultKeycloakTransactionManager.commitWithTracing(DefaultKeycloakTransactionManager.java:168)
 	at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
 	at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:396)
 	at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:361)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:463)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:336)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:326)
 	at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.lambda$process$5(PersistentSessionsWorker.java:110)
 	at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:102)
 	at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:93)
 	at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.lambda$process$7(PersistentSessionsWorker.java:107)
 	at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:337)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:456)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:336)
 	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:326)
 	at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.process(PersistentSessionsWorker.java:91)
 	at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.run(PersistentSessionsWorker.java:76)
 Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,VERSION=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and VERSION=?
 	at org.hibernate.jdbc.Expectations.checkBatched(Expectations.java:80)
 	at org.hibernate.jdbc.Expectation$RowCount.verifyOutcome(Expectation.java:174)
 	at org.hibernate.engine.jdbc.batch.internal.BatchImpl.checkRowCounts(BatchImpl.java:321)
 	at org.hibernate.engine.jdbc.batch.internal.BatchImpl.lambda$performExecution$2(BatchImpl.java:285)
 	at org.hibernate.engine.jdbc.mutation.internal.PreparedStatementGroupSingleTable.forEachStatement(PreparedStatementGroupSingleTable.java:67)
 	at org.hibernate.engine.jdbc.batch.internal.BatchImpl.performExecution(BatchImpl.java:264)
 	at org.hibernate.engine.jdbc.batch.internal.BatchImpl.execute(BatchImpl.java:242)
 	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:188)
 	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:674)
 	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:511)
 	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:414)
 	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:41)
 	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:127)
 	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1429)
 	... 35 more
 
k2025-01-15 13:14:16,186 logLevel=ERROR [org.keycloak.transaction.JtaTransactionWrapper] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) Exception during transaction operation. Request context: Non-HTTP task, Transaction: TransactionImple < ac, BasicAction: 0:ffff0a013505:b541:6787aed0:31c5 status: ActionStatus.ABORTED >

3 replies

hlxtsc Jan 17, 2025

We have the same problem with the ISPN cache.

ahus1 Jan 17, 2025
Collaborator

See #36380 (reply in thread) for an answer. If you think this is a bug after reading that comment, create a GitHub issue of type bug with relevant logs attached.

SomeDeveloper13 Oct 7, 2025

We have the same issues with 2 or more pods.
Running with a single pod, this issue stops.

Error while implementing the distribution cache (ispn) in Keycloak Deployment #36380

Uh oh!

nitindhiman314e Jan 13, 2025

Description

Replies: 2 comments · 19 replies

Uh oh!

darius-m Jan 13, 2025

Uh oh!

Uh oh!

Nowheresly Jan 15, 2025

Uh oh!

ahus1 Jan 15, 2025 Collaborator

Uh oh!

Nowheresly Jan 16, 2025

Uh oh!

ahus1 Jan 16, 2025 Collaborator

Uh oh!

nitindhiman314e Jan 23, 2025 Author

Uh oh!

Nowheresly Jan 15, 2025

Uh oh!

hlxtsc Jan 17, 2025

Uh oh!

ahus1 Jan 17, 2025 Collaborator

Uh oh!

SomeDeveloper13 Oct 7, 2025

nitindhiman314e
Jan 13, 2025

Replies: 2 comments 19 replies

darius-m
Jan 13, 2025

ahus1 Jan 15, 2025
Collaborator

ahus1 Jan 16, 2025
Collaborator

nitindhiman314e Jan 23, 2025
Author

Nowheresly
Jan 15, 2025

ahus1 Jan 17, 2025
Collaborator