Keycloak Operator cannot get secret keycloak-db 401 #36801

vudev98 · 2025-01-24T10:07:39Z

vudev98
Jan 24, 2025

Before reporting an issue

I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

operator

Describe the bug

Hi Support Team,

We found out an log error where keycloak-operator cannot access secret keycloak-db 401.

2025-01-24 04:55:24,100 INFO  [io.fab.kub.cli.dsl.int.AbstractWatchManager] (vert.x-eventloop-thread-3) Watch connection error recieved 35383 times without progress, will reconnect if possible: io.fabric8.kubernetes.client.KubernetesClientException: Received 401 on websocket. Failure executing: GET at: https://172.20.0.1:443/api/v1/namespaces/keycloak/secrets?allowWatchBookmarks=true&labelSelector=app%3Dkeycloak%2Capp.kubernetes.io%2Fmanaged-by%3Dkeycloak-operator&resourceVersion=731560967&timeoutSeconds=600&watch=true. Message: Unauthorized.
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$start$3(WatchConnectionManager.java:86)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$buildWebSocket$17(StandardHttpClient.java:244)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.utils.AsyncUtils.lambda$retryWithExponentialBackoff$3(AsyncUtils.java:90)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.vertx.VertxHttpClient.lambda$buildWebSocketDirect$3(VertxHttpClient.java:89)
	at io.vertx.core.impl.future.FutureImpl$2.onFailure(FutureImpl.java:117)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:303)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:265)
	at io.vertx.core.impl.future.PromiseImpl.onFailure(PromiseImpl.java:54)
	at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:43)
	at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23)
	at io.vertx.core.http.impl.HttpClientBase.lambda$webSocket$1(HttpClientBase.java:170)
	at io.vertx.core.impl.future.FutureImpl$4.onFailure(FutureImpl.java:188)
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:75)
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:305)
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:295)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:72)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:265)
	at io.vertx.core.Promise.fail(Promise.java:89)
	at io.vertx.core.http.impl.Http1xClientConnection.lambda$toWebSocket$10(Http1xClientConnection.java:1014)
	at io.vertx.core.http.impl.WebSocketHandshakeInboundHandler.handshakeComplete(WebSocketHandshakeInboundHandler.java:103)
	at io.vertx.core.http.impl.WebSocketHandshakeInboundHandler.channelRead(WebSocketHandshakeInboundHandler.java:84)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

Any suggestion to resolve this issue would be very much appreciated!

Thanks!

Environment:

Keycloak-Operator version: 24.0.3
Kubernetes version (use kubectl version): 1.29

Version

24.0.3

Regression

The issue is a regression

Expected behavior

No error

Actual behavior

As described error

How to Reproduce?

Deploy Keycloak-operator to EKS

Anything else?

No response

2025-01-24T10:19:08Z

keycloak-github-bot[bot]
bot Jan 24, 2025

Thanks for reporting this issue, but as this is reported against an older and unsupported release we are not able to evaluate the issue. Please verify with the nightly build or the latest release.

If the issue can be reproduced in the nightly build or latest release add a comment with additional information, otherwise this issue will be automatically closed within 14 days.

0 replies

shawkins · 2025-01-24T18:25:44Z

shawkins
Jan 24, 2025
Collaborator

As the bot comment requests, do you see the same behavior on later Keycloak versions?

A 401 would indicate a authorization issue. The RBAC included with the operator gives it permission to watch all the necessary resources.

How did you install the operator?

3 replies

vudev98 Feb 7, 2025
Author

Hi @shawkins,

I installed the keycloak-operator according to this guide | Installing by using kubectl without Operator Lifecycle Manager: https://www.keycloak.org/operator/installation

Here is the role policies of operator

I still encounter this issue, hope you could help.

shawkins Feb 7, 2025
Collaborator

I still encounter this issue, hope you could help.

Thanks for the confirmation. Can you describe how reproducible this is for you - does it always occur after a certain amount of uptime?

My initial attempt to reproduce in #36801 (reply in thread) showed the new token being used, so I may need more information on exactly when this manifests for you.

vudev98 Feb 10, 2025
Author

This is a first time I encounter this issue. Before that, everything worked just fine.

nicolamacoir · 2025-01-31T12:06:00Z

nicolamacoir
Jan 31, 2025

We’re encountering the same error here. The operator has been working for a long time, and only recently have we started getting this error. Restarting the operator seems to resolve the issue, but it reappears after a while. It appears to be related to the expiration of the JWT token associated with the service account. Although the token is replaced by Kubernetes, it seems that the client is not using the new token.
I’ve noticed similar complaints for the fabric8io Kubernetes client, which appears to be the one being used by the operator. Here’s the link to the issue: fabric8io/kubernetes-client#2112

Version: 24.0.4 ( and very recently upgraded Keycloak and it's operator to 25.0.6 but same issue)
Kubernetes: 1.30
Kubernetes Provider: Azure AKS

EDIT: as a workaround to have the operator always available, I used following hacky trick to make it restart every hour:

          livenessProbe:
            httpGet: null
            exec:
              command:
                - /bin/bash
                - -c
                - exit 1
            failureThreshold: 1
            initialDelaySeconds: 3600

1 reply

shawkins Feb 3, 2025
Collaborator

I’ve noticed similar complaints for the fabric8io Kubernetes client, which appears to be the one being used by the operator. Here’s the link to the issue: fabric8io/kubernetes-client#2112

Both that and fabric8io/kubernetes-client#4802 are in the client used by Keycloak 24, and I'm not aware of any new changes in the one used by Keycloak 26, but it would be good to confirm the issue still persists for you there.

When using a service account the expectation is that once per minute, or after a 401, we'll reload the Config to pickup the token from the file system.

Although the token is replaced by Kubernetes, it seems that the client is not using the new token.

@nicolamacoir just to make sure, have you confirmed that the token is being rotated on the Pod filesystem? And is it possible that the Pod is picking up a kubeconfig file instead?

I tried to reproduce this on an version 25 operator installed through OLM, but did not see any issue after the token was rotated.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Keycloak Operator cannot get secret keycloak-db 401 #36801

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Keycloak Operator cannot get secret keycloak-db 401 #36801

Uh oh!

vudev98 Jan 24, 2025

Before reporting an issue

Area

Describe the bug

Version

Regression

Expected behavior

Actual behavior

How to Reproduce?

Anything else?

Replies: 3 comments · 4 replies

Uh oh!

keycloak-github-bot[bot] bot Jan 24, 2025

Uh oh!

shawkins Jan 24, 2025 Collaborator

Uh oh!

Uh oh!

vudev98 Feb 7, 2025 Author

Uh oh!

shawkins Feb 7, 2025 Collaborator

Uh oh!

vudev98 Feb 10, 2025 Author

Uh oh!

Uh oh!

nicolamacoir Jan 31, 2025

Uh oh!

shawkins Feb 3, 2025 Collaborator

vudev98
Jan 24, 2025

Replies: 3 comments 4 replies

keycloak-github-bot[bot]
bot Jan 24, 2025

shawkins
Jan 24, 2025
Collaborator

vudev98 Feb 7, 2025
Author

shawkins Feb 7, 2025
Collaborator

vudev98 Feb 10, 2025
Author

nicolamacoir
Jan 31, 2025

shawkins Feb 3, 2025
Collaborator