+
Skip to content

Provide CLI Parameters for jgroups.* options #40690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 7, 2025
Merged

Provide CLI Parameters for jgroups.* options #40690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/documentation/upgrading/topics/changes/changes-26_4_0.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ Starting with this release, and when using the original PostgreSQL driver, {proj

You can override this behavior by setting your own value for `targetServerType` in the DB URL or additional properties.

=== JGroups system properties replaced with CLI options

Until now it was necessary to configure JGroups network addresses and ports using the `+jgroups.bind.*+` and `+jgroups.external_*+`
system properties. In this release we have introduced the following CLI options to allow these addresses and ports to be
configured directly via {project_name}: `cache-embedded-network-bind-address`, `cache-embedded-network-bind-port`,
`cache-embedded-network-external-address`, `cache-embedded-network-external-port`. Configuring ports using the old
properties will still function as before, but we recommend to change to the CLI options as this may change in the future.

// ------------------------ Deprecated features ------------------------ //
== Deprecated features

Expand Down
25 changes: 12 additions & 13 deletions docs/guides/server/caching.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,22 +321,25 @@ To ensure a healthy {project_name} clustering, some network ports need to be ope
The table below shows the TCP ports that need to be open for the `jdbc-ping` stack, and a description of the traffic that goes through it.

|===
|Port |Property | Description
|Port |Option| Property | Description

m|7800
m|cache-embedded-network-bind-port
m|jgroups.bind.port
|Unicast data transmission.

m|57800
m|
m|jgroups.fd.port-offset
|Failure detection by protocol `FD_SOCK2`.
It listens to the abrupt closing of a socket to suspect a {project_name} server failure.
The `jgroups.fd.port-offset` property defines the offset from the `jgroups.bind.port`.
The `jgroups.fd.port-offset` property defines the offset from the `cache-embedded-network-bind-port` option or `jgroups.bind.port` property.
By default, the offset is set to 50000, making the failure detection port 57800.

|===

NOTE: Use `-D<property>=<value>` to modify the ports above in your `JAVA_OPTS_APPEND` environment variable or in your CLI command.
NOTE: If an option is not available for the port you require, configure it using a system property `-D<property>=<value>`
in your `JAVA_OPTS_APPEND` environment variable or in your CLI command.

[#network-bind-address]
== Network bind address
Expand All @@ -345,11 +348,9 @@ To ensure a healthy {project_name} clustering, the network port must be bound on

By default, it picks a site local (non-routable) IP address, for example, from the 192.168.0.0/16 or 10.0.0.0/8 address range.

To override the address, set the `jgroups.bind.address` property.
To override the address, set the option `cache-embedded-network-bind-address=<IP>`.

NOTE: Use `-Djgroups.bind.address=<IP>` to modify the bind address in your `JAVA_OPTS_APPEND` environment variable or in your CLI command.

The following special values are also recognized for `jgroups.bind.address`:
The following special values are also recognized:

|===
|Value |Description
Expand Down Expand Up @@ -399,21 +400,19 @@ For more details about JGroups transport, check the http://jgroups.org/manual5/i
If you run {project_name} instances on different networks, for example behind firewalls or in containers, the different instances will not be able to reach each other by their local IP address.
In such a case, set up a port forwarding rule (sometimes called "`virtual server`") to their local IP address.

When using port forwarding, use the following properties so each node correctly advertises its external address to the other nodes:
When using port forwarding, use the following options so each node correctly advertises its external address to the other nodes:

|===
|Property | Description
|Option | Description

m|jgroups.external_port
m|cache-embedded-network-external-port
|Port that other instances in the {project_name} cluster should use to contact this node.

m|jgroups.external_addr
m|cache-embedded-network-external-address
|IP address that other instances in the {project_name} should use to contact this node.

|===

NOTE: Use `-D<property>=<value>` set this in your `JAVA_OPTS_APPEND` environment variable or in your CLI command.

== Exposing metrics from caches

Metrics from caches are automatically exposed when the metrics are enabled.
Expand Down
2 changes: 2 additions & 0 deletions .../org/keycloak/spi/infinispan/impl/embedded/DefaultCacheEmbeddedConfigProviderFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
import static org.keycloak.connections.infinispan.InfinispanConnectionProvider.ALL_CACHES_NAME;
import static org.keycloak.connections.infinispan.InfinispanConnectionProvider.CLUSTERED_MAX_COUNT_CACHES;
import static org.keycloak.connections.infinispan.InfinispanConnectionProvider.LOCAL_CACHE_NAMES;
import static org.keycloak.spi.infinispan.impl.embedded.JGroupsConfigurator.createJGroupsProperties;

/**
* The default implementation of {@link CacheEmbeddedConfigProviderFactory}.
Expand Down Expand Up @@ -121,6 +122,7 @@ public List<ProviderConfigProperty> getConfigMetadata() {
Stream.concat(Arrays.stream(LOCAL_CACHE_NAMES), Arrays.stream(CLUSTERED_MAX_COUNT_CACHES))
.forEach(name -> Util.copyFromOption(builder, CacheConfigurator.maxCountConfigKey(name), "max-count", ProviderConfigProperty.INTEGER_TYPE, CachingOptions.maxCountOption(name), false));
createTopologyProperties(builder);
createJGroupsProperties(builder);
return builder.build();
}

Expand Down
101 changes: 93 additions & 8 deletions ...finispan/src/main/java/org/keycloak/spi/infinispan/impl/embedded/JGroupsConfigurator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,31 @@

package org.keycloak.spi.infinispan.impl.embedded;

import static org.infinispan.configuration.global.TransportConfiguration.STACK;
import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_PREFIX;

import java.lang.invoke.MethodHandles;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Objects;

import javax.net.ssl.KeyManager;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLServerSocket;
import javax.net.ssl.TrustManager;

import org.infinispan.commons.configuration.attributes.Attribute;
import org.infinispan.configuration.global.TransportConfigurationBuilder;
import org.infinispan.configuration.parsing.ConfigurationBuilderHolder;
import org.infinispan.remoting.transport.jgroups.EmbeddedJGroupsChannelConfigurator;
import org.infinispan.remoting.transport.jgroups.JGroupsTransport;
import org.jboss.logging.Logger;
import org.jgroups.Global;
import org.jgroups.conf.ClassConfigurator;
import org.jgroups.conf.ProtocolConfiguration;
import org.jgroups.protocols.TCP;
Expand All @@ -40,6 +51,8 @@
import org.jgroups.util.DefaultSocketFactory;
import org.jgroups.util.SocketFactory;
import org.keycloak.Config;
import org.keycloak.config.CachingOptions;
import org.keycloak.config.Option;
import org.keycloak.connections.infinispan.InfinispanConnectionProvider;
import org.keycloak.connections.infinispan.InfinispanConnectionSpi;
import org.keycloak.connections.jpa.JpaConnectionProvider;
Expand All @@ -48,15 +61,10 @@
import org.keycloak.infinispan.util.InfinispanUtils;
import org.keycloak.jgroups.protocol.KEYCLOAK_JDBC_PING2;
import org.keycloak.models.KeycloakSession;
import org.keycloak.provider.ProviderConfigProperty;
import org.keycloak.provider.ProviderConfigurationBuilder;
import org.keycloak.spi.infinispan.JGroupsCertificateProvider;

import javax.net.ssl.KeyManager;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLServerSocket;
import javax.net.ssl.TrustManager;

import static org.infinispan.configuration.global.TransportConfiguration.STACK;
import org.keycloak.spi.infinispan.impl.Util;

/**
* Utility class to configure JGroups based on the Keycloak configuration.
Expand Down Expand Up @@ -88,6 +96,7 @@ public static void configureJGroups(Config.Scope config, ConfigurationBuilderHol
if (stack != null) {
transportOf(holder).stack(stack);
}
configureTransport(config);
configureDiscovery(holder, session);
configureTls(holder, session);
warnDeprecatedStack(holder);
Expand Down Expand Up @@ -126,6 +135,17 @@ public static void configureTopology(Config.Scope config, ConfigurationBuilderHo
}
}

static void createJGroupsProperties(ProviderConfigurationBuilder builder) {
Util.copyFromOption(builder, SystemProperties.BIND_ADDRESS.configKey, "address", ProviderConfigProperty.STRING_TYPE, CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_ADDRESS, false);
Util.copyFromOption(builder, SystemProperties.BIND_PORT.configKey, "port", ProviderConfigProperty.INTEGER_TYPE, CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_PORT, false);
Util.copyFromOption(builder, SystemProperties.EXTERNAL_ADDRESS.configKey, "address", ProviderConfigProperty.STRING_TYPE, CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS, false);
Util.copyFromOption(builder, SystemProperties.EXTERNAL_PORT.configKey, "port", ProviderConfigProperty.INTEGER_TYPE, CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT, false);
}

private static void configureTransport(Config.Scope config) {
Arrays.stream(SystemProperties.values()).forEach(p -> p.set(config));
}

private static void configureTls(ConfigurationBuilderHolder holder, KeycloakSession session) {
var provider = session.getProvider(JGroupsCertificateProvider.class);
if (provider == null || !provider.isEnabled()) {
Expand Down Expand Up @@ -269,4 +289,69 @@ public void afterCreation(Protocol protocol) {
}
}
}

private enum SystemProperties {
BIND_ADDRESS(CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_ADDRESS, Global.BIND_ADDR, "jgroups.bind.address"),
BIND_PORT(CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_PORT, Global.BIND_PORT, "jgroups.bind.port"),
EXTERNAL_ADDRESS(CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS, Global.EXTERNAL_ADDR),
EXTERNAL_PORT(CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT, Global.EXTERNAL_PORT);

final Option<?> option;
final String property;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be a list. jgroups.bind.address is an Infinispan property, but public static final String BIND_ADDR="jgroups.bind_addr"; needs to be set too. Same for the bind port.
Using a custom JGroups configuration file won't be able to read the Infinispan properties.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Me and Pedro discussed this and the jgroups.bind_addr system property always takes precedence over the Infinispan property jgroups.bind.address and any values configured in the JGroup's stack, so it's acceptable to just set the jgroups.bind_addr property.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have also made it so that we warn if either jgroups.bind_addr or jgroups.bind.address is set with the CLI option.

final String altProperty;
final String configKey;

SystemProperties(Option<?> option, String property) {
this(option, property, null);
}

SystemProperties(Option<?> option, String property, String altProperty) {
this.option = option;
this.property = property;
this.altProperty = altProperty;
this.configKey = configKey();
}

void set(Config.Scope config) {
String userConfig = fromConfig(config);
if (userConfig == null) {
// User property is either already set or missing, so do nothing
return;
}
checkPropertyAlreadySet(userConfig, property);
if (altProperty != null)
checkPropertyAlreadySet(userConfig, altProperty);
System.setProperty(property, userConfig);
}

void checkPropertyAlreadySet(String userValue, String property) {
String userProp = System.getProperty(property);
if (userProp != null) {
logger.warnf("Conflicting system property '%s' and CLI arg '%s' set, utilising CLI value '%s'",
property, option.getKey(), userValue);
System.clearProperty(property);
}
}

String fromConfig(Config.Scope config) {
if (option.getType() == Integer.class) {
Integer val = config.getInt(configKey);
return val == null ? null : val.toString();
}
return config.get(configKey);
}

String configKey() {
// Strip the scope from the key and convert to camelCase
String key = option.getKey().substring(CACHE_EMBEDDED_PREFIX.length() + 1);
StringBuilder sb = new StringBuilder(key);
for (int i = 0; i < sb.length(); i++) {
if (sb.charAt(i) == '-') {
sb.deleteCharAt(i);
sb.replace(i, i+1, String.valueOf(Character.toUpperCase(sb.charAt(i))));
}
}
return sb.toString();
}
}
}
28 changes: 27 additions & 1 deletion quarkus/config-api/src/main/java/org/keycloak/config/CachingOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,18 @@ public class CachingOptions {

public static final String CACHE_CONFIG_FILE_PROPERTY = "cache-config-file";

private static final String CACHE_EMBEDDED_PREFIX = "cache-embedded";
public static final String CACHE_EMBEDDED_PREFIX = "cache-embedded";
private static final String CACHE_EMBEDDED_MTLS_PREFIX = CACHE_EMBEDDED_PREFIX + "-mtls";
public static final String CACHE_EMBEDDED_MTLS_ENABLED_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-enabled";
public static final String CACHE_EMBEDDED_MTLS_KEYSTORE_FILE_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-key-store-file";
public static final String CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-key-store-password";
public static final String CACHE_EMBEDDED_MTLS_TRUSTSTORE_FILE_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-trust-store-file";
public static final String CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-trust-store-password";
public static final String CACHE_EMBEDDED_MTLS_ROTATION_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-rotation-interval-days";
public static final String CACHE_EMBEDDED_NETWORK_BIND_ADDRESS_PROPERTY = CACHE_EMBEDDED_PREFIX + "-network-bind-address";
public static final String CACHE_EMBEDDED_NETWORK_BIND_PORT_PROPERTY = CACHE_EMBEDDED_PREFIX + "-network-bind-port";
public static final String CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS_PROPERTY = CACHE_EMBEDDED_PREFIX + "-network-external-address";
public static final String CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT_PROPERTY = CACHE_EMBEDDED_PREFIX + "-network-external-port";

private static final String CACHE_REMOTE_PREFIX = "cache-remote";
public static final String CACHE_REMOTE_HOST_PROPERTY = CACHE_REMOTE_PREFIX + "-host";
Expand Down Expand Up @@ -109,6 +113,28 @@ public String toString() {
.description("Rotation period in days of automatic JGroups MTLS certificates.")
.build();

public static final Option<String> CACHE_EMBEDDED_NETWORK_BIND_ADDRESS = new OptionBuilder<>(CACHE_EMBEDDED_NETWORK_BIND_ADDRESS_PROPERTY, String.class)
.category(OptionCategory.CACHE)
.description("IP address used by clustering transport. By default, SITE_LOCAL is used.")
.build();

public static final Option<Integer> CACHE_EMBEDDED_NETWORK_BIND_PORT = new OptionBuilder<>(CACHE_EMBEDDED_NETWORK_BIND_PORT_PROPERTY, Integer.class)
.category(OptionCategory.CACHE)
.description("The Port the clustering transport will bind to. By default, port 7800 is used.")
.build();

public static final Option<String> CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS = new OptionBuilder<>(CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS_PROPERTY, String.class)
.category(OptionCategory.CACHE)
.description("IP address that other instances in the cluster should use to contact this node. Set only if it is " +
"different to %s, for example when this instance is behind a firewall.".formatted(CACHE_EMBEDDED_NETWORK_BIND_ADDRESS_PROPERTY))
.build();

public static final Option<Integer> CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT = new OptionBuilder<>(CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT_PROPERTY, Integer.class)
.category(OptionCategory.CACHE)
.description("Port that other instances in the cluster should use to contact this node. Set only if it is different " +
"to %s, for example when this instance is behind a firewall".formatted(CACHE_EMBEDDED_NETWORK_BIND_PORT_PROPERTY))
.build();

public static final Option<String> CACHE_REMOTE_HOST = new OptionBuilder<>(CACHE_REMOTE_HOST_PROPERTY, String.class)
.category(OptionCategory.CACHE)
.description("The hostname of the external Infinispan cluster.")
Expand Down
30 changes: 26 additions & 4 deletions .../main/java/org/keycloak/quarkus/runtime/configuration/mappers/CachingPropertyMappers.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package org.keycloak.quarkus.runtime.configuration.mappers;

import static org.keycloak.quarkus.runtime.configuration.Configuration.getOptionalKcValue;
import static org.keycloak.quarkus.runtime.configuration.mappers.PropertyMapper.fromOption;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
Expand All @@ -8,8 +11,6 @@
import java.util.Optional;
import java.util.function.BooleanSupplier;

import com.google.common.base.CaseFormat;
import io.smallrye.config.ConfigSourceInterceptorContext;
import org.keycloak.common.Profile;
import org.keycloak.config.CachingOptions;
import org.keycloak.config.Option;
Expand All @@ -19,8 +20,9 @@
import org.keycloak.quarkus.runtime.configuration.Configuration;
import org.keycloak.utils.StringUtil;

import static org.keycloak.quarkus.runtime.configuration.Configuration.getOptionalKcValue;
import static org.keycloak.quarkus.runtime.configuration.mappers.PropertyMapper.fromOption;
import com.google.common.base.CaseFormat;

import io.smallrye.config.ConfigSourceInterceptorContext;

final class CachingPropertyMappers {

Expand Down Expand Up @@ -98,6 +100,26 @@ public static PropertyMapper<?>[] getClusteringPropertyMappers() {
.isEnabled(() -> Configuration.isTrue(CachingOptions.CACHE_EMBEDDED_MTLS_ENABLED), "property '%s' is enabled".formatted(CachingOptions.CACHE_EMBEDDED_MTLS_ENABLED.getKey()))
.validator(CachingPropertyMappers::validateCertificateRotationIsPositive)
.build(),
fromOption(CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_ADDRESS)
.paramLabel("address")
.to("kc.spi-cache-embedded--default--network-bind-address")
.isEnabled(CachingPropertyMappers::cacheSetToInfinispan, "Infinispan clustered embedded is enabled")
.build(),
fromOption(CachingOptions.CACHE_EMBEDDED_NETWORK_BIND_PORT)
.paramLabel("port")
.to("kc.spi-cache-embedded--default--network-bind-port")
.isEnabled(CachingPropertyMappers::cacheSetToInfinispan, "Infinispan clustered embedded is enabled")
.build(),
fromOption(CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_ADDRESS)
.paramLabel("address")
.to("kc.spi-cache-embedded--default--network-external-address")
.isEnabled(CachingPropertyMappers::cacheSetToInfinispan, "Infinispan clustered embedded is enabled")
.build(),
fromOption(CachingOptions.CACHE_EMBEDDED_NETWORK_EXTERNAL_PORT)
.paramLabel("port")
.isEnabled(CachingPropertyMappers::cacheSetToInfinispan, "Infinispan clustered embedded is enabled")
.to("kc.spi-cache-embedded--default--network-external-port")
.build(),
fromOption(CachingOptions.CACHE_REMOTE_HOST)
.paramLabel("hostname")
.to("kc.spi-cache-remote--default--hostname")
Expand Down
Loading
Loading
点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载