Can not set a jgroups stack other than the defaults from Infinispan

Describe the bug

It is not possible to configure a custom JGroups stack to the cache container but use only those provided OOTB.

Because of that, it is not possible to configure a JGroups stack with additional protocols such as ASYM_ENCRYPT.

The issue does not happen in 16.1.1 but is in upstream/main. It is related to the ISPN upgrade from 13.0.0 to 13.0.5.

Version

17.0.0-SNAPSHOT

Expected behavior

The server must allow user-defined jgroups stacks.

Actual behavior

The udp stack is always set as the JGroups stack, ignoring any custom stack configured in the cache configuration file. The reason is that it is set as a default option value.

How to Reproduce?

Create a custom JGroups stack by creating a jgroups-asym-enc.xml with the following content:

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.2.xsd">
   <!-- jgroups.udp.address is deprecated and will be removed, see ISPN-11867 -->
   <UDP bind_addr="${jgroups.bind.address,jgroups.udp.address:SITE_LOCAL}"
        bind_port="${jgroups.bind.port,jgroups.udp.port:0}"
        mcast_addr="${jgroups.mcast_addr:228.6.7.8}"
        mcast_port="${jgroups.mcast_port:46655}"
        tos="0"
        ucast_send_buf_size="1m"
        mcast_send_buf_size="1m"
        ucast_recv_buf_size="20m"
        mcast_recv_buf_size="25m"
        ip_ttl="${jgroups.ip_ttl:2}"
        thread_naming_pattern="pl"
        enable_diagnostics="false"
        bundler_type="transfer-queue"
        max_bundle_size="8500"

        thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
        thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
        thread_pool.keep_alive_time="60000"

        thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}"
   />
   <PING num_discovery_runs="3"/>
   <MERGE3 min_interval="10000"
           max_interval="30000"
   />
   <FD_SOCK/>
   <!-- Suspect node `timeout` to `timeout + timeout_check_interval` millis after the last heartbeat -->
   <FD_ALL timeout="10000"
           interval="2000"
           timeout_check_interval="1000"
   />
   <VERIFY_SUSPECT timeout="1000"/>
   <SSL_KEY_EXCHANGE keystore_name="server.jks"
                        keystore_password="password"/>
<ASYM_ENCRYPT asym_keylength="2048"
                    asym_algorithm="RSA"
                    change_key_on_coord_leave = "false"
                    change_key_on_leave = "false"
                    use_external_key_exchange = "true"
                    />

   <pbcast.NAKACK2 xmit_interval="100"
                   xmit_table_num_rows="50"
                   xmit_table_msgs_per_row="1024"
                   xmit_table_max_compaction_time="30000"
                   resend_last_seqno="true"
   />
   <UNICAST3 xmit_interval="100"
             xmit_table_num_rows="50"
             xmit_table_msgs_per_row="1024"
             xmit_table_max_compaction_time="30000"
   />
   <pbcast.STABLE desired_avg_gossip="5000"
                  max_bytes="1M"
   />
   <pbcast.GMS print_local_addr="false"
               join_timeout="${jgroups.join_timeout:2000}"
   />
   <UFC max_credits="4m"
        min_threshold="0.40"
   />
   <MFC max_credits="4m"
        min_threshold="0.40"
   />
   <FRAG3 frag_size="8000"/>
</config>

Create a cache configuration file conf/cache-ispn-secure.xml:

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

<jgroups>
    <stack name="encrypt-udp" extends="udp">
      <SSL_KEY_EXCHANGE keystore_name="server1.jks"
                        keystore_password="password"
                        stack.combine="INSERT_AFTER"
                        stack.position="VERIFY_SUSPECT"/>
      <ASYM_ENCRYPT asym_keylength="2048"
                    asym_algorithm="RSA"
                    change_key_on_coord_leave = "false"
                    change_key_on_leave = "false"
                    use_external_key_exchange = "true"
                    stack.combine="INSERT_BEFORE"
                    stack.position="pbcast.NAKACK2"/>
    </stack>
  </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="encrypt-udp"/>
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>            
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>

Build the server as follows:

kc.sh build --cache-config-file cache-ispn-secure.xml

Then try to start two server instances where one of them does not have the proper key material to join the cluster.

Anything else?

No response

[thomasdarimont]

@xgp I think your JDBC url is wrong. It should be jdbc:postgresql://... instead of jdbc:postgres://....

The following JDBC_PING stack element worked fine for me with Keycloak 17:

    <JDBC_PING connection_driver="org.postgresql.Driver"
               connection_username="keycloak"
               connection_password="keycloak"
               connection_url="jdbc:postgresql://acme-keycloak-db/keycloak"
               initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
               info_writer_sleep_time="500"
               remove_all_data_on_view_change="true"
    />

You can check which stack is used by creating a breakpoint in org.jgroups.JChannel#init(org.jgroups.conf.ProtocolStackConfigurator)

[pedroigor]

@dev-e @hariubc Thanks. We can improve docs for sure.

We could also make the server smarter about this and ignore cache-stack if a stack is defined in the configuration file, but I would avoid doing it now because the caching config is about to change with the introduction of the new store.

Created https://github.com/keycloak/keycloak/issues/12862.