Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Latest Keycloak Container errors on database migrations during startup

See original GitHub issue

Describe the bug

The previous version of the Keycloak container was running perfectly well in my Docker/Nomadproject.io setup, however the most recent version of the container fails with a Wyfly error

Version

12:48:11,770 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 2.0.0.Final
12:48:11,810 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.13.Final
12:48:11,816 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
12:48:11,888 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 16.1.0 (WildFly Core 18.0.0.Final) starting
12:48:12,361 INFO  [org.wildfly.security] (ServerService Thread Pool -- 19) ELY00001: WildFly Elytron version 1.18.1.Final

Expected behavior

The container should start correctly and perform any database migrations that are outstanding

Actual behavior

with the following stack trace:

11:20:18,724 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 2) WFLYCTL0403: Unexpected failure during execution of the following operation(s): [{
    "operation" => "add",
    "address" => [("subsystem" => "infinispan")]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb")
    ],
    "default-cache" => "dist",
    "marshaller" => "PROTOSTREAM",
    "aliases" => ["sfsb"],
    "modules" => ["org.wildfly.clustering.ejb.infinispan"]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb"),
        ("transport" => "jgroups")
    ],
    "lock-timeout" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb"),
        ("distributed-cache" => "dist")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb"),
        ("distributed-cache" => "dist"),
        ("component" => "locking")
    ],
    "isolation" => "REPEATABLE_READ"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb"),
        ("distributed-cache" => "dist"),
        ("component" => "transaction")
    ],
    "mode" => "BATCH"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "ejb"),
        ("distributed-cache" => "dist"),
        ("store" => "file")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak")
    ],
    "marshaller" => "JBOSS",
    "modules" => ["org.keycloak.keycloak-model-infinispan"]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("transport" => "jgroups")
    ],
    "lock-timeout" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "realms")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "realms"),
        ("memory" => "heap")
    ],
    "size" => 10000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "users")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "users"),
        ("memory" => "heap")
    ],
    "size" => 10000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "authorization")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "authorization"),
        ("memory" => "heap")
    ],
    "size" => 10000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "keys")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "keys"),
        ("memory" => "heap")
    ],
    "size" => 1000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("local-cache" => "keys"),
        ("component" => "expiration")
    ],
    "max-idle" => 3600000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("replicated-cache" => "work")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("replicated-cache" => "work"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "sessions")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "sessions"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "authenticationSessions")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "authenticationSessions"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "offlineSessions")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "offlineSessions"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "clientSessions")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "clientSessions"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "offlineClientSessions")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "offlineClientSessions"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "loginFailures")
    ],
    "owners" => 1
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "loginFailures"),
        ("component" => "expiration")
    ],
    "lifespan" => 900000000000000000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "actionTokens")
    ],
    "owners" => 2
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "actionTokens"),
        ("memory" => "heap")
    ],
    "size" => -1L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "keycloak"),
        ("distributed-cache" => "actionTokens"),
        ("component" => "expiration")
    ],
    "interval" => 300000L,
    "lifespan" => 900000000000000000L,
    "max-idle" => -1L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "server")
    ],
    "default-cache" => "default",
    "marshaller" => "PROTOSTREAM",
    "aliases" => [
        "singleton",
        "cluster"
    ],
    "modules" => ["org.wildfly.clustering.server"]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "server"),
        ("transport" => "jgroups")
    ],
    "lock-timeout" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "server"),
        ("replicated-cache" => "default")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "server"),
        ("replicated-cache" => "default"),
        ("component" => "transaction")
    ],
    "mode" => "BATCH"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web")
    ],
    "default-cache" => "dist",
    "marshaller" => "PROTOSTREAM",
    "modules" => ["org.wildfly.clustering.web.infinispan"]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("transport" => "jgroups")
    ],
    "lock-timeout" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("replicated-cache" => "sso")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("replicated-cache" => "sso"),
        ("component" => "locking")
    ],
    "isolation" => "REPEATABLE_READ"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("replicated-cache" => "sso"),
        ("component" => "transaction")
    ],
    "mode" => "BATCH"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("distributed-cache" => "dist")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("distributed-cache" => "dist"),
        ("component" => "locking")
    ],
    "isolation" => "REPEATABLE_READ"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("distributed-cache" => "dist"),
        ("component" => "transaction")
    ],
    "mode" => "BATCH"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("distributed-cache" => "dist"),
        ("store" => "file")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "web"),
        ("distributed-cache" => "routing")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate")
    ],
    "marshaller" => "JBOSS",
    "modules" => ["org.infinispan.hibernate-cache"]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("transport" => "jgroups")
    ],
    "lock-timeout" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("local-cache" => "local-query")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("local-cache" => "local-query"),
        ("memory" => "heap")
    ],
    "size" => 10000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("local-cache" => "local-query"),
        ("component" => "expiration")
    ],
    "max-idle" => 100000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("local-cache" => "pending-puts")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("local-cache" => "pending-puts"),
        ("component" => "expiration")
    ],
    "max-idle" => 60000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("invalidation-cache" => "entity")
    ]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("invalidation-cache" => "entity"),
        ("component" => "transaction")
    ],
    "mode" => "NON_XA"
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("invalidation-cache" => "entity"),
        ("memory" => "heap")
    ],
    "size" => 10000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("invalidation-cache" => "entity"),
        ("component" => "expiration")
    ],
    "max-idle" => 100000L
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "infinispan"),
        ("cache-container" => "hibernate"),
        ("replicated-cache" => "timestamps")
    ]
}]: java.lang.RuntimeException: WFLYCTL0195: Interrupted awaiting transaction commit or rollback
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTransactionControl.operationPrepared(ParallelBootOperationStepHandler.java:458)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ModelController$OperationTransactionControl.operationPrepared(ModelController.java:131)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:872)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:803)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)

11:20:18,723 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0403: Unexpected failure during execution of the following operation(s): [{
    "operation" => "add",
    "address" => [("subsystem" => "security-manager")]
}, {
    "operation" => "add",
    "address" => [
        ("subsystem" => "security-manager"),
        ("deployment-permissions" => "default")
    ],
    "maximum-permissions" => [{"class" => "java.security.AllPermission"}]
}]: java.lang.RuntimeException: WFLYCTL0195: Interrupted awaiting transaction commit or rollback
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTransactionControl.operationPrepared(ParallelBootOperationStepHandler.java:458)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ModelController$OperationTransactionControl.operationPrepared(ModelController.java:131)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:872)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:803)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
	at org.jboss.as.controller@18.0.0.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)

How to Reproduce?

Launch the latest Keycloak container with the following configuration flags set:

DB_ADDR=db.service
DB_VENDOR=mariadb
DB_PORT=3306
DB_USER="keycloak"
DB_PASSWORD="my keycloak password"
KEYCLOAK_USER="admin"
KEYCLOAK_PASSWORD="kc_admin_password"
PROXY_ADDRESS_FORWARDING=true
KEYCLOAK_HOSTNAME="admin.service.my.tld"
KEYCLOAK_FRONTEND_URL="admin.service.my.tld"

Anything else?

I am not a java developer, so I’m struggling to understand what this error actually means.

The container has access to the internet, the backend database exists and the credentials work, and the container has been provided with 1024Mhz of CPU and 4GB of RAM.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
MrDeerlycommented, Jan 6, 2022

I can confirm that this also happens with the “older” 15.X versions. Some more details about the issue: https://issues.redhat.com/browse/KEYCLOAK-6411

Occurs with 6.0.1 when using mariadb java client version 2.4.1 => http://central.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.4.1/mariadb-java-client-2.4.1.jar Does not occur for http://central.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.3.0/mariadb-java-client-2.3.0.jar

Looks like the change in MariaDB 10.3.4 which added “ADD PERIOD FOR SYSTEM_TIME” (https://mariadb.com/kb/en/library/alter-table/#add-period-for-system_time) for the “ALTER TABLE” command is the problem. Old version of mariadb-java-client seems to call “ALTER TABLE ADD col_name” instead of “ALTER TABLE ADD COLUMN col_name” which would not trigger this problem. To temporary solve this issue, the following attribute could be added to the problematic liquibase changeset: objectQuotingStrategy=“QUOTE_ALL_OBJECTS”

0reactions
proffalkencommented, Feb 14, 2022

@stianst good spot, thanks, I’ll disable that now, thank you!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Keycloak HA - Failing to startup due to database exception #525
This can be caused by a failed migration (like the pod was terminated in the middle of the migration without a clean rollback)...
Read more >
Upgrading Guide - Keycloak
Keycloak can automatically migrate the database schema, or you can choose to do it manually. By default the database is automatically migrated ...
Read more >
Keycloak docker service cannot connect to database ...
Keycloak docker service cannot connect to database. Throwable while attempting to get a new connection: null : Unable to create connection.
Read more >
KEYCLOAK + MYSQL + DOCKER --> Failed to start
According to https://github.com/keycloak/keycloak-containers/blob/master/docker-compose-examples/keycloak-mysql.yml, ...
Read more >
Upgrading 14.0->15.1. MSSQL issue - Getting advice - Keycloak
2021-08-12 10:58:58,769 ERROR ... Error: Migration failed for change set ... Pool -- 62) Error has occurred while updating the database: ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Install failed follow the official guide, configuration error : 'security-realms' isn't an allowed element here

Next page

Refresh token with offline_access scope affected by session idle/session max