Unable to use custom email sender provider when running in production mode.

Describe the bug

I have a custom provider that implements EmailSenderProvider. This custom provider works perfectly fine when running the Keycloak server in development mode. However, when running the Keycloak server in production mode, it seems to revert to the DefaultEmailSenderProvider.

My custom provider sends emails using an external service and they have their own custom java package as a wrapper for API calls. I have migrated some of my other custom providers to Keycloak v18.0.2 (Quarkus) without any issues. I followed this documentation provided.

While running my Keycloak server in development mode, I attached a remote debugger and was able to step through my custom email provider code without any issue and was getting emails as expected via the third party service. But when I start the server in production mode, it does not reach my custom provider at all.

This is my docker-compose file for my Keycloak server, some parts have been omitted for brevity.

---
version: '3.9'

services:
    keycloak:
      build: .
      environment:
        - DEBUG_PORT='*:9101'
        - ENVIRONMENT=${ENVIRONMENT:-local}
        - KC_DB=${DB_VENDOR:-postgres}
        - KC_DB_PASSWORD=${DB_PASSWORD:-postgres}
        - KC_DB_URL_HOST=${DB_ADDR:-keycloak-database}
        - KC_DB_URL_PORT=${DB_PORT:-5432}
        - KC_DB_USERNAME=${DB_USERNAME:-keycloak}
        - KC_FEATURES=admin-fine-grained-authz,token-exchange
        - KC_HEALTH_ENABLED=true
        - KC_HTTP_RELATIVE_PATH=/auth
        - KC_HTTPS_CERTIFICATE_FILE=/opt/keycloak/conf/server.crt.pem
        - KC_HTTPS_CERTIFICATE_KEY_FILE=/opt/keycloak/conf/server.key.pem
        - KC_HTTPS_PORT=8080
        - KEYCLOAK_ADMIN=${KEYCLOAK_USER:-admin}
        - KEYCLOAK_ADMIN_PASSWORD=${KEYCLOAK_PASSWORD:-password}
      networks:
        - database
        - keycloak
      volumes:
        - ./local/realm-template/local-realm.template:/opt/keycloak/data/import/local-realm.json:ro
        - ./local/conf/keycloak.conf:/opt/keycloak/conf/keycloak.conf:ro
        - ./local/conf/server.crt.pem:/opt/keycloak/conf/server.crt.pem:ro
        - ./local/conf/server.key.pem:/opt/keycloak/conf/server.key.pem:ro
      healthcheck:
        test: curl --fail -w "%{http_code}" -k http://localhost:8080/auth/health/ -o /dev/null -s || exit 1
        interval: 1m30s
        timeout: 10s
        retries: 3
      entrypoint: /bin/sh
      command: >
        -c
        "/opt/keycloak/bin/kc.sh 
        build
        && /opt/keycloak/bin/kc.sh 
        --verbose 
        start
        --import-realm 
        --debug
        --spi-email-sender-provider=customEmailSender"
      ports:
        - "8080:8080"
        - "9101:9101"
    ....
  networks:
    keycloak:

Keycloak Dockerfile:

ARG DOCKER_REGISTRY=quay.io
ARG DOCKER_BASE_IMAGE_REGISTRY=${DOCKER_REGISTRY}
ARG KEYCLOAK_IMAGE_NAME=keycloak/keycloak
ARG DOCKER_BASE_IMAGE_NAME=${KEYCLOAK_IMAGE_NAME}
ARG KEYCLOAK_VERSION=18.0.2
ARG DOCKER_BASE_IMAGE_TAG=${KEYCLOAK_VERSION}
FROM ${DOCKER_BASE_IMAGE_REGISTRY}/${DOCKER_BASE_IMAGE_NAME}:${DOCKER_BASE_IMAGE_TAG}
USER 1000

COPY ./providers /opt/keycloak/providers

Keycloak configuration output when the server is running:

Current Mode: none
Runtime Configuration:
	kc.cache =  ispn (PersistedConfigSource)
	kc.config.args =  show-config (SysPropConfigSource)
	kc.db =  postgres (PropertiesConfigSource[source=file:/opt/keycloak/bin/../conf/keycloak.conf])
	kc.db-url-host =  keycloak-database (KcEnvVarConfigSource)
	kc.db-url-port =  5432 (KcEnvVarConfigSource)
	kc.db.password =  postgres (EnvConfigSource)
	kc.db.url.host =  keycloak-database (EnvConfigSource)
	kc.db.url.port =  5432 (EnvConfigSource)
	kc.db.username =  keycloak (EnvConfigSource)
	kc.features =  admin-fine-grained-authz,token-exchange (KcEnvVarConfigSource)
	kc.health-enabled =  true (PropertiesConfigSource[source=file:/opt/keycloak/bin/../conf/keycloak.conf])
	kc.home.dir =  /opt/keycloak/bin/../ (SysPropConfigSource)
	kc.hostname =  localhost (PropertiesConfigSource[source=file:/opt/keycloak/bin/../conf/keycloak.conf])
	kc.http-enabled =  false (PropertiesConfigSource[source=jar:file:///opt/keycloak/lib/lib/main/org.keycloak.keycloak-quarkus-server-18.0.2.jar!/META-INF/keycloak.conf])
	kc.http-relative-path =  /auth (EnvConfigSource)
	kc.https.certificate.file =  /opt/keycloak/conf/server.crt.pem (EnvConfigSource)
	kc.https.certificate.key.file =  /opt/keycloak/conf/server.key.pem (EnvConfigSource)
	kc.https.port =  8080 (EnvConfigSource)
	kc.log-console-output =  default (PropertiesConfigSource[source=jar:file:///opt/keycloak/lib/lib/main/org.keycloak.keycloak-quarkus-server-18.0.2.jar!/META-INF/keycloak.conf])
	kc.log-file =  /opt/keycloak/bin/../data/log/keycloak.log (PropertiesConfigSource[source=jar:file:///opt/keycloak/lib/lib/main/org.keycloak.keycloak-quarkus-server-18.0.2.jar!/META-INF/keycloak.conf])
	kc.metrics-enabled =  false (PersistedConfigSource)
	kc.provider.file.custom-provider-1.4.0-SNAPSHOT-jar-with-dependencies.jar.last-modified =  1661348195202 (PersistedConfigSource)
	kc.quarkus-properties-enabled =  false (PersistedConfigSource)
	kc.show.config =  none (SysPropConfigSource)
	kc.version =  18.0.2 (SysPropConfigSource)

In the legacy wildfly distribution, My standalone.xml / standalone-ha.xml had the following:

            <spi name="emailSender">
                <provider name="default" enabled="false"/>
                <provider name="customEmailSender" enabled="true"/>
            </spi>

I must be missing something in my current Keycloak server configuration, however doesn’t really explain why it works as expected in development mode, but reverts to the default email sender when running in production. Would anyone be able to lend a hand? Additionally if there is improvements to be made in my docker-compose file, feedback is welcomed.

Version

18.0.2

Expected behavior

When sending any type of email action for example, Update password. The custom email sender provider is used, rather than the default email provider.

Actual behavior

When attempting to send any type of email action for example, Update password Keycloak is reverting to the default email provider:

2022-09-21 08:22:56,674 ERROR [org.keycloak.services] (executor-thread-50) KC-SERVICES0088: Failed to send execute actions email: org.keycloak.email.EmailException: org.keycloak.email.EmailException: Please provide a valid address
	at org.keycloak.email.DefaultEmailSenderProvider.send(DefaultEmailSenderProvider.java:153)
	at org.keycloak.email.freemarker.FreeMarkerEmailTemplateProvider.send(FreeMarkerEmailTemplateProvider.java:264)
	at org.keycloak.email.freemarker.FreeMarkerEmailTemplateProvider.send(FreeMarkerEmailTemplateProvider.java:259)
	at org.keycloak.email.freemarker.FreeMarkerEmailTemplateProvider.send(FreeMarkerEmailTemplateProvider.java:250)
	at org.keycloak.email.freemarker.FreeMarkerEmailTemplateProvider.send(FreeMarkerEmailTemplateProvider.java:207)
	at org.keycloak.email.freemarker.FreeMarkerEmailTemplateProvider.sendExecuteActions(FreeMarkerEmailTemplateProvider.java:170)
	at org.keycloak.services.resources.admin.UserResource.executeActionsEmail(UserResource.java:832)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:192)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:152)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:183)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:152)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:183)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:152)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:183)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:141)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:32)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:151)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.handle(VertxRequestHandler.java:82)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.handle(VertxRequestHandler.java:42)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:201)
	at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:67)
	at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:55)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:201)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$5.handle(VertxHttpRecorder.java:380)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$5.handle(VertxHttpRecorder.java:358)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:201)
	at org.keycloak.quarkus.runtime.integration.web.QuarkusRequestFilter.lambda$createBlockingHandler$1(QuarkusRequestFilter.java:71)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.keycloak.email.EmailException: Please provide a valid address
	at org.keycloak.email.DefaultEmailSenderProvider.toInternetAddress(DefaultEmailSenderProvider.java:167)
	at org.keycloak.email.DefaultEmailSenderProvider.send(DefaultEmailSenderProvider.java:126)

������
	... 61 more

How to Reproduce?

• Create a custom email sender provider to override EmailSenderProvider.
• Start your Keycloak server in production mode.

Anything else?

https://keycloak.discourse.group/t/keyloak-17-18-override-internal-spi/15253/3

[lexcao]

Hi, I just forgot it was named auto-build at its first introduction. 🥲 Thanks for your detailed investigation. 🎉

[kwame-mintah]

@lexcao thanks for the help and responding to the issue raised, very much appreciated. I’m going to close this issue as I have been able to get my custom email provider to start working again. I have updated my docker-compose command to no longer use an entrypoint and not run a build at the start:

...
      command: [
        "--verbose",
        "start",
        "--auto-build",
        "--import-realm",
        "--spi-email-sender-provider=customEmailSender"
      ]
...