Unstable AuthenticationFlowCallbackProviderTest for undertow-map

Describe the bug

The AuthenticationFlowCallbackProviderTest is unstable for base testsuite, specifically for undertow-map.

[INFO] Running org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest
Feb 15, 2022 10:27:04 AM org.jboss.arquillian.drone.webdriver.factory.HtmlUnitDriverFactory setClientOptions
INFO: Setting HtmlDriver web client options: {htmlunitCssEnabled=false, htmlunitHistoryPageCacheLimit=1}
--------- org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest output start ---------
AuthenticationFlowCallbackProviderTest ++ 10:27:04,824 INFO  [org.keycloak.testsuite.arquillian.AppServerTestEnricher] 
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ TEST CONTEXT: org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ 10:27:04,825 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] [AuthenticationFlowCallbackProviderTest] loaEssentialNonExisting() STARTED
AuthenticationFlowCallbackProviderTest ++ 10:27:04,838 INFO  [org.keycloak.testsuite.drone.KeycloakDronePostSetup] Configuring driver settings. implicitWait=1500, pageLoadTimeout=90000
AuthenticationFlowCallbackProviderTest ++ 10:27:04,908 DEBUG [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] loading test realms
AuthenticationFlowCallbackProviderTest ++ 10:27:04,908 DEBUG [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] Adding test realm for import from testrealm.json
AuthenticationFlowCallbackProviderTest ++ 10:27:04,909 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] importing test realms
AuthenticationFlowCallbackProviderTest ++ 10:27:04,909 DEBUG [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] --importing realm: test
AuthenticationFlowCallbackProviderTest ++ 10:27:05,731 INFO  [org.keycloak.testsuite.util.URLUtils] starting navigation
AuthenticationFlowCallbackProviderTest ++ 10:27:05,732 INFO  [org.keycloak.testsuite.util.URLUtils] current URL:  https://localhost:8543/auth/realms/test/login-actions/authenticate?client_id=test-app&tab_id=YkdO9mpU39o
AuthenticationFlowCallbackProviderTest ++ 10:27:05,732 INFO  [org.keycloak.testsuite.util.URLUtils] navigating to https://localhost:8543/auth/realms/test/super-random-page
AuthenticationFlowCallbackProviderTest ++ 10:27:05,750 INFO  [org.keycloak.testsuite.util.URLUtils] new current URL:  https://localhost:8543/auth/realms/test/super-random-page
AuthenticationFlowCallbackProviderTest ++ 10:27:05,750 INFO  [org.keycloak.testsuite.util.URLUtils] navigation complete
AuthenticationFlowCallbackProviderTest ++ 10:27:05,750 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] deleting cookies in 'test' realm
AuthenticationFlowCallbackProviderTest ++ 10:27:05,750 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] calling all TestCleanup
AuthenticationFlowCallbackProviderTest ++ 10:27:05,756 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] [AuthenticationFlowCallbackProviderTest] loaEssentialNonExisting() FINISHED
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ 10:27:05,756 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] [AuthenticationFlowCallbackProviderTest] errorWithCustomProvider() STARTED
AuthenticationFlowCallbackProviderTest ++ 10:27:07,515 INFO  [org.keycloak.testsuite.util.URLUtils] starting navigation
AuthenticationFlowCallbackProviderTest ++ 10:27:07,515 INFO  [org.keycloak.testsuite.util.URLUtils] current URL:  https://localhost:8543/auth/realms/master/app/auth?state=724ab7a1-5474-4096-af7e-f1efe291e12a&session_state=ad4e76e0-eded-4986-90b5-49ded2baa91a&code=c82258b0-3416-4a99-a598-f97c47a983ca.ad4e76e0-eded-4986-90b5-49ded2baa91a.a6381554-9265-493d-9ac5-5253f5ef5472
AuthenticationFlowCallbackProviderTest ++ 10:27:07,515 INFO  [org.keycloak.testsuite.util.URLUtils] navigating to https://localhost:8543/auth/realms/test/super-random-page
AuthenticationFlowCallbackProviderTest ++ 10:27:07,532 INFO  [org.keycloak.testsuite.util.URLUtils] new current URL:  https://localhost:8543/auth/realms/test/super-random-page
AuthenticationFlowCallbackProviderTest ++ 10:27:07,532 INFO  [org.keycloak.testsuite.util.URLUtils] navigation complete
AuthenticationFlowCallbackProviderTest ++ 10:27:07,532 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] deleting cookies in 'test' realm
AuthenticationFlowCallbackProviderTest ++ 10:27:07,532 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] calling all TestCleanup
AuthenticationFlowCallbackProviderTest ++ 10:27:07,549 ERROR [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] [AuthenticationFlowCallbackProviderTest] errorWithCustomProvider() FAILED
AuthenticationFlowCallbackProviderTest ++ 10:27:07,550 INFO  [org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest] [AuthenticationFlowCallbackProviderTest] errorWithCustomProvider() FINISHED
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ 
AuthenticationFlowCallbackProviderTest ++ 10:27:07,550 INFO  [org.keycloak.testsuite.arquillian.AuthServerTestEnricher] removing test realms after test class
AuthenticationFlowCallbackProviderTest ++ 10:27:07,575 INFO  [org.keycloak.testsuite.arquillian.AuthServerTestEnricher] removed realms: test, 

--------- org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest output end  ---------
Error:  Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.783 s <<< FAILURE! - in org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest
Error: OR] org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest.errorWithCustomProvider  Time elapsed: 1.793 s  <<< ERROR!
-- org.openqa.selenium.NoSuchElementException: 
-- Unable to locate element with ID: kc-page-title
-- For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
-- Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
-- System info: host: 'fv-az241-411', ip: '10.1.0.189', os.name: 'Linux', os.arch: 'amd64', os.version: '5.11.0-1028-azure', java.version: '11.0.14'
-- Driver info: driver.version: HtmlUnitDriver
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElementById(HtmlUnitDriver.java:1204)
-- 	at org.openqa.selenium.By$ById.findElement(By.java:188)
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver$4.call(HtmlUnitDriver.java:1970)
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver$4.call(HtmlUnitDriver.java:1)
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver.implicitlyWaitFor(HtmlUnitDriver.java:1606)
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:[1966](https://github.com/keycloak/keycloak/runs/5198202511?check_suite_focus=true#step:8:1966))
-- 	at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:754)
-- 	at org.jboss.arquillian.graphene.enricher.WebElementUtils.dropProxyAndFindElement(WebElementUtils.java:168)
-- 	at org.jboss.arquillian.graphene.enricher.WebElementUtils.findElement(WebElementUtils.java:70)
-- 	at org.jboss.arquillian.graphene.enricher.SearchContextInterceptor.intercept(SearchContextInterceptor.java:46)
-- 	at org.jboss.arquillian.graphene.proxy.InvocationContextImpl.invoke(InvocationContextImpl.java:87)
-- 	at org.jboss.arquillian.graphene.enricher.StaleElementInterceptor$1.apply(StaleElementInterceptor.java:47)
-- 	at org.jboss.arquillian.graphene.enricher.StaleElementInterceptor$1.apply(StaleElementInterceptor.java:43)
-- 	at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:260)
-- 	at org.jboss.arquillian.graphene.wait.WebDriverWaitImpl.until(WebDriverWaitImpl.java:96)
-- 	at org.jboss.arquillian.graphene.enricher.StaleElementInterceptor.intercept(StaleElementInterceptor.java:43)
-- 	at org.jboss.arquillian.graphene.proxy.InvocationContextImpl.invoke(InvocationContextImpl.java:87)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler$2.call(GrapheneContextualHandler.java:241)
-- 	at org.jboss.arquillian.graphene.context.BrowserActions.performAction(BrowserActions.java:62)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler.invoke(GrapheneContextualHandler.java:237)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler.intercept(GrapheneContextualHandler.java:263)
-- 	at org.jboss.arquillian.drone.webdriver.htmlunit.DroneHtmlUnitDriver$$EnhancerByGraphene$$8df30a9.findElement(<generated>)
-- 	at org.keycloak.testsuite.pages.PageUtils.getPageTitle(PageUtils.java:9)
-- 	at org.keycloak.testsuite.pages.ErrorPage.isCurrent(ErrorPage.java:55)
-- 	at org.keycloak.testsuite.pages.AbstractPage.assertCurrent(AbstractPage.java:49)
-- 	at jdk.internal.reflect.GeneratedMethodAccessor1494.invoke(Unknown Source)
-- 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
-- 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneProxyHandler.invokeReal(GrapheneProxyHandler.java:129)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler$1.invoke(GrapheneContextualHandler.java:169)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler$2.call(GrapheneContextualHandler.java:241)
-- 	at org.jboss.arquillian.graphene.context.BrowserActions.performAction(BrowserActions.java:62)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler.invoke(GrapheneContextualHandler.java:237)
-- 	at org.jboss.arquillian.graphene.proxy.GrapheneContextualHandler.intercept(GrapheneContextualHandler.java:263)
-- 	at org.keycloak.testsuite.pages.ErrorPage$$EnhancerByGraphene$$b30292ee.assertCurrent(<generated>)
-- 	at org.keycloak.testsuite.forms.AuthenticationFlowCallbackProviderTest.errorWithCustomProvider(AuthenticationFlowCallbackProviderTest.java:76)
-- 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
-- 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
-- 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
-- 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
-- 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
-- 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
-- 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
-- 	at org.jboss.arquillian.junit.Arquillian$8$1.invokeMethod(Arquillian.java:325)
-- 	at org.jboss.arquillian.junit.MethodInvoker$1.invoke(MethodInvoker.java:18)
-- 	at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:57)
-- 	at org.keycloak.testsuite.arquillian.ModelTestExecutor.execute(ModelTestExecutor.java:49)
-- 	at jdk.internal.reflect.GeneratedMethodAccessor679.invoke(Unknown Source)
-- 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
-- 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
-- 	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
-- 	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
-- 	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
-- 	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
-- 	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
-- 	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
-- 	at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:50)
-- 	at jdk.internal.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
-- 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
-- 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
-- 	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
-- 	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
-- 	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
-- 	at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:128)
-- 	at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:118)
-- 	at jdk.internal.reflect.GeneratedMethodAccessor677.invoke(Unknown Source)
-- 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
-- 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

[mposolda]

@hmlnarik The test was unfortunately failing quite frequently, rough estimate is around 50% of cases. I’ve merged the PR when it was added on Monday and there are at least 2 PRs failing with that test since then: https://github.com/keycloak/keycloak/pull/10261 https://github.com/keycloak/keycloak/pull/10240 and at least one other occurence (when I re-run the PR, so likely not available anymore). So it seems to me much worse than RefreshTokenTest.tokenRefreshRequest_ClientES512_RealmRS256 .

I think we all agree that this needs to be investigated with high prirority and properly fix in Keycloak 18.

But my vote is to re-add the test in the same PR where we also fix it. This is not ideal, but looks to me better alternative from 2 bad alternatives 😃 I am just not very keen to the idea of adding the test, which fails in +/- 50% of cases as this is very often and adding such test has also quite big price to pay (EG. unaware people investigating why their PR are failing, other people without administration rights re-running their job to be able to merge PRs, which adds more pressure to GH actions, which seems to be already overloaden these days etc…)

[mabartos]

@anantk24 Thanks for checking that! The test case assumes the Error page is displayed, but in some rare cases, the authentication is successful and the page is redirected to the default “success” page. However, I’m able to reproduce it after multiple executions of the method, ergo the solution might be quite “achievable”.