Flaky-test: org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail
See original GitHub issueorg.apache.pulsar.broker.service.ReplicatorTest is flaky. The testResetCursorNotFail test method fails sporadically.
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail() didn't finish within the time-out 30000
Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click. example failure 2021-02-03T01:05:58.8564231Z example failure 2021-02-02T16:11:47.7564123Z example failure 2021-02-02T16:03:18.2799391Z example failure 2021-02-02T16:01:08.0326536Z example failure 2021-02-02T13:49:56.6101213Z example failure 2021-02-01T23:49:57.6075482Z example failure 2021-02-01T23:16:15.9997490Z example failure 2021-02-01T20:11:13.5145694Z example failure 2021-02-01T18:24:10.9311136Z example failure 2021-02-01T16:24:25.1910906Z example failure 2021-02-01T15:44:16.9748908Z example failure 2021-02-01T15:42:02.4182452Z example failure 2021-02-01T15:39:41.2878811Z example failure 2021-02-01T09:27:54.9174939Z example failure 2021-02-01T09:25:44.3137183Z example failure 2021-02-01T03:58:24.9345942Z example failure 2021-02-01T03:54:51.8139627Z example failure 2021-01-31T12:12:19.0872946Z example failure 2021-01-21T23:26:21.8643558Z
Full exception stacktrace
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail() didn't finish within the time-out 30000
at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371)
at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605)
at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Client is not authorized to Produce
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:979)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:95)
at org.apache.pulsar.broker.service.ReplicatorTestBase$MessageProducer.<init>(ReplicatorTestBase.java:301)
at org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail(ReplicatorTest.java:507)
Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click. example failure 2021-02-02T13:46:14.3674803Z example failure 2021-02-01T03:56:01.6136236Z
Full exception stacktrace
org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Client is not authorized to Produce
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:979)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:95)
at org.apache.pulsar.broker.service.ReplicatorTestBase$MessageProducer.<init>(ReplicatorTestBase.java:301)
at org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail(ReplicatorTest.java:507)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Client is not authorized to Produce
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:979)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:94)
at org.apache.pulsar.broker.service.ReplicatorTestBase$MessageProducer.<init>(ReplicatorTestBase.java:301)
at org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail(ReplicatorTest.java:507)
Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click. example failure 2021-02-01T01:35:59.5083094Z
Full exception stacktrace
org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Client is not authorized to Produce
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:979)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:94)
at org.apache.pulsar.broker.service.ReplicatorTestBase$MessageProducer.<init>(ReplicatorTestBase.java:301)
at org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail(ReplicatorTest.java:507)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (5 by maintainers)
Yes, it’s possible that the last failure in the logs isn’t the root cause. There are test retries in place for most unit tests so that a test failure gets retried once. It’s possible to disable the custom test retry solution for TestNG tests by passing
-DtestRetryCount=0
.There are quite a few different failures in ReplicatorTest class. I have also reported #9457 and #9458 . It probably makes sense to handle those at the same time.
I also noticed that there are also failures in this test class for
activeBrokerParse
. This test runs before thetestResetCursorNotFail
test. It seems like an additional test failure, but I haven’t been able to figure that out yet. I’ll look into both. Seems like some of the failures @lhotari references above have a failure fromactiveBrokerParse
.