Exponential Retry for Tracking event processor not happening for transient exceptions when using postgres JdbcTokenStore
See original GitHub issueWhenever any tracking processor enters retry mode it should attempt to retry in 1s, 2s, 4s, 8s, 16s and so on till it reaches max interval of 60s and thereafter retrying in every 60s. But this behaviour is not observed when using JdbcTokenStore with postgres
Basic information
- Axon Framework version: 4.5.3
- JDK version: 11
- Complete executable reproducer if available (e.g. GitHub Repo): https://github.com/shubhojitr/axon-retry.git
Note: please use below property to toggle between JPA and JDBC token store in the sample repository provided to replicate this issue
token-store:
jdbc:
enable: true
Steps to reproduce
Axon configurations:
@Bean
public CommandGateway sCSmartPayCommandGateway(org.axonframework.config.Configuration configurer) {
CommandBus commandBus = configurer.commandBus();
commandBus.registerHandlerInterceptor(new LoggingInterceptor<>());
CommandGatewayFactory gatewayFactory = CommandGatewayFactory.builder().commandBus(commandBus).build();
return gatewayFactory.createGateway(CommandGateway.class);
}
@Bean
@ConditionalOnMissingBean
public PropagatingErrorHandler propagatingErrorHandler() {
return PropagatingErrorHandler.instance();
}
@Bean
public TokenStore tokenStore(DataSource dataSource, @Qualifier("serializer") Serializer serializer) {
return JdbcTokenStore.builder()
.connectionProvider(new DataSourceConnectionProvider(dataSource))
.serializer(serializer)
.claimTimeout(Duration.ofSeconds(10))
.schema(tokenSchema())
.contentType(byte[].class)
.nodeId(ManagementFactory.getRuntimeMXBean().getName())
.build();
}
private TokenSchema tokenSchema() {
return TokenSchema.builder().setTokenTable("token_entry_jdbc")
.setProcessorNameColumn("processor_name").setSegmentColumn("segment")
.setTokenColumn("token").setTokenTypeColumn("token_type")
.setTimestampColumn("timestamp").setOwnerColumn("owner").build();
}
Token entry sql :
CREATE TABLE token_entry_jdbc (
processor_name varchar(255) NOT NULL,
segment int4 NOT NULL,
"owner" varchar(255) NULL,
"timestamp" varchar(255) NOT NULL,
"token" bytea NULL,
token_type varchar(255) NULL,
CONSTRAINT token_entry_pkey_jdbc PRIMARY KEY (processor_name, segment)
);
Expected behaviour
Expected behaviour for a tracking processor once it enters error mode, is to continue retrying until the transient exception cause is rectified. Below logs are from JPATokenStore with postgres where the retry is working correctly.
2021-08-25 23:21:36.049 INFO 14012 --- [tryProcessor]-0] com.axon.handler.TestEventHandler : TestEvent Start TestEvent(id=7b5feb66-253a-4fdb-bee6-2775da3ba7ae, message=ba ba black sheep..!!!) : map :{}
2021-08-25 23:21:36.353 WARN 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Error occurred. Starting retry mode.
com.axon.handler.exception.TestTransientException: hikori dicori dock..!!
at com.axon.handler.TestEventHandler.on(TestEventHandler.java:26) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:144) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.annotation.AnnotatedHandlerInspector$NoMoreInterceptors.handle(AnnotatedHandlerInspector.java:372) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AnnotationEventHandlerAdapter.handle(AnnotationEventHandlerAdapter.java:94) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.SimpleEventHandlerInvoker.handle(SimpleEventHandlerInvoker.java:112) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.MultiEventHandlerInvoker.handle(MultiEventHandlerInvoker.java:89) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$1(AbstractEventProcessor.java:165) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:65) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.interceptors.LoggingInterceptor.handle(LoggingInterceptor.java:83) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.lambda$new$1(TrackingEventProcessor.java:185) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.lambda$processInUnitOfWork$2(AbstractEventProcessor.java:173) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:86) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.processInUnitOfWork(AbstractEventProcessor.java:159) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:451) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:294) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1005) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1149) ~[axon-messaging-4.5.3.jar:4.5.3]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2021-08-25 23:21:36.353 WARN 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Releasing claim on token and preparing for retry in 1s
2021-08-25 23:21:36.770 INFO 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Released claim
2021-08-25 23:21:41.871 INFO 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Fetched token: IndexTrackingToken{globalIndex=1} for segment: Segment[0/0]
2021-08-25 23:21:43.507 INFO 14012 --- [tryProcessor]-0] com.axon.handler.TestEventHandler : TestEvent Start TestEvent(id=7b5feb66-253a-4fdb-bee6-2775da3ba7ae, message=ba ba black sheep..!!!) : map :{7b5feb66-253a-4fdb-bee6-2775da3ba7ae=1}
2021-08-25 23:21:43.815 WARN 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Releasing claim on token and preparing for retry in 2s
2021-08-25 23:21:44.329 INFO 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Released claim
2021-08-25 23:21:50.266 INFO 14012 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Fetched token: IndexTrackingToken{globalIndex=1} for segment: Segment[0/0]
2021-08-25 23:21:51.802 INFO 14012 --- [tryProcessor]-0] com.axon.handler.TestEventHandler : TestEvent Start TestEvent(id=7b5feb66-253a-4fdb-bee6-2775da3ba7ae, message=ba ba black sheep..!!!) : map :{7b5feb66-253a-4fdb-bee6-2775da3ba7ae=2}
2021-08-25 23:21:51.803 INFO 14012 --- [tryProcessor]-0] com.axon.handler.TestEventHandler : TestEvent End map: {}
Actual behaviour
Once the Tracking event processor enters retry mode, its gets stuck after printing Releasing claim on token and preparing for retry in 1s
2021-08-25 23:19:27.901 INFO 22704 --- [tryProcessor]-0] com.axon.handler.TestEventHandler : TestEvent Start TestEvent(id=3e22d5ae-2fd4-441b-b408-d8f3090a64ab, message=ba ba black sheep..!!!) : map :{}
2021-08-25 23:19:27.934 WARN 22704 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Error occurred. Starting retry mode.
com.axon.handler.exception.TestTransientException: hikori dicori dock..!!
at com.axon.handler.TestEventHandler.on(TestEventHandler.java:26) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:144) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.annotation.AnnotatedHandlerInspector$NoMoreInterceptors.handle(AnnotatedHandlerInspector.java:372) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AnnotationEventHandlerAdapter.handle(AnnotationEventHandlerAdapter.java:94) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.SimpleEventHandlerInvoker.handle(SimpleEventHandlerInvoker.java:112) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.MultiEventHandlerInvoker.handle(MultiEventHandlerInvoker.java:89) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$1(AbstractEventProcessor.java:165) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:65) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.interceptors.LoggingInterceptor.handle(LoggingInterceptor.java:83) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.lambda$new$1(TrackingEventProcessor.java:185) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.lambda$processInUnitOfWork$2(AbstractEventProcessor.java:173) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:86) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.AbstractEventProcessor.processInUnitOfWork(AbstractEventProcessor.java:159) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:451) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:294) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1005) ~[axon-messaging-4.5.3.jar:4.5.3]
at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1149) ~[axon-messaging-4.5.3.jar:4.5.3]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2021-08-25 23:19:27.934 WARN 22704 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Releasing claim on token and preparing for retry in 1s
2021-08-25 23:19:28.146 INFO 22704 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Released claim
2021-08-25 23:19:30.068 INFO 22704 --- [tryProcessor]-0] o.a.e.TrackingEventProcessor : Fetched token: IndexTrackingToken{globalIndex=3} for segment: Segment[0/0]
Issue Analytics
- State:
- Created 2 years ago
- Comments:9 (7 by maintainers)
Top Results From Across the Web
Exponential Retry for Tracking event processor not happening for ...
Exponential Retry for Tracking event processor not happening for transient exceptions when using postgres JdbcTokenStore.
Read more >Release Axon Framework v4.6.0 - AxonIQ Discuss
... Exponential Retry for Tracking event processor not happening for transient exceptions when using postgres JdbcTokenStore #1920 ...
Read more >axon - Retry Execption originating from Event Handlers
I am trying to configure an ErrorHandler for a TrackingEventProcessor so that any exception in my @ ...
Read more >Saga Tracking Event Processor doesn't retry event processing ...
Subj. While non-saga tracking event processors working as before, releasing claim and retrying with backoff schema. Am I missing some of Saga configuration?...
Read more >AxonFramework Versions - Open Source Agenda
... Fix typos #2016; Exponential Retry for Tracking event processor not happening for transient exceptions when using postgres JdbcTokenStore #1920 ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I just had a call with the customer. For future people experiencing the issue: Injecting the
DataSource
as aBean
into theJdbcTokenStore
creates the issue. Use theDataSource
provided by theDataSourceProvider
of Axon for it, then it will use the same transaction manager.Hi @shubhojitr , I’ve tried looking into this with the source code you have provided. The source code did not work out of the box for me due to the resources folder being in the wrong place. But once I got it working, I did get an exponential retry both on the JPA implementation and on the JDBC implementation.
This is of the JDBC implementation:
I did create the token table in a little bit of a different way than you did, perhaps that makes the difference? I used the
PostgresTokenTableFactory
to create it, as can be seen below in the bean definition:Does it work with this configuration for you?