Make the shutdown timeout configurable

Hi,

I am not entirely sure whether I would classify this as a bug or as a missing feature. Summarized: It should be possible to make the shutdown timeout in DefaultConfigurer (which is five seconds) configurable, as it is possible that some components take longer to stop.

Please consider the following demo application which uses Axon’s Spring Boot Starter.

@SpringBootApplication
public class DemoApplication {

	public static void main( final String[] args ) {
		final var applicationContext = new SpringApplicationBuilder( DemoApplication.class )
				.properties( "axon.axonserver.enabled=false" )
				.run( args );

		final var eventGateway = applicationContext.getBean( EventGateway.class );
		eventGateway.publish( 0 );

		await( ).until( ( ) -> MyEventHandler.elementProcessed.get( ) );

		applicationContext.close( );
	}

	@Autowired
	private void configure( final EventProcessingConfigurer configurer ) {
		configurer.registerTrackingEventProcessorConfiguration( MyEventHandler.PROCESSING_GROUP, c -> TrackingEventProcessorConfiguration
				.forParallelProcessing( 4 )
				.andEventAvailabilityTimeout( 20, TimeUnit.SECONDS ) );
	}

	@Component
	@ProcessingGroup( MyEventHandler.PROCESSING_GROUP )
	static class MyEventHandler {

		public static final String PROCESSING_GROUP = "MyEventHandler";

		private static AtomicBoolean elementProcessed = new AtomicBoolean( );

		@EventHandler
		public void on( final Integer i ) {
			elementProcessed.set( true );
		}

	}

}

In this demo application, we have a tracking event processor with four segments and an event awailability timeout of 20 seconds. We publish a single “event” for this handler (the zero) and wait until the event has been processed (we use Awaitility to wait for this to happen). Now we close the application context and - by doing so - shut the Axon infrastructure down. What we get is the following:

2021-10-19 13:14:32.802  INFO 22484 --- [           main] o.a.e.TrackingEventProcessor             : Shutdown state set for Processor 'MyEventHandler'.
2021-10-19 13:14:32.803  INFO 22484 --- [nPool-worker-19] o.a.e.TrackingEventProcessor             : Processor 'MyEventHandler' awaiting termination...
2021-10-19 13:14:37.812  WARN 22484 --- [           main] o.a.config.DefaultConfigurer             : Timed out during shutdown phase [1073741823] after 5 seconds. Proceeding to following phase

Notice that the DefaultConfigurer gives up after five seconds and simply continues. However, the threads of the TrackingEventProcessor are still running. This leads to a lot of exceptions after approximately 20 seconds (which is exactly the configured event awailability timeout). We start and stop application contexts very often in our integration tests. This behaviour is problematic for multiple reasons:

1. Some processors are still active once the framework executes the following tests. I am not entirely sure whether this can lead to weird and flaky test behaviour or not.
2. The log is full of error messages which are not related to the actual test. This makes it very difficult for us to see whether the test has actual issues.
3. Stopping and starting the event processor during tests (e.g. to test the reset behaviour) takes a long time. This might rather be our own fault, because the API clearly states: “Note that some storage engines for the EmbeddedEventStore do not support streaming. They may […] wait for the timeout to occur.”

I have two ideas how this might be solved:

• As originally stated: Make the shutdown timeout configurable. We could set it to safe 25 seconds and be done with it.
• Interrupt the tracking event processor during the shutdown and force him out of the processing loop.

Current Behaviour

The DefaultConfigurer waits exactly five seconds for all components to handle the current lifecycle phase.

Wanted Behaviour

Provide a possibility to configure the timeout duration for each lifecycle phase.

Workarounds

One could use a custom SmartLifecycle which runs before Axon and stops all event handlers asynchronously with a longer timeout.

	@Bean
	SmartLifecycle eventProcessorStoppingSmartLifecycle( @Autowired final EventProcessingConfiguration eventProcessingConfiguration ) {
		return new SmartLifecycle( ) {

			private boolean running;

			@Override
			public void stop( ) {
				try {
					eventProcessingConfiguration.eventProcessors( )
							.values( )
							.stream( )
							.map( EventProcessor::shutdownAsync )
							.reduce( ( cf1, cf2 ) -> CompletableFuture.allOf( cf1, cf2 ) )
							.orElse( CompletableFuture.completedFuture( null ) )
							.get( 25, TimeUnit.SECONDS );
				} catch ( InterruptedException | ExecutionException | TimeoutException e ) {
					// Do something smart with the exception
				}
				running = false;
			}

			@Override
			public void start( ) {
				running = true;
			}

			@Override
			public boolean isRunning( ) {
				return running;
			}
		};
	}

Thank you and best regards,

Nils

[smcvb]

Just notifying you here @OLibutzki because I’ve released 4.5.13 about two hours ago. 😃

[smcvb]

Hi @OLibutzki!

I have found sometime to move #2037 and #2041 to axon-4.5.x. This means they’ll be part of Axon Framework release 4.5.13. Furthermore, I hope to release 4.5.13 this week. Hence you’ll be able to use these changes (hopefully) very soon. 😃