InvalidStateStoreException when REPLACE_THREAD is used in StreamsUncaughtExceptionHandler

@en-vee commented on Sun Sep 19 2021

Describe the issue I have used the following dependencies in my Spring Cloud Stream Kafka Streams application so as to be able to use Apache Kafka 2.8.0 clients. The reason why I have done this is so as to use the new StreamsUncaughtExceptionHandler feature which is part of Kafka 2.8.0 streams library.

ext {
	set('kafkaVersion',"2.8.0")
	set('springKafkaVersion',"2.7.6")
	set('springCloudVersion', "2020.0.0")
}

dependencies {
	implementation project(':common')
	implementation 'org.springframework.boot:spring-boot-starter-web'
	implementation 'org.springframework.boot:spring-boot-starter-actuator'
	implementation 'org.springframework.boot:spring-boot-starter-test'
	implementation 'io.micrometer:micrometer-registry-prometheus'
	
	
	implementation 'org.springframework.cloud:spring-cloud-stream'
	implementation 'org.springframework.cloud:spring-cloud-stream-binder-kafka'
	implementation 'org.springframework.cloud:spring-cloud-stream-binder-kafka-streams'
	//implementation 'org.springframework.kafka:spring-kafka'
	
	implementation "org.springframework.kafka:spring-kafka:${springKafkaVersion}"

    implementation "org.apache.kafka:kafka-clients:${kafkaVersion}"
    implementation "org.apache.kafka:kafka-streams:${kafkaVersion}" // optional - only needed when using kafka-streams
    testImplementation "org.springframework.kafka:spring-kafka-test:${springKafkaVersion}"
    testImplementation "org.apache.kafka:kafka-clients:${kafkaVersion}:test"
    testImplementation "org.apache.kafka:kafka_2.13:${kafkaVersion}"
    testImplementation "org.apache.kafka:kafka_2.13:${kafkaVersion}:test"
    
    testImplementation 'org.junit.jupiter:junit-jupiter-api'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
    testCompileOnly 'junit:junit:4.12' 
	testRuntimeOnly 'org.junit.vintage:junit-vintage-engine'
}

My StreamsUncaughtExceptionHandler code is as below in my streams customizer bean :

			factoryBean.setKafkaStreamsCustomizer(new KafkaStreamsCustomizer() {
				
				@Override
				public void customize(KafkaStreams kafkaStreams) {
					// TODO Auto-generated method stub
					LOG.debug("Customizing Kafka Streams");
					kafkaStreams.setUncaughtExceptionHandler(new StreamsUncaughtExceptionHandler() {
						
						@Override
						public StreamThreadExceptionResponse handle(Throwable exception) {
							// Only replace the thread if this is an InvalidPidMappingException				
							if(exception.getCause() instanceof InvalidPidMappingException) {
								LOG.info("InvalidPidMappingException encountered. Thread will be replaced.");
								return StreamThreadExceptionResponse.REPLACE_THREAD;
							}
							return StreamThreadExceptionResponse.SHUTDOWN_APPLICATION;
						}
					});
				}
			});

The reason for doing this is to avoid the Streams application from shutting down when an InvalidPidMappingException is thrown.

While this helps in that it spawns a new Thread and also continues processing the event, the InteractiveQueryService does not work after this exception InvalidPidMappingException has been raised. (I have a REST controller which accepts interactive queries via HTTP to query the State Store). The exception I get after trying to query the State Store via my REST controller is InvalidStateStoreException.

To Reproduce Steps to reproduce the behavior:

1. Set the following parameters to small values on the Kafka brokers and restart them: transactional.id.expiration.ms transaction.max.timeout.ms

2. Start a Kafka Streams Client application which also exposes access to a Local State Store via a REST endpoint. The application should also have a StreamsUncaughtExceptionHandler implemented with a policy of REPLACE_THREAD.

3. Make sure no Messages are sent to the Topic on which the streams application is consuming messages for at least the value set in transactional.id.expiration.ms

4. Then send a message to the topic.

5. This will result in InvalidPidMappingException and will also cause the thread to be replaced and the message will also get eventually processed by the Streams application.

6. Now try to access the State Store via the REST endpoint. This will cause the InvalidStateStoreException.

Version of the framework Spring Cloud Stream 3.1.0 (Kafka Binder and Kafka Streams Binder) Sprint Boot 2.4.2 Apache Kafka Broker version 2.7+ Apache Kafka Streams,Clients 2.8.0

Expected behavior State Store is accessible via InteractiveQueryService

Screenshots None

Additional context None

---

@en-vee commented on Wed Sep 22 2021

Hi All, Any inputs on this one ? Is there a way to (re)gain access to the state stores via the interactive query service ?

[en-vee]

@sobychacko I am unable to close this issue. can you assist please ? thanks.

[en-vee]

@sobychacko I will close this issue then. Thanks again for your help.