AggregationSpout does not release IsInQuery boolean sometimes

First remark: I haven’t identified the cause of this issue, but I think it could still be good for others to be aware of it, therefore I’m opening this issue.

Sometimes my AggregationSpout gets stuck doing nothing because the isInQuery boolean is never reset to false. I suppose this can happen when somehow the onSuccess() and onFailure() methods are not called. Currently I’m considering adding in functionality to reset the isInQuery boolean after a certain amount of time.

Maybe others have encountered this issue as well and have a nicer solution, that would be great to know. I will update this ticket in case I find out more.

[thijswesterveld]

Hi @jnioche,

We may have found the root cause of this. The issue seems to be with the onFailure() methods in Hybridspout and AggregationSpout. The onFailure() method in AggregationSpout calls markQueryReceivedNow(), but the one in HybridSpout doesn’t .

We are using a variant of Hybridspout. Failures are always handled by the onFailure() method in that child class, even though the query or response that is causing the failure is from the parent AggregationSpout class.

onResponse() decides between parent/child handling by checking the response properties. Ideally, we’d do something similar in onFailure(), but I don’t think we have the information there to make that decision.

In our variants of the Hybridspout we have solved it by removing the onFailure() override our variant of HybridSpout to make sure we always release isInQuery. This of course introduces the reverse issue of releasing too quickly if the failure comes from a single queue .

[jnioche]

hi @thijswesterveld, I have just pushed a commit which should solve the problem. We will now use a separate listener to handle the results for specific queues and will let the super class deal with onFailure(), which I think is a lot cleaner. Would be great if you could give it a try. Feel free to reopen this issue if necessary. Thanks again for your reporting it in the first place and the detective work to get to the bottom of the problem.