Retries on push notifications

PushNotificationListener could use its own notification queue to schedule retries.

See discussion at: https://groups.google.com/d/msgid/killbilling-users/CAJE4nfvdYty4SYyCfe1m94uoJWqaZwLGPCJwAAwCsB4Ht-o50A%40mail.gmail.com

[pierre]

Do you think we can ask for a status = 404 here?

I think we should keep the logic as is: if the destination app is down (e.g. connect timeout), or unavailable (e.g. returns 500) or misconfigured (e.g. returns 301), we should assume the notification wasn’t received and we should retry later (i.e. send it again).

Does “retry” means insert this event into notification queue again?

We should create a new queue, independent of the queue of the original event, responsible for these events. The queue would be similar to the one handling payment retries and do some sort of exponential delay retries over the span of, say, a few days (configurable).

I would also suggest we always insert in this new queue (effective date == now()) and attempt to send in the notification queue callback, to avoid blocking the external bus (since the HTTP POST can take some time). @sbrossie thoughts?

Does it make sense?

[matias-aguero-hs]

@pierre I created a PR https://github.com/killbill/killbill/pull/589 with this feature implemented.