Duplicate messages exactly every 15 minutes

Environment details

• OS: Kubernetes Engine
• Node.js version: 6.3.0
• npm version: –
• @google-cloud/pubsub version: 0.16.2

Steps to reproduce

This is most likely related to the discussion in https://github.com/googleapis/nodejs-pubsub/issues/2#issuecomment-356423284 (and following comments in that thread). However since the discussion there somewhat faded out I want to report my findings in a new issue.

My subscriber is consuming messages at a rate of roughly 500/s and it is receiving small batches of duplicate messages exactly every 15 minutes. Those batches typically contain between 100 and 400 duplicate messages. Here’s a plot of the number of duplicates over time:

Most of the duplicates are being delivered to my subscriber within less then a second. Here’s a histogram of the durations between redeliveries in milliseconds:

As you can see, the batches of duplicates coincide with spikes in Stackdriver graphs on StreamingPull Operations and StreamingPull Acknowledge Requests (please note that Stackdriver shows Berlin time while the above graph shows UTC, hence 1h time difference):

From the comments in the other thread I did not really understand whether the behavior we see is actually expected. What’s the reason for this to happen precisely every 15 minutes?

Even though the absolute number of duplicates is well below 1%, this still looks pretty odd, unexpected and unnecessary. I’d love to understand better what’s causing this issue and how it could potentially be fixed.

/cc @kir-titievsky @callmehiphop @rossj

[callmehiphop]

@rossj I agree with you, however the client was built with specifications from the PubSub team and this particular issue has been brought up several times before. I think we want input from the PubSub team on what the correct solution to this issue actually is.

[callmehiphop]

@ctavan glad to hear it, thanks for all your help!