question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Duplicate messages exactly every 15 minutes

See original GitHub issue

Environment details

  • OS: Kubernetes Engine
  • Node.js version: 6.3.0
  • npm version: –
  • @google-cloud/pubsub version: 0.16.2

Steps to reproduce

This is most likely related to the discussion in https://github.com/googleapis/nodejs-pubsub/issues/2#issuecomment-356423284 (and following comments in that thread). However since the discussion there somewhat faded out I want to report my findings in a new issue.

My subscriber is consuming messages at a rate of roughly 500/s and it is receiving small batches of duplicate messages exactly every 15 minutes. Those batches typically contain between 100 and 400 duplicate messages. Here’s a plot of the number of duplicates over time:

batches

Most of the duplicates are being delivered to my subscriber within less then a second. Here’s a histogram of the durations between redeliveries in milliseconds:

redelivery

As you can see, the batches of duplicates coincide with spikes in Stackdriver graphs on StreamingPull Operations and StreamingPull Acknowledge Requests (please note that Stackdriver shows Berlin time while the above graph shows UTC, hence 1h time difference):

stackdriver

From the comments in the other thread I did not really understand whether the behavior we see is actually expected. What’s the reason for this to happen precisely every 15 minutes?

Even though the absolute number of duplicates is well below 1%, this still looks pretty odd, unexpected and unnecessary. I’d love to understand better what’s causing this issue and how it could potentially be fixed.

/cc @kir-titievsky @callmehiphop @rossj

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:11 (9 by maintainers)

github_iconTop GitHub Comments

3reactions
callmehiphopcommented, Feb 13, 2018

@rossj I agree with you, however the client was built with specifications from the PubSub team and this particular issue has been brought up several times before. I think we want input from the PubSub team on what the correct solution to this issue actually is.

1reaction
callmehiphopcommented, Apr 13, 2018

@ctavan glad to hear it, thanks for all your help!

Read more comments on GitHub >

github_iconTop Results From Across the Web

If you are receiving duplicate copies of email messages - IU KB
If you are receiving duplicate messages from only a single mailing list, you might be subscribed to that list more than once. Request...
Read more >
iPhone: Duplicate Text Message Notifications Fix - Technipages
Select “Messages“; Slide “iMessages” to off for about 3 minutes, then turn it back on. Your iPhone will be re-registered and may clear...
Read more >
Duplicate sent messages appear in Sent folder when sending ...
The messages then appear every 15 minutes in the sent folder of my hotmail (Outlook) account.
Read more >
Repeating / duplicate incoming text messages - Page 2
Doesn't happen all the time, and not with every message. ... each and then the fourth comes after precisely 11 minutes and then...
Read more >
Galaxy Watch LTE duplicate messages | T-Mobile Community
that the duplicate messages arrive, perhaps 15 minutes after disconnect.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found