Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pubsub 0.35.3 stop consuming messages - ConsumeBidirectionalStream exited RPC

See original GitHub issue

After an ~hour, the rpc connection is closed:

[2018-06-06 11:02:36,258 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,259 - INFO - Thread-ConsumeBidirectionalStream - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - **ERROR - Thread-ConsumeBidirectionalStream - bidi] The bidirectional RPC unexpectedly exited. <google.api_core.grpc_helpers._StreamingResponseIterator object at 0x7fb758946090>**

I need to cancel and start the subscriber again.

OS type and version - “Debian GNU/Linux 8 (jessie)” Version - Python 2.7.9 4. pip freeze | grep pubsub google-cloud-pubsub==0.35.3

Code example:

        def subscriber_handler(message):
            path = message.attributes.get("type")
            self.handler_function(path=path, body=message.data)  
            message.ack()
        self.listening_subscriber = self.subscriber.subscribe(subscription_path, callback=subscriber_handler, flow_control=flow_control)

All the other info is the same as in https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5325.

Issue Analytics

State:
Created 5 years ago
Comments:16 (11 by maintainers)

Top GitHub Comments

2reactions

seanmt13commented, Jun 26, 2018

I upgraded to 35.4 and i am still having the issue where my subscriber stops pulling from a subscription. My environment is in kubernetes and should be pulling 40-50 messages a second. Looking at the logs i get the following:

Jun 26, 2018, 12:58:14 PM | INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting. INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting. INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

After this it seems to go into an endless loop of:

INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Re-established stream INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

Every ten minutes after that it re-establishes the stream then immediately gets the 504 error without pulling any more messages from the subscription. Any ideas?

1reaction

theacodescommented, Jun 6, 2018

0.35.4 has been released which resolves this as far as I’m able to test. Please let me know if you’re able to reproduce this with that version and we’ll continue digging.

Top Results From Across the Web

Error Codes | Cloud Pub/Sub Documentation

Error HTTP Code Description UNAUTHENTICATED 401 The client is not authenticated properly. UNAVAILABLE 503 The service was unable to process a request.