Pub/sub: '504 Deadline Exceeded' response from 'SubscriberClient.pull'

google-cloud-pubsub 1.0.0 return 504 Deadline Exceeded.

When I downgrade version to 0.45.0, then the error does not appear.

Environment details

Python 3.7.2

google-cloud-pubsub version 1.0.0

pubsub subscription

 $ gcloud pubsub subscriptions describe projects/xxx/subscriptions/yyy
ackDeadlineSeconds: 20
expirationPolicy:
  ttl: 26784000s
messageRetentionDuration: 604800s
name: projects/xxx/subscriptions/yyy
pushConfig: {}
retainAckedMessages: true
topic: projects/xxx/topics/zzz

follow packages are installed together.

$ pip list | grep google-cloud
google-cloud-bigquery     1.17.0     
google-cloud-container    0.3.0      
google-cloud-core         1.0.2      
google-cloud-logging      1.11.0     
google-cloud-pubsub       1.0.0      
google-cloud-storage      1.17.0     

Steps to reproduce

Run follow script.

from google.cloud import pubsub

subscriber_client = pubsub.SubscriberClient()
subscription_path = subscriber_client.subscription_path("xxx","yyy")
pull_response = subscriber_client.pull(subscription=subscription_path, max_messages=1)

Stack trace

Traceback (most recent call last):
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1570074717.534613000","description":"Error received from peer ipv4:172.217.31.138:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "pubsub_bug.py", line 5, in <module>
    pull_response = subscriber_client.pull(subscription=subscription_path, max_messages=1)
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/cloud/pubsub_v1/_gapic.py", line 40, in <lambda>
    fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw)  # noqa
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 1005, in pull
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/bbbb/venv/aaaa/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

[HenleyKuang]

I suspect this is related to 31e96ae decreasing the value of initial_rpc_timeout_millis from 25000 to 5000. If I change it back to 25000, I no longer see 504 Deadline Exceeded.

Here’s the code snippet that was triggering 504 errors:

client = pubsub_v1.SubscriberClient()
sub_path = client.subscription_path(project_name, sub_name)
while True:
  response = client.pull(subscription=sub_path, max_messages=1, return_immediately=False)
  for message in response.received_messages:
    ProcessMessage(message.message)
  client.acknowledge(sub_path, [m.ack_id for m in response.received_messages])

Adding client.pull(..., timeout=30.0) also seems to fix the problem.

Yeah, I’m still getting this error when my sub has no messages. Anything under 20 seconds would raise DeadlineExceeded

[pmarks-net]

I suspect this is related to https://github.com/googleapis/google-cloud-python/commit/31e96ae9411b79c21260518a379a593657efe5b3 decreasing the value of initial_rpc_timeout_millis from 25000 to 5000. If I change it back to 25000, I no longer see 504 Deadline Exceeded.

Here’s the code snippet that was triggering 504 errors:

client = pubsub_v1.SubscriberClient()
sub_path = client.subscription_path(project_name, sub_name)
while True:
  response = client.pull(subscription=sub_path, max_messages=1, return_immediately=False)
  for message in response.received_messages:
    ProcessMessage(message.message)
  client.acknowledge(sub_path, [m.ack_id for m in response.received_messages])

Adding client.pull(..., timeout=30.0) also seems to fix the problem.