Stackdriver: list descending entries fails after a short moment

1. OS type and version: macOS 10.13.4
2. Python version and virtual environment information: Python 3.6.5, venv using pipenv
3. google-cloud-python version: 0.33.1
4. Stacktrace if available:

Stacktrace

Traceback (most recent call last):
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py", line 121, in inner
    return to_call(*args)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/grpc/_channel.py", line 500, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/grpc/_channel.py", line 434, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Insufficient tokens for quota 'logging.googleapis.com/read_requests' and limit 'ReadRequestsPerMinutePerProject' of service 'logging.googleapis.com' for consumer 'project_number:307713667560'.)>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/Users/iddan/stackdriver-cli/stackdriver/cli.py”, line 11, in <module>
for entry in client.list_entries(order_by=DESCENDING):
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py”, line 186, in _items_iter
for page in self._page_iter(increment=False):
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py”, line 223, in _page_iter
page = self._next_page()
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py”, line 430, in _next_page
items = six.next(self._gax_page_iter)
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/init.py”, line 465, in next
response = self._func(self._request, **self._kwargs)
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/api_callable.py”, line 376, in inner
return a_func(*args, **kwargs)
File “/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py”, line 127, in inner
’ classified as transient’, exception)
google.gax.errors.RetryError: RetryError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Insufficient tokens for quota ‘logging.googleapis.com/read_requests’ and limit ‘ReadRequestsPerMinutePerProject’ of service ‘logging.googleapis.com’ for consumer ‘project_number:307713667560’.)>)

6. Steps to reproduce

• Install google cloud python
• Write script importing stackdriver list_entries() with order_by=DESCENDING
• Run for a short moment until it crashes

1. Code example

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client.from_service_account_json("credentials.json")

for entry in client.list_entries(order_by=DESCENDING):
    print(entry)

[tseaver]

@iddan I’m sorry we weren’t able to work out how to address your usecase better with GCP logging.

[yan-hic]

@tseaver not sure Retry will do so if 403 rateLimitExceeded is hit.

That error is currently not listed as transient API error as per https://google-cloud-python.readthedocs.io/en/latest/core/retry.html?highlight=retry#google.api_core.retry.if_transient_error

Would be good to add but in the meantime, I guess you can pass the error as custom predicate. However the error type is not defined in the library or in http_client (at best it matches 403 FORBIDDEN).

We went down the road of a custom linear retry (sleep 2 sec)