SSL Certificate verification issue

From @Spikey123 on August 23, 2017 13:5

Hello.

I’ve been using tswast’s method to query BigQuery results and it’s been purring along fine for a while now. Over the last several days, I’ve been getting an SSL certificate verification issue. Below is the full error message and below that is the code I use.

I’ll note that I updated the python client, bigquery library, cloud storage library, and gcloud components. pip install --upgrade google-api-python-client pip install --upgrade google-cloud-bigquery pip install --upgrade google-cloud-storage gcloud components update

Service account permissions should be fine

Update: I’m getting the same exact error on other scripts I have running - like this one that deletes blobs in Google Cloud Storage:

Traceback (most recent call last):
  File "sfyExport/sfyExport.py", line 334, in <module>
    compression="GZIP" if args.gzip else "NONE")
  File "sfyExport/sfyExport.py", line 203, in main
    didDelete = delete_blobs_matching("mwg-bigqueryexport", deleteString)
  File "sfyExport/sfyExport.py", line 50, in delete_blobs_matching
    blobs = list_blobs(bucket_name)
  File "sfyExport/sfyExport.py", line 38, in list_blobs
    bucket = storage_client.get_bucket(bucket_name)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/client.py", line 173, in get_bucket
    bucket.reload(client=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/_helpers.py", line 99, in reload
    _target_object=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

I checked the certs on accounts.google.com and they appeared to be fine. So I’m not sure why the SSL error would be thrown.

Thanks!

Traceback (most recent call last):
  File "bqTests.py", line 295, in <module>
    args.num_retries)
  File "bqTests.py", line 254, in main
    failCount = async_query("SELECT Count(*) from [Tests.{0}]".format(bqTest["tempTable"]))
  File "bqTests.py", line 104, in async_query
    query_job.begin()
  File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 380, in begin
    method='POST', path=path, data=self._build_resource())
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

The code is:

87- # [START wait_for_job]
88- def wait_for_job(job):
89-     while True:
90-        job.reload()  # Refreshes the state via a GET request.
91-        if job.state == 'DONE':
92-            if job.error_result:
93-                raise RuntimeError(job.errors)
94-            return
95-        time.sleep(1)
96- # [END wait_for_job]

99- # [START async_query]
100- def async_query(query):
101-    client = bigquery.Client()
102-    query_job = client.run_async_query(str(uuid.uuid4()), query)
103-    query_job.use_legacy_sql = True
104-    query_job.begin()
105-
106-    wait_for_job(query_job)
107-
108-    rows = query_job.results().fetch_data(max_results=10)
109-    
110-    for row in rows:
111-        return row[0]
112-    
113- # [END async_query]

Copied from original issue: GoogleCloudPlatform/google-cloud-python#3857

[theacodes]

There are a lot of things that can lead to this and none of them are the fault of this library. I would check, in no particular order:

1. Your system clock.
2. Your Python version. Older versions of 2.7 lack SNI support.
3. Your version of openssl, older versions can lack SNI support.
4. Your version of requests and certifi.

You can try installing requests[security] to see if it automatically fixes it.

[thomasw]

https://github.com/requests/requests/issues/3859#issuecomment-311275977 is likely the solution to this issue if the following fails for you.

import requests
requests.get('https://google.com/')

Upgrading certifi and requests alone isn’t going to solve the issue because it’s being caused by a buggy openssl installation.