Causes Random Broken Pipe Exception

The following functions are for creating a service account and get policy and update that policy and it gets called one after the other.

def create_service_account(PROJECT_ID, name, displayName, SERVICE):
         SERVICE = googleapiclient.discovery.build('iam', 'v1', cache_discovery=False)
         my_service_account = SERVICE.projects().serviceAccounts().create(
			name='projects/' + PROJECT_ID,
			body={
				'accountId': name,
				'serviceAccount': {
					'displayName': displayName
				}
			}).execute()

def get_policy(project_id, role, member):
        SERVICE1 = googleapiclient.discovery.build("cloudresourcemanager", "v1",cache_discovery=False)

        policy = (
		SERVICE1.projects()
			.getIamPolicy(
				resource=project_id,
				body={"options": {"requestedPolicyVersion": 1}},
			)
			.execute()
		)
		binding = {"role": role, "members": [member]}
		policy["bindings"].append(binding)
  
		result_policy = (
			SERVICE1.projects()
			.setIamPolicy(resource=project_id, body={"policy": policy})
			.execute()
		)

These APIs are giving random BrokenPipeError([Errno 32] Broken pipe).

  body={"options": {"requestedPolicyVersion": 1}},
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/googleapiclient/http.py", line 901, in execute
    headers=self.headers,
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/googleapiclient/http.py", line 204, in _retry_request
    raise exception
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google_auth_httplib2.py", line 198, in request
    uri, method, body=body, headers=request_headers, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/httplib2/__init__.py", line 1558, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 993, in send
    self.sock.sendall(data)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1034, in sendall
    v = self.send(byte_view[count:])
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1003, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe 

Tried updating the versions but still facing the same issue,

google-api-core==1.28.0
google-api-python-client==2.7.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.4
google-cloud-bigquery==2.18.0
google-cloud-core==1.6.0
google-cloud-datacatalog==3.2.0
google-cloud-datastore==2.1.3
google-cloud-storage==1.38.0
google-cloud-secret-manager==2.4.0
google-resumable-media==1.3.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3

[anitakar]

I believe many Google Cloud services may use the same connections for a long time. Increasing number of retries by default should help with connections being killed because they have been inactive for a long time. I am just preparing a PR.

https://github.com/apache/airflow/pull/16376

[busunkim96]

Hi folks,

There is now a google-cloud-* client library with proper support for Cloud Resource Manager. Please see https://github.com/googleapis/python-resource-manager/tree/main/google/cloud/resourcemanager_v3

As for broken pipe exceptions / socket errors, I believe those are unfortunately due to the underlying dependencies of this library, httplib2. https://github.com/googleapis/google-api-python-client/issues/1118 tracks plans to move away from httplib2 and google-auth-httplib2.