AWS Glue API calls are hanging when used from the context of an AWS Glue job

I’m trying to read the list of tables from a data catalog in AWS Glue from within a Glue job with the following code:

session = boto3.Session(region_name='us-east-2')
glue = session.client('glue')
tables = glue.get_tables(
    DatabaseName='customer1'
)
print tables

The result is that the job hangs for 15 minutes and fails with the following error message:

botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=‘glue.us-east-2.amazonaws.com’, port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to glue.us-east-2.amazonaws.com timed out. (connect timeout=60)’))

I can use the S3 API just fine so the problem is specific to the Glue API.

I’ve gone through all my security groups and opened up all the ports to traffic from anywhere. I’ve even added self-referencing rules. But this is to no avail.

I can’t figure out what could be causing the connection to be blocked. Is AWS specifically blocking glue requests?

[rogsmith]

+1

[botchniaque]

I had this problem and fixed it by provisioning my dev-endpoint in an internal subnet https://stackoverflow.com/a/51210082/1680826