Boto3 incompatible with python zip import

One of Python’s useful features is its ability to load modules from a .zip archive (PEP here), allowing you to package up multiple dependencies into a single file.

Boto breaks when trying to import it from a .zip, throwing:

  File "C:\code sandbox\boto.zip\boto3\session.py", line 263, in client
  File "C:\code sandbox\boto.zip\botocore\session.py", line 799, in create_client
  File "C:\code sandbox\boto.zip\botocore\session.py", line 668, in _get_internal_component
  File "C:\code sandbox\boto.zip\botocore\session.py", line 870, in get_component
  File "C:\code sandbox\boto.zip\botocore\session.py", line 150, in create_default_resolver
  File "C:\code sandbox\boto.zip\botocore\loaders.py", line 132, in _wrapper
  File "C:\code sandbox\boto.zip\botocore\loaders.py", line 424, in load_data
botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints

How to Reproduce:

1. Create a .zip containing boto3 and botocore
2. Create a .py file in the same directory as the zip (access keys removed for obvious reasons):

sys.path.insert(0, 'boto.zip')
import boto3

s3 = boto3.client('s3', aws_access_key_id='access_key', aws_secret_access_key='secret_key')

1. Run

Tested on Python 3.6.7 boto3 1.9.39 botocore 1.12.39

[LukeBolly]

What are the odds of getting this implemented? Its preventing us from distributing boto3, which makes it very hard to provide a package that depends on it in PySpark.

[alete89]

Found a work around for this. You can pass spark conf args to have spark unzip the dependencies and include in path, something like this,

	--conf spark.yarn.dist.archives=s3://<bucket+path>/sparkapp.zip#deps" \
	--conf spark.yarn.appMasterEnv.PYTHONPATH=deps" \
	--conf spark.executorEnv.PYTHONPATH=deps" \

Worked with EMR 6.2.0 and Python 3.7.9

Hey @dsonavane-rgare I’m trying this without success. Can you elaborate a bit more? This is how I was sending my file and deps (this throws boto3 not found because one of my zipped files uses boto3):

spark-submit --py-files s3://<bucket>/code/spark/dependencies.zip s3://<bucket>/code/spark/job.py args

This is what I’ve tried now, based on your example:

spark-submit --conf spark.yarn.dist.archives=s3://<bucket>/code/spark/dependencies.zip#deps --conf spark.yarn.appMasterEnv.PYTHONPATH=deps --conf spark.executorEnv.PYTHONPATH=deps s3://<bucket>/code/spark/job.py args

and this as well:

spark-submit --py-files s3://<bucket>/code/spark/dependencies.zip --conf spark.yarn.dist.archives=s3://<bucket>/code/spark/dependencies.zip#deps --conf spark.yarn.appMasterEnv.PYTHONPATH=deps --conf spark.executorEnv.PYTHONPATH=deps s3://<bucket>/code/spark/job.py 2021-12-01

Thanks