Intermittent 'PermissionError: Access Denied' when trying to read S3 file from AWS Lambda

I’m running a Python 3.7 script in AWS Lambda, which runs queries against AWS Athena and tries to download the CSV results file that Athena stores on S3 once the query execution has completed.

Any ideas why I’d be running into the error below intermittently?

s3_query_result_path = f's3://{bucket}/{results_key}'

[ERROR] PermissionError: Access Denied
Traceback (most recent call last):
  File "/var/task/lambdas/process_file/lambda_function.py", line 91, in lambda_handler
    df = pd.read_csv(s3_query_result_path)
  File "/var/task/pandas/io/parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/var/task/pandas/io/parsers.py", line 440, in _read
    filepath_or_buffer, encoding, compression
  File "/var/task/pandas/io/common.py", line 207, in get_filepath_or_buffer
    filepath_or_buffer, encoding=encoding, compression=compression, mode=mode
  File "/var/task/pandas/io/s3.py", line 36, in get_filepath_or_buffer
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  File "/var/task/fsspec/spec.py", line 669, in open
    autocommit=ac, **kwargs)
  File "/var/task/s3fs/core.py", line 303, in _open
    autocommit=autocommit)
  File "/var/task/s3fs/core.py", line 920, in __init__
    cache_type=cache_type)
  File "/var/task/fsspec/spec.py", line 864, in __init__
    self.details = fs.info(path)
  File "/var/task/s3fs/core.py", line 479, in info
    return super().info(path)
  File "/var/task/fsspec/spec.py", line 477, in info
    out = self.ls(self._parent(path), detail=True, **kwargs)
  File "/var/task/s3fs/core.py", line 497, in ls
    files = self._ls(path, refresh=refresh)
  File "/var/task/s3fs/core.py", line 430, in _ls
    return self._lsdir(path, refresh)
  File "/var/task/s3fs/core.py", line 336, in _lsdir
    raise translate_boto_error(e)

As you can see above, I’m using the pandas library, which then uses s3fs under the hood.

The Lambda works about 80% of the time, and I can’t figure out anything unique about the times it fails.

Feel free to let me know if I should be posting this question in pandas or elsewhere instead - thanks for your help!

[briandegenhart]

We came across a similar problem beginning with s3fs==0.3.0. It was a FileNotFoundError message, however, but also unpredictably intermittent (approximately 15-20% of the time) and within a lambda that was being triggered by the presence of a file in an S3 bucket. The issue occurred with files as small as 18kb. Unfortunately, I can’t reproduce it outside of our production environment, where we’ve stuck with s3fs==0.2.2. This is the stack trace from last time we had the issue, which was using s3fs==0.3.1:

Traceback (most recent call last):
  File /var/task/src/filemetadata.py, line 29, in _read_file_data_definition
    2) as reader:
  File /var/task/src/StreamReader.py, line 29, in __enter__
    self.__opened_file = self.__fs.open(self.__bucket_url, 'r')
  File /var/task/fsspec/spec.py, line 666, in open
    return io.TextIOWrapper(self.open(path, mode, block_size, **kwargs))
  File /var/task/fsspec/spec.py, line 670, in open
    autocommit=ac, **kwargs)
  File /var/task/s3fs/core.py, line 302, in _open
    s3_additional_kwargs=kw, cache_type=cache_type)
  File /var/task/s3fs/core.py, line 904, in __init__
    cache_type=cache_type)
  File /var/task/fsspec/spec.py, line 865, in __init__
    self.details = fs.info(path)
  File /var/task/s3fs/core.py, line 469, in info
      return super().info(path)
  File /var/task/fsspec/spec.py, line 482, in info
      raise FileNotFoundError(path)
FileNotFoundError: adobe-campaign-hub-input/directmail/NAE_CHQ_146501861_245774912_2019-07-22_121116.txt

[birdsarah]

Also, just to restate the obvious, this problem doesn’t occur with s3fs 0.2.0 (which is what I’m using now) so whatever you were doing before was working just fine.