Intermittent / transient fiona.open errors using AWS S3 + lambda

Expected behavior and actual behavior.

I’m using fiona==1.8.6 on AWS lambda to open shapefiles on S3. I’m seeing occasional clusters of fiona.open errors and was wondering (1) whether anyone else has seen similar behavior, and (2) whether there is anything I can do to reduce the frequency of these errors.

The stack traces look like this:

my_shapefile = fiona.open(shapefile_uri)
File "/var/task/fiona/__init__.py", line 249, in open
enabled_drivers=enabled_drivers, **kwargs)
File "/var/task/fiona/collection.py", line 160, in __init__
self.session.start(self, **kwargs)
File "fiona/ogrext.pyx", line 468, in fiona.ogrext.Session.start
File "fiona/_shim.pyx", line 73, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: '/vsis3/my_aws_bucket/my_reference_data/index.shp' not recognized as a supported file format.

In case it matters, index.shp in the stack trace above is a US national index shapefile pointing to state-specific child shapefiles.

The stack traces can also look like this:

my_shapefile = fiona.open(shapefile_uri)
File "/var/task/fiona/env.py", line 406, in wrapper
return f(*args, **kwargs)
File "/var/task/fiona/__init__.py", line 253, in open
layer=layer, enabled_drivers=enabled_drivers, **kwargs)
File "/var/task/fiona/collection.py", line 159, in __init__
self.session.start(self, **kwargs)
File "fiona/ogrext.pyx", line 484, in fiona.ogrext.Session.start
File "fiona/_shim.pyx", line 80, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: Failed to read all values for 33144 records in .shx file: No such file or directory.

These errors occur infrequently but are clustered in time – see below.

Steps to reproduce the problem.

I’m not able to reproduce the problem consistently. My AWS lambda code opens the same S3 shapefile every time the lambda is invoked. The fiona.open call succeeds most of the time (say, >95%), but I occasionally get the errors described above (and these errors tend to be clustered in time).

Operating system

AWS lambda with runtime = "python3.6" (see https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html).

Fiona and GDAL version and provenance

I’ve gotten this error with both fiona==1.8.6 and fiona==1.8a2.

[sgillies]

@atorch I have a correction to what I wrote above. Fiona has never enabled dataset handle sharing and so shared handles aren’t implicated.

[sgillies]

@atorch this sounds similar to the problem reported in https://rasterio.groups.io/g/main/message/225. I think it’s likely to involve GDAL and curl and that more clues will be found by running this program with CPL_CURL_VERBOSE=YES https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE. This setting dumps will dump out all the HTTP request headers and status codes.