DriverError: Operation timed out when opening shapefile with geopandas from S3 bucket

I am getting this error inside a docker container built with the only command FROM pangeo/pangeo-notebook:latest, I hope this helps defining the environment, but I’m happy to provide more explicit information if needed.

I have verified that I have access to the private S3 bucket where the file is located, and I can correctly open parquet files (for example) from the same bucket with geopandas, so this issue seems specific to fiona (and/or gdal).

In [1]: import geopandas as gpd

In [2]: gpd.read_file("s3://path/to/shapefile.shp")
---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
fiona/_shim.pyx in fiona._shim.gdal_open_vector()

fiona/_err.pyx in fiona._err.exc_wrap_pointer()

CPLE_AppDefinedError: Operation timed out after 1001 milliseconds with 0 bytes received

During handling of the above exception, another exception occurred:

DriverError                               Traceback (most recent call last)
<ipython-input-2-068760f4508f> in <module>
----> 1 gpd.read_file("s3://path/to/shapefile.shp").geometry

/srv/conda/envs/notebook/lib/python3.9/site-packages/geopandas/io/file.py in _read_file(filename, bbox, mask, rows, **kwargs)
    199
    200     with fiona_env():
--> 201         with reader(path_or_bytes, **kwargs) as features:
    202
    203             # In a future Fiona release the crs attribute of features will

/srv/conda/envs/notebook/lib/python3.9/site-packages/fiona/env.py in wrapper(*args, **kwargs)
    406     def wrapper(*args, **kwargs):
    407         if local._env:
--> 408             return f(*args, **kwargs)
    409         else:
    410             if isinstance(args[0], str):

/srv/conda/envs/notebook/lib/python3.9/site-packages/fiona/__init__.py in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, **kwargs)
    254
    255         if mode in ('a', 'r'):
--> 256             c = Collection(path, mode, driver=driver, encoding=encoding,
    257                            layer=layer, enabled_drivers=enabled_drivers, **kwargs)
    258         elif mode == 'w':

/srv/conda/envs/notebook/lib/python3.9/site-packages/fiona/collection.py in __init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, **kwargs)
    160             if self.mode == 'r':
    161                 self.session = Session()
--> 162                 self.session.start(self, **kwargs)
    163             elif self.mode in ('a', 'w'):
    164                 self.session = WritingSession()

fiona/ogrext.pyx in fiona.ogrext.Session.start()

fiona/_shim.pyx in fiona._shim.gdal_open_vector()

DriverError: Operation timed out after 1001 milliseconds with 0 bytes received

[LewisJarrod]

Thanks, and I can confirm we are using version 1.8.21. We will implement the workaround for now!

[sgillies]

@LewisJarrod I wonder if this is related to https://github.com/rasterio/rasterio/issues/1521? The fiona 1.8.x series lacks that fix, and I will look into that.

@rbuffat’s suggestion of creating a new session object for each use of fiona is the workaround for now.