Getting a 500 when using the newest version of Rasterio (1.0.22)

When deploying a Lambda function through Zappa, using the newest version of Rasterio (1.0.22), we get a 500 for some tiles, while others load fine.

When reverting back to Rasterio 1.0.13 everything works fine.

zappa tail gives the following:

[1554888957364] [ERROR] 2019-04-10T09:35:57.326Z b24c3c65-85ed-4390-a18b-c0cf347feec0 Exception on /rgb/italy/2018/7/70/47.png [GET]
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 213, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 64, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsis3/italy-composite/rasters/italy_2018_red.tif' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/terracotta/drivers/raster_base.py", line 464, in _get_raster_tile
  src = es.enter_context(rasterio.open(path))
  File "/var/task/rasterio/env.py", line 423, in wrapper
  return f(*args, **kwds)
  File "/var/task/rasterio/__init__.py", line 216, in open
  s = DatasetReader(path, driver=driver, **kwargs)
  File "rasterio/_base.pyx", line 215, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: '/vsis3/italy-composite/rasters/italy_2018_red.tif' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/flask/app.py", line 2292, in wsgi_app
  response = self.full_dispatch_request()
  File "/var/task/flask/app.py", line 1815, in full_dispatch_request
  rv = self.handle_user_exception(e)
  File "/var/task/flask/app.py", line 1718, in handle_user_exception
  reraise(exc_type, exc_value, tb)
  File "/var/task/flask/_compat.py", line 35, in reraise
  raise value
  File "/var/task/flask/app.py", line 1813, in full_dispatch_request
  rv = self.dispatch_request()
  File "/var/task/flask/app.py", line 1799, in dispatch_request
  return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/task/terracotta/server/flask_api.py", line 50, in inner
  return fun(*args, **kwargs)
  File "/var/task/terracotta/server/rgb.py", line 85, in get_rgb
  return _get_rgb_image(keys, tile_xyz=tile_xyz)
  File "/var/task/terracotta/server/rgb.py", line 135, in _get_rgb_image
  some_keys, rgb_values, stretch_ranges=stretch_ranges, tile_xyz=tile_xyz, **options
  File "/var/lang/lib/python3.6/contextlib.py", line 52, in inner
  return func(*args, **kwds)
  File "/var/task/terracotta/handlers/rgb.py", line 85, in rgb
  band_data = band_data_future.result()
  File "/var/lang/lib/python3.6/concurrent/futures/_base.py", line 432, in result
  return self.__get_result()
  File "/var/lang/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
  raise self._exception
  File "/var/lang/lib/python3.6/concurrent/futures/thread.py", line 56, in run
  result = self.fn(*self.args, **self.kwargs)
  File "/var/task/cachetools/__init__.py", line 87, in wrapper
  v = method(self, *args, **kwargs)
  File "/var/lang/lib/python3.6/contextlib.py", line 52, in inner
  return func(*args, **kwds)
  File "/var/task/terracotta/drivers/raster_base.py", line 466, in _get_raster_tile
  raise IOError('error while reading file {}'.format(path))
OSError: error while reading file s3://italy-composite/rasters/italy_2018_red.tif

[j08lue]

New insights. We can get rid of the issue with rasterio.errors.RasterioIOError: '/vsis3/italy-composite/rasters/italy_2018_red.tif' not recognized as a supported file format. by setting GDAL_DISABLE_READDIR_ON_OPEN=FALSE. We were using EMPTY_DIR before.

We now tried FALSE to see whether we would get a different error message than the not recognized as supported file format, following https://github.com/mapbox/rasterio/issues/1706.

The EMPTY_DIR was recommended among others here: https://github.com/mapbox/rasterio/issues/987.

Whether the change from EMPTY_DIR to FALSE actually prohibits the actual cause of the S3 read failure or just prevents some race condition because it changes the timing of things, we do not know. But it seems to work consistently.

[mrpgraae]

Small update on this. I’m seeing quite a few changes to /vsiscurl/ and /vsis3/ in the GDAL 2.4.0 changelog. I’m currently trying to build the Rasterio wheels with an older GDAL version to see if that fixes it.