Large rasters, metadata & "bounds" error

Hey all!

First of all, great work on Terracotta, it’s proven itself very useful .

We’re running it on an EC2 and trying to use large rasters through Lambda (The process is -> TIFF added to S3 bucket -> Lambda picks it up, and passes it to the EC2 to be processed -> the EC2 serves it to the front end). While it works for “small” rasters, we’re getting errors whenever the files’ resolution is bigger than 30K * 30K - we start running into memory issues: Unable to allocate 5.96 GiB for an array with shape (1, 80000, 80000) and data type uint8 (sometimes data type = bool). Thing is, Lambda can use up to 10GB so I’m struggling to see what the problem is.

We also get this error sometimes => PerformanceWarning: Processing a large raster file, but crick failed to import. Reading whole file into memory instead.

After digging in the code, we decided to use the use_chunks parameter but we keep getting an error back, and it’s not the most descriptive => 'bounds' is all we get back. Here’s how use it: driver.insert(keys, raster_file , metadata={'raster_path': f'{raster_file}', 'use_chunks': True}, override_path=f'{raster_file}')

So here are my questions:

• is there any limitation in terms of dimensions?
• why would crick fail to import?
• how are we supposed to use use_chunks ? I believe this should fix our issue, but I just can’t get it to work.

Thanks a lot in advance!

[dionhaefner]

BTW, you are getting KeyError: bounds from your call because you use the metadata keyword argument incorrectly. It expects a dict of pre-computed raster metadata (e.g. by driver.compute_metadata), and one of the keys is bounds, which is missing in your dict.

The correct call would be

metadata = driver.compute_metadata(raster_file, use_chunks=True)
driver.insert(keys, raster_file, metadata=metadata)

(but no need to do this, the default should work fine in your case)

See also: Terracotta Python API

[dionhaefner]

Yes, that is what we call ingestion. In that case the solution should be simple. If crick is not installed, Terracotta has to load the whole raster into memory which is why you get this error.

You have to make sure that crick is installed inside the Lambdas. How to do that depends on your deployment method. Then you should get away with using much less memory (< 1GB).