Ditch databases?

Currently, we have the following tables in a Terracotta database:

• keys: Contains defined key names and their description
• datasets: Maps key values to physical raster path
• metadata: Maps key values to raster metadata (such as min and max value, bounds, footprint, …)

An alternative model could be to save raster metadata on the rasters themselves. In that case, it would be much less likely for the raster metadata to go out of date. Having the metadata in a database only makes sense if we want to search it, which we currently don’t allow.

Doing this would even allow us to decouple the database from Terracotta entirely. We could then have an external database that the frontend can query for valid rasters, and request them from Terracotta by filename. This gives users flexibility to have a searchable catalogue outside of TC, and we could recover the current behavior by running directory listings on the raster folder.

[dionhaefner]

Simplest mode of usage:

(only recommended for data exploration)

$ terracotta serve -r myrasters/{date}/{tile}/{band}.tif

In this case

• /keys just gives [date, tile, band] with empty descriptions
• /datasets runs glob with the given pattern and extracts / filters keys
• /metadata opens GTiff and reads tags, if there are no tags metadata is computed sloppily based on an overview
• Metadata access triggers warning about missing tags and possibly missing cloud-optimization

Slightly more advanced:

(recommended usage)

# This optimizes the files and dumps them into the S3 bucket when done
$ terracotta prepare-rasters myrasters/**/*.tif -o s3://myrasters
$ export TC_KEY_DESC="s3://key_desc.json"
$ terracotta serve -r s3://myrasters/{date}/{tile}/{band}.tif

• /keys reads descriptions from given JSON file
• /datasets retrieves files list of entire bucket and filters it according to query parameters
• /metadata reads from GTiff tags

Advanced:

$ terracotta prepare-rasters myrasters/**/*.tif -o s3://myrasters
$ terracotta ingest s3://myrasters/{date}/{tile}/{band}.tif -o s3://myrasters/tc.sqlite
$ export TC_KEY_DESC="s3://key_desc.json"
$ terracotta serve -d s3://myrasters/tc.sqlite

• /datasets now runs efficient queries on the SQLite database

Custom paths to rasters:

(keys are not coupled to file paths)

Same as before, but use Python API to create SQLite database

External database:

Option 1a

$ terracotta serve -r s3://myrasters/{date}/{tile}/{band}.tif --external

• /keys and /datasets are disabled (TC is just used for serving metadata and tiles)
• User supplies an external discovery API that maps searchable parameters to Terracotta’s keys (or just one ID key)
• Discovery API could be a Terracotta plugin / related project

Option 1b

$ terracotta serve --external

• /keys and /datasets are disabled (TC is just used for serving metadata and tiles)

• User supplies an external discovery API that maps searchable parameters to file paths

• All raster queries require full paths:

  $ curl example.com/metadata?path=s3://myrasters/20180101/25XEL/B05.tif
  

Option 2a

$ terracotta serve -r s3://myrasters/{date}/{tile}/{band}.tif --external mysql://example.com:123456

• Enforce some sort of database structure

• Ingestion and creation could go through Terracotta

• /keys returns searchable fields

• /datasets does equality checks on searchable fields and returns keys used for raster retrieval:

  $ curl example.com/datasets?sensor=S2
  [{date: 20120101, band:B04, tile:25XEL}]  # note: no sensor here!
  

Option 2b

$ terracotta serve -r s3://myrasters/{date}/{tile}/{band}.tif --external mysql://example.com:123456

• Enforce some sort of database structure

• Ingestion and creation could go through Terracotta

• /keys returns searchable fields

• /datasets supports SQL SELECT syntax:

  $ curl example.com/datasets?where="sensor == 'S2' AND tile == '25XEL' AND cloudcover<=95"
  [{date: 20120101, band:B04, tile:25XEL}] 
  

[dionhaefner]

Well, the data has to live somewhere? S3 was just an example, could be any filesystem.