Postgres support for shredded events

Sorry, couldn’t find any existing tickets regarding this (except #2475), but pretty sure they exist.

Just wanted to clarify what we need to do to make this happen, since we now considering removing Postgres support from StorageLoader/RDB Loader.

From my point of view, main missing components are:

1. Spark job that takes events and JSON paths and produces TSV. Rough scalaish pseudo-code:

def convertJsonToTsv(jsonPaths: JsonPath, json: Hierarchy): TSV = ???

def produceTsv(igluResolver: Resolver, json: Hierarchy): TSV = {
  val jsonPaths = igluResolver.fetchJsonPaths(json.schema)
  convertJsonToTsv(jsonPaths, json)
} 

1. Notion of JSON Paths in Iglu.
2. Actual RDB Postgres loading code.

@alexanderdean did I miss something?

[claudiomartins]

+1 for Postgres shredding - we extensively use Postgres for all our needs and would help to not have to run another ETL to export data out of Redshift into Postgres.

[chuwy]

Moved to https://github.com/snowplow/snowplow-rdb-loader/issues/47, closing this one.