RDB Loader: Inconsistent resolution of Snowplow-hosted jsonpaths

https://github.com/snowplow/snowplow/blob/8e5e1901af4c9073d0dceda2e2450ea6e6b486bb/4-storage/rdb-loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/ShreddedType.scala#L158

The RDB Loader is hard-coded to use s3://snowplow-hosted-assets[-region] to retrieve jsonpaths for Snowplow-authored jsonschemas.

This seems inconsistent with other Snowplow components which use the configurable value in config.yaml (aws/s3/buckets/assets) and with the Iglu model in which each repo has its own jsonpaths store.

Is there a reason for this? If not I am happy to write a PR.

[chuwy]

Sorry, @acgray, I’m still not following. You still can use aws.s3.buckets.jsonpath_assets setting in config.yml to point RDB Loader to your mirror. If your mirror is identical to snowplow-hosted-assets then Loader won’t ever reach our bucket, because custom assets bucket (jsonpath_assets) is always prioritized. This is how StorageLoader worked and I see it’s quite consistent with other parts of pipeline. Please, correct me if I missing something. If what I described is not possible now - then it’s quite serious issue.

You’re totally right at this point. We’re working on providing more consistent (and format-independent) approach to work with Iglu registries. But so far, there’s no clear direction for this problem. I cannot give you any clues or ETA so far to not also give false promises, but you can be sure - we’re agree and working on that.

[chuwy]

Hey @acgray. I’m moving this into new repository, let’s continue discussion there: https://github.com/snowplow/snowplow-rdb-loader/issues/4