Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Scala Common Enrich: add validation for contexts added by enrichments

See original GitHub issue

API Request enrichment works on already validated contexts and self-described events and does not validate produced contexts, therefore there’s always a chance that output data won’t correspond to outputs.schema.

Issue Analytics

State:
Created 5 years ago
Reactions:4
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

chuwycommented, Nov 22, 2021

Hey @benjben, sorry missed your last message. As we discussed in internal RFC doc yesterday, there are many more opportunities for loaders to fail. Biggest ones are:

Unavailable (or wrong version of) Iglu Registry. Loaders have their own requirements for registries
Unavailable DB. BQ failed inserts exist because BigQuery was unavailable
Bug in a shredding/casting algorithm
Improperly versioned schema

0reactions

chuwycommented, Nov 21, 2019

Possible duplicate: https://github.com/snowplow/snowplow/issues/2639

Top Results From Across the Web

R71: JSON validation in Scala Common Enrich - Enrichment ...

Hi guys We're looking to upgrade our pipeline and was trying to reconcile differences between processing a set of raw logs on an...

Enrichment and batch processing in Snowplow - datamindedbe

Although important and central to Snowplow's value-add, validation is not actually a step but instead happens throughout the enrichment process.

Everything about Snowplow Analytics - Aswin Kumar Rajendran

Enrichment process parses raw Snowplow events and performs the following: Extracts data. Validates data against Snowplow Tracker Protocol and JSON schema.

Create "enriched" type from case class with Shapeless in Scala

Instead of DatabaseIncomingThing val updated: DatabaseIncomingThing = genDatabase.from(structuralVersionOfIncomingThing + ('id ->> UUID.

Perpetual Stream - squbs

In general, you do not need to add an extra dependency. ... The PerpetualStream gets exposed as the PerpetualStream trait for Scala and ......