Improve derived_tstamp/implement better approximation of actual time of event

Currently, the derived timestamp logic is as follows:

derived_tstamp = collector_tstamp - (dvce_sent_tstamp - dvce_created_tstamp)

As I understand it, the logic is that the collector timestamp is the first time we confidently establish a measurement of actual time, so we take that as the basis, and subtract lag between creating and sending the event.

This leads to unusual values in the case of events which are sent late due to connectivity issues.

A suggested improvement on this logic is as follows:

• Assume a reasonable time between collector_tstamp and dvce_sent_tstamp (I think this would be within a second or a few seconds)
• Calculate difference between collector_tstamp and dvce_sent_tstamp, if it’s above that number - this gives you how far off the actual time of day the devices clock is.
• Offset the dvce_created_tstamp by that delta to produce a more consistently accurate measurement of actual time of event.

I believe this is better because:

• no matter what, the dvce_created_tstamp always gives you when the events happened in relation to each other.
• Collector and dvce_sent timestamps should be very very close to each other. The difference between these values is therefore presumed to be an accurate measure of the device clock’s offset from actual time UTC
• Therefore the dvce_created_tstamp adjusted by this offset returns a good approximation of the actual time of event.

There’s still scope for inaccuracies in the rare case that the device clock changes between events, but avoids strange values resulting from lag between sending consecutive events.

[alexanderdean]

Right, yes - the calculation is sensitive to a change in clock between creation and sending.

We could probably mostly fix this by tracking device TZ before sending and then adjusting as needed (i.e. manual correction to dvce_tstamp in tracker). Only mostly fix because we can’t detect other changes like a manual clock adjustment. But they must be extremely rare on mobile.

[chuwy]

Moved to https://github.com/snowplow/enrich/issues/15