Trackers: add stress-testing modules

To facilitate stress-testing for trackers we can add modules providing boilerplate code for each tracker allowing us to:

• check how long does it take to send big queue of events
• check if there’s any errors happening due sending big volumes of events
• compare performance of trackers

Though, I don’t see it as unit-test, but rather as manual approach with manual deployment of collector.

Here’s Node.js example:

// npm install node-uuid

// Random generator
var uuid = require('node-uuid');

// Initialization
var snowplow = require('snowplow-tracker');
var emitter = snowplow.emitter;
var tracker = snowplow.tracker;
var errors = [],
    successes = [],
    unknowns = [];

// Sent event counter
var sentEventCounter = 0;


function trackCallback(error, body, response) { // Callback called for each request
    sentEventCounter++;
    if (error) {
        errors.push({'error': error, 'sentEventCounter': sentEventCounter});
    } else if (body.statusCode === 200){
        successes.push(sentEventCounter);
    } else {
        unknowns.push({
            'error': error,
            'body': body,
            'response': response,
            'sentEventCounter': sentEventCounter
        });
    }
}

var e = emitter('collectorUri', 'http', 80, 'POST', 0, cb);
var t = tracker([e], 'myTracker', 'myApp', false);

// Getters
function getSuccesses() { return successes; }
function getErrors() { return errors; }
function getUnknowns() { return unknowns; }

function generateEvent() {
    var rnd = uuid.v4();
    var pageView = {
        'pageUrl': 'http://example.com/' + rnd,
        'pageTitle': rnd,
        'referrer': rnd
    };
    t.trackPageView(pageView.pageUrl, pageView.pageTitle, pageView.referrer);
}

function start() {
    for (var i = 0; i < 10000; i++) {
        generateEvent();
    }
}

module.exports = { 
    generateOneEvent: generateEvent, 
    getErrors: getErrors, 
    getSuccesses: getSuccesses, 
    getUnknowns: getUnknowns, 
    start: start 
};

[chuwy]

Hey QA-workgroup (@alexanderdean @BenFradet @jbeemster @ihortom)!

TL;DR I propose a TSV file format that trackers can read to get events and then we can compare Redshift content and these files.

I implemented small PoC of “QuickCheck for Snowplow” in Scala Tracker. It basically consist of:

1. Random event generator
2. Logic to save objects to TSV files with format similar to following:pv chuwy.me/page.html Main page http://google.com ttm:12312313253 where first pv refers to page view and subsequent columns are just tracking functions argument. Having that most of our trackers use same positional argument, I believe this should be quite ubiquitous
3. Logic to create emitter and tracker, read these TSV files and flood collector with these events from TSV.

What do I propose here? As I said we can re-use this format to see if there’s any data-loss on Tracker -> Collector part of pipeline, compare tracker’s performance and discover problems.

[alexanderdean]

Move to general tracker (protocol) repo.