Replace Gavel Validation With Custom Validation

Gavel currently uses a simple text diff to validate HTML response entity bodies. I would like Gavel to support X/HTML validation in the future using XML Schema. However, here is how I am circumventing Gavel validation, and validating a response myself.

var hooks = require('hooks');
var stash = {};
...
/**
 * Validate HTML generated from an API Blueprint. Gavel validates X/HTML using
 * Simple Text Diff. Simple Text Diff is primitive. We can do better.
 * If the HTML passes our validation, then force Gavel to pass by masking the
 * real response body with the expected body, else fail before Gavel validation.
 *
 * @see {@link https://www.relishapp.com/apiary/gavel/docs/expectations}
 */
function validateBlueprintHtml(transaction) {
  var protagonist = require('protagonist');
  var cheerio = require('cheerio');
  var $ = cheerio.load(transaction.real.body);
  var apibName = protagonist.parseSync(transaction.request.body).ast.name;
  var htmlName = $('#apiHeadline').text();
  var valid = apibName === htmlName;

  if(valid) {
    transaction.real.body = transaction.expected.body;
  } else {
    transaction.fail = "Generated HTML Does Not Include An API Name From Its Blueprint.";
  }
}
...
hooks.beforeValidation('Blueprint > Generate Blueprint HTML > Generate Blueprint HTML', validateBlueprintHtml);

As a user, I want to replace Gavel validation in cases where testing a transaction could not feasibly pass because a real response body will never equal an expected response body.

Perhaps there could be a hooks.validation() or hooks.duringValidation() that allows for an alternative to Gavel. Perhaps there could also be transaction.pass, with similar intention as transaction.fail, to pass before or during Gavel validation.

[artem-zakharchenko]

As I understood the original intention is to replace Gavel validation by arbitrary validation. In that case the usage of Gavel as a validator should be opted-out, meaning Gavel must not be called in the first place.

Perhaps, a hooks.validate() hook to perform the validation by yourself? It would be required to meet the Gavel’s unified validation result structure in that case. It’s quite generic at this point, so it’s not coupled with any kind of underlying validator. This way the input to Dredd would be reliable, as there can be multiple things related on the validation result (argument against manual transaction mutation to imitate custom validation).

[honzajavorek]

Dredd’s life cycle, as it was designed and documented, was broken until now - you could not change the transaction.test object in hooks and in some cases, Dredd would emit test fail events to reporters right after Gavel validation. This is now fixed in https://github.com/apiaryio/dredd/pull/648.

I’m pretty confident we have now, as @netmilk wrote, everything in place. Moreover, everything should also work as expected and documented.

Hence implementing this in hooks should be entirely possible now and not only with afterEachValidation hooks, but also with after and afterEach hooks. This issue can be closed if the use case gets a nice, working example in documentation (and preferably also test).