Summary mode

Thanks again for the excellent work. Here’s a suggestion to hopefully make the CLI more versatile.

Goals

1. focus on only high level info
2. efficiently go through many requests
3. use in continuous monitoring and trigger alarms

Proposal

There is a helpful --verbose mode, but there doesn’t seem to be a way to make httpyac more concise.

For just one request being tested in the normal mode, with a tiny body response:

https://the.server.dev/checks/health => 200
GET https://the.server.dev/checks/health
accept: */*
accept-encoding: gzip, deflate, br
user-agent: httpyac

HTTP/1.1 200 OK
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400
cf-ray: 66b9e0e55d873469-NRT
connection: close
content-length: 15
content-type: application/json
date: Thu, 08 Jul 2021 14:07:22 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
nel: {"report_to":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v2?s=u2dylFQnhzyetY8zUENOcv4l0%2FvmxRW8%2FmAKXfC0Kyna8uVZddEaWx6oSfQBJfM%2FwDfKchWeiwRdwhXSP7VykaQnTuM0mNTl0HCkd4%2BWhjtUB%2FcmtTroVQnfMEe%2Bs%2F8dw8Lh"}],"group":"cf-nel","max_age":604800}
server: cloudflare
vary: Accept-Encoding

{
  "status": "OK"
}
Tests for /checks/health
✓ Request succeeded
✖ Schema is valid (AssertionError [ERR_ASSERTION]: Response schema is incorrect: does not exactly match expected constant: ok - t.IntellijHttpClient.assert (/Users/name/.nvm/versions/node/v14.15.5/lib/node_modules/httpyac/dist/index.js:1:32228)

This is very helpful a lot of the time, but it is a bit difficult to use when running large collections of tests, or tests with large bodies, or if trying to use it in a continuous monitoring situation.

We could imagine something like this as an extra output mode:

Summary line

The summary line would be of some simple format like:

? requests tested (? succeeded, ? failed with ? CRITICAL)

So that an evaluation script may look at the last line and check that 0 failed or 0 CRITICAL is found, or do something if not (like trigger alerts, save or forward the full log…).

Options

Perhaps something like:

--summary             summary as above, with all requests included
--summary --failed    same format but showing just requests involving at least one failed test
--summary --critical  same format but showing just requests which have CRITICAL failures
--summary --json      outputs everything in json format for easy parsing

[AnWeber]

Yes, your guess is right. The Name of a request is used as variable name (used with @ref). I wanted to test, if I can reuse the title as name, but I guess not:-) I will change it to title. You don’t have to feel obligated to test. I think it’s great, but it should be fun for us first and foremost.

updated npm to 2.20.0-beta.2

[AnWeber]

Thanks again for all the ideas. I think I currently have everything covered in this ticket. I will deal with anweber/vscode-httpyac#50 next