GraalJS and runner and reporter improvements - performance, streams, config

this is an “umbrella” issue and I may close some of the existing issues as duplicates. there is a refactoring that will go along with this, perhaps the graal migration

• ✅ introduce a scenario stream concept
  • a “generator function” that can produce scenarios at run-time, lazily for best performance
  • split scenarios for parallel execution
  • result streams can be combined to aggregate results
  • can Scenario Outline take a stream as the data-source ? may be hard
  • at the very least support a “generator” as the call argument
• ✅ test-run progress reporting
  • something we’ve discussed for the karate-runner VS Code plugin project
  • remaining tests progress indicator
  • dump stats to file that VS Code can pick up as it changes
  • move / rename previous report to backup file so it is not lost / overwritten (good to have)
• ✅ improve Embed design to store files and not hold all the bytes of large images in memory
• run in “quiet” mode, no reports and less logs, to enable bots and high-perf use
• ✅ report performance #1245
  • don’t write JSON, XML, HTML when instructed
  • can a file-stream be introduced for perf especially when the no of scenarios is huge
  • re-write JUnit XML report to use something SAX-like not DOM, also see https://stackoverflow.com/q/63747415/143475
• ✅ report vs console logging
  • decouple report verbosity and console logger, maybe use command-line / sys-prop #1156
  • request param or sys-prop should switch debug level for mock also
  • improve logging for multipart https://stackoverflow.com/q/64391922/143475
• ❌ runner pipelines (defer to later)
  • use-case to be able to sequence some tags (or scenario streams) to run first
  • and some control to be able to conditionally abort test after that phase
  • if that is too hard, at the very least plan to introduce a karate.abortAll() api to stop the entire test-suite
• ✅ retry
  • re-design the results objects so that we keep track of re-tries and still report “final” results without messing up scenario / pass / fail counts. this should play well with the “scenario streams” above
  • also support being able to “combine” 2 Result instances, so you can use the Runner multiple times in a “suite” but have a single report #1263
• ✅ user custom doc can appear inline #951
• ✅ improve HTML reporting
  • use new GraalJS based engine
  • templating and re-use system, allow users to create custom reports and contribute as plugin / JAR files
• ✅ report dry-run concept and @todo tag
  • the goal is to NOT require a test-management tool (as far as possible 😃
  • you should be able to run tests in “dry run” mode. this will give you the same old HTML report showing what features will be run, we will show all steps so you can read the HTML report. note that we now include comments - but of course the actual times, and logs will be missing. everything will pass
  • the useful part of the “dry run” report is the “Tags” section, showing what tags will run. this should suffice as a feature “coverage” report, provided you have a rich set of tags. e.g. @smoke @module=one @module=two etc
• ✅ because graal-js JS / match improvements - e.g. the JS arrow operator will finally be usable
• ✅ improve match failure reporting https://github.com/intuit/karate/issues/1012
• ✅ remove / deprecate cucumber reports from stand-alone JAR
• ✅ unify CLI #1177 - right now there are 2 for karate-core and karate-netty
• ✅ debug - improve variables display https://github.com/kirksl/karate-runner/issues/68
• ✅ header names should be case-insensitive #1360

[ptrthomas]

I’m going to declare this as done ! have updated the first comment with the 2 pieces we didn’t get to. but very happy with all that we were able to accomplish. we are on GraalJS, karate runs on JDK 16 and the code has been cleaned up very effectively

[ptrthomas]

@dersvenhesse please see https://github.com/intuit/karate/wiki/1.0-upgrade-guide - so right now you can test 0.9.9.RC3