Improve Programmatic Usage

What I did

I’ve got a bunch of tests that I want to run in a specific order, e.g. “create the user” -> “log in as user” -> “log out as user” (where each of those has a bunch of tests).

Each of those tests is in its own file, then I created a test runner so I can manage state easier.

The test files all export a default function that takes test (the suite) and assert (for convenience):

// tests/*.js
export default (test, assert) => {
	test('example test', () => {
		assert.is(true, true);
	});
}

The custom test runner sets up state, and manually controls the suite ordering:

// runner.js
import { exec, suite } from 'uvu'
import * as assert from 'uvu/assert'

const scenarios = [
	'tests/aaa.js',
	'tests/ccc.js',
	'tests/bbb.js'
]

const run = async () => {
	for (const scenario of scenarios) {
		const test = suite(scenario)
		const run = await import(scenario)
		run.default(test, assert)
		test.run()
	}
	return exec()
}

run()
	.then(() => { console.log('done') })
	.catch(error => { console.error('error', error) })

Then I execute the runner simply node ./runner.js

What I expect

When I execute the runner, normal uvu output should list each suite:

$ node ./runner.js

 tests/aaa.js  •   (1 / 1)
 tests/ccc.js  •   (1 / 1)
 tests/bbb.js  •   (1 / 1)

  Total:     3
  Passed:    3
  Skipped:   0
  Duration:  0.29ms

What actually happens

If I try reordering the tests, they are all run in order, except always the first one is skipped and prints this instead:

$ node ./runner.js

function () { [native code] }
 tests/ccc.js  •   (1 / 1)
 tests/bbb.js  •   (1 / 1)

  Total:     2
  Passed:    2
  Skipped:   0
  Duration:  0.29ms

Other information

$  node -v
v16.0.0

$  npm -v
7.12.1

$  hostinfo 
Mach kernel version:
	 Darwin Kernel Version 20.4.0: Fri Mar  5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101

[lukeed]

Just an update:

I’m working on this now, which is effectively a rewrite of uvu. It will allow you to call a new run() export which will provide a JSON report of the results. I don’t have anything concrete yet (I started on this last night) but it will touch on multiple open issues: #14, #38, #52, #60, #80, #130

[saibotsivad]

Just a note to myself / whoever else stumbles across this: if you don’t bail (i.e. uvu.exec(true)) then the way to get the exit status code is using process.exitCode afterward, e.g.:

const run = async () => {
  // ... any of the forms @lukeed wrote
  return uvu.exec()
}
// ... where you call `run`
await run()
// after the promise resolves
if (process.exitCode) {
  // one or more tests failed
} else {
  // all tests passed
}