Proposal: test.resource()

The problem

When integration-testing web apps (which AVA is great for since it runs tests in parallel), I sometimes run into issues with what I think is resource constraints, even on my high-end Macbook Pro. This results in the following sadness:

To fix this, I need to run AVA with --c 6 to lower the concurrency which is fine but if my MBP is using more resources for other things, I need to lower it even more.

The reason for this, is that I in each test suite run my HTTP server, set up a bunch of helpers for testing my API, connect to websockets, and then finally run my test. This is my pattern:

test('GET /todos returns all todos', withServer(async (t, api) => {
  const todos = await api.getTodos()
  t.true(todos.length > 0)
}))

Where withServer is just a factory function that sets up the server:

const axios = require('axios')
const createServer = require('../src/server')

function withServer (testFn) {
  return async function (t) {
    const server = (await createServer()).listen()
    const host = `http://127.0.0.1:${server.address().port}`
    const client = axios.create({ baseURL: host })
    const api = {
      getTodos: () => client.get('/todos').then(r => r.data)
    }
    
    await testFn(t, api)
  }
}

This makes it super easy for me to write integration tests for my API. Problem is, each test need their own connection to the server (in case of websockets), and even when I memoize the server (which only works per suite due to each suite being in a separate process), I still run into the above issue.

I think this is because I am using rethinkdbdash which manages a large connection pool. For a single server instance or 4 that might be fine, but for 10+, that might be why it starts having issues.

The proposed solution

What I would really like to do, is set up just a single instance for all my tests to work with.

One way to do it is to run npm start & npm test, but if npm start takes too long to actually start the server, then that’s gonna be a problem.

Instead, what if we had a way to tell AVA to run a JS function before starting the test processes? For example, if I was able to do this:

// integration-test.resource.js
import test from 'ava'
import createServer from '../src/createServer'

// This runs before any other test processes have been spun up
test.resource('Set up API server', async (t) => {
  const server = (await createServer()).listen()
  const host = `http://127.0.0.1:${server.address().port}`
  
  // Only JSON-serializable values can be provided to test processes
  t.resource('serverHost', host)
  
  return async function dispose () {
    // When all tests are done, dispose of the resource
    server.close()
  }
})

And in my test

test('GET /todos', async (t) => {
  const client = axios.create({ baseURL: t.resource('serverHost') })
  const { data } = await client.get('/todos')
  t.true(data.length > 0)
})

The result is a single HTTP server that is ready when tests run, and a way to tear it down once all tests have run (pass or fail, does not matter). This is way more CI friendly than the npm start & npm test approach.

Proposed Configuration

To load these resources via CLI:

ava 'test/*.test.js' --resources 'test/resources/*.resource.js'

Via package.json

{
  "ava": {
    "resources": [
      "test/resources/*.resource.js"
    ]
  }
}

[jeffijoe]

Yes, exactly. Helper worker is an even better name for it.

It will make AVA more scalable in the sense that I don’t have to consolidate a bunch of assertions into a single test case just to avoid the extra overhead.

Setting up API servers for each test also takes a while; most of the time I want to wait for certain connections to be made before I start listening, adding startup time to each test case. If I could do this once per run, that would cut down test run time dramatically.

[ronen]

Minor additional thought on the proposed API: In the above examples, AVA doesn’t know about the serverHost keyword until after the server has successfully spun up and t.provide() is called, making it hard for AVA to administer the resources.

Instead, I think the resource keyword should be supplied to test.resource() rather than to t.provide():

test.resource('myServer', async (t) => {
  const server = (await createServer()).listen()
  const host = `http://127.0.0.1:${server.address().port}`
  t.provide({ value: host, concurrency: 8 })
  return async function dispose () {
    server.close()
  }
})

This way:

• As soon as AVA executes all the test.resource() methods, it will immediately know all the resource keywords. It wouldn’t need to wait for the async resource definition methods to resolve, it could leave them running asynchronously and start executing the tests

• When a test calls await t.resource(keyword), AVA will know immediately if no such resource were defined, and could immediately fail. Otherwise it would block until that resource calls its t.provide() method.

• The configuration files/CLI could allow configuration for specific resource to be keyed by its keyword. E.g. something like:

  {
     "resources": {
         "myServer": { 
                 "concurrency": 4,
                 "params": { "...": "arbitrary data" }
          },
     }
  }
  

  And then the execution context (t param) to the resource definition could include that configuration info, allowing const server = (await createServer(t.params)).listen() or whatever.