`--amount` is not precise with high concurrency

Example server (modified from restify/benchmarks):

// bare.js
'use strict'

let i = 0
const server = require('http').createServer(function (req, res) {
  i++
  res.setHeader('content-type', 'application/json; charset=utf-8')
  res.end(JSON.stringify({ hello: 'world' }))
})
process.on('SIGINT', () => {process._rawDebug(`Reqs' ${i}`); process.exit(0)})

server.listen(3000)

Running npx against the server above with -a 1000000 and with the same parameters used on restify/benchmarks:

$ node benchmarks/bare.js &
[1] 14966

$ npx autocannon --debug -c 100 -p 10 -a 1000000 localhost:3000
Running 1000000 requests test @ http://localhost:3000
100 connections with 10 pipelining factor

running [====================] 99%┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%   │ Avg     │ Stdev   │ Max      │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────────┤
│ Latency │ 0 ms │ 0 ms │ 11 ms │ 13 ms │ 1.42 ms │ 3.75 ms │ 130.8 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────────┘
┌───────────┬────────┬────────┬─────────┬─────────┬──────────┬──────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%     │ 97.5%   │ Avg      │ Stdev    │ Min    │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼──────────┼────────┤
│ Req/Sec   │ 5211   │ 5211   │ 68927   │ 75967   │ 62447.63 │ 17040.52 │ 5208   │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼──────────┼────────┤
│ Bytes/Sec │ 855 kB │ 855 kB │ 11.3 MB │ 12.5 MB │ 10.2 MB  │ 2.8 MB   │ 854 kB │
└───────────┴────────┴────────┴─────────┴─────────┴──────────┴──────────┴────────┘

Req/Bytes counts sampled once per second.

999k requests in 16.08s, 164 MB read

$ kill -INT 14966
Reqs' 999969                                                                                                                                                                                                                                                                                                                                                                                                                                 
[1]  + 14966 done       node benchmarks/bare.js

autocannon reports 999k requests (instead of 1000k), and when we stop the server it reports 999969 requests (instead of 1000000). With -c 1 and -p 1 the server will reply to 1000000 requests correctly.

This might be the server unable to handle that many concurrent requests, but autocannon doesn’t report any errors (not even with --debug), so it might be a bug on autocannon as well.

I saw this issue on both Node.js v10 and v12, and on autocannon 3.2.2 as well as 4.4.0.

[elirandav]

sure. I created a pull request @mcollina

[mcollina]

@elirandav Would you like to send a Pull Request to address this issue? Remember to add unit tests.