ERR_HTTP_HEADERS_SENT from onProxyReq with keep-alive

Under load, when using a keep-alive agent, if the # of incoming requests exceeds the max sockets, we are seeing a ERR_HTTP_HEADERS_SENT very quickly.

Is this a bug report?

Yes

Steps to reproduce

1. Configure proxy middleware with a keep-alive agent
2. Add an onProxyReq handler that sets a header
3. Apply load with a larger number of threads/VUs than available sockets.

Expected behavior

No errors, perhaps timeouts due to socket availability.

Actual behavior

Hard crash with ERR_HTTP_HEADERS_SENT.

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ClientRequest.setHeader (_http_outgoing.js:485:11)
    at ProxyServer.onProxyReq (C:\dev\nwe-jss-test\jss13.0.1-sc9.3.0\keepalive-middleware-test\index.js:23:18)
    at ProxyServer.emit (C:\dev\nwe-jss-test\jss13.0.1-sc9.3.0\keepalive-middleware-test\node_modules\eventemitter3\index.js:184:35)
    at ClientRequest.<anonymous> (C:\dev\nwe-jss-test\jss13.0.1-sc9.3.0\keepalive-middleware-test\node_modules\http-proxy\lib\http-proxy\passes\web-incoming.js:133:16)
    at ClientRequest.emit (events.js:215:7)
    at tickOnSocket (_http_client.js:684:7)
    at onSocketNT (_http_client.js:723:5)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

Setup

• http-proxy-middleware: 1.0.5, reproducible on 0.2.0 as well
• http-proxy-middleware configuration: see below
• express: 4.16.4
• agentkeepalive: 4.1.3

client info

k6, see below

target server info

IIS

Reproducible Demo

Server

const express = require('express');
const { createProxyMiddleware } =  require('http-proxy-middleware');
const keepAlive = require("agentkeepalive");

const apiHost = 'http://localhost';

const agent = new keepAlive({
    maxSockets: 120,
    maxFreeSockets: 120,
    timeout: 60000,
    freeSocketTimeout: 30000,
});

const server = express();
server.get('/connections', function(req, res, next) {
    res.status(200).send({ sockets: agent.getCurrentStatus() })
  });
  server.use('*', createProxyMiddleware({
    agent,
    target: apiHost,
    changeOrigin: true,
    onProxyReq: (proxyReq, req, res) => {
        proxyReq.setHeader('TEST', 'hello world');
    }
}));
server.listen(3000);

K6 Project

import http from 'k6/http';
import { Counter } from "k6/metrics";

export let options = {
    discardResponseBodies: false,
    scenarios: {
        headRequest: {
            executor: 'constant-vus',
            exec: 'headRequest',
            vus: 150,
            duration: '30s',
        }
    },
    thresholds: {
        "pageError": [{threshold: "count<1", abortOnFail: true}],
    }
};

const pageError = new Counter("pageError");

export function headRequest() {
    var response = http.request('GET', 'http://localhost:3000/');
    if (response.error || response.status !== 200) {
        pageError.add(1);
    }
}

[bobalong79]

I’ve been having the same issue and I think it happens when any agent is used, not something specific to keepalive (I’m using global-agent’s HttpsProxyAgent). My workaround is to use the custom filter mechanism to alter the headers prior to the proxy request being set up. It looks like this filter mechanism was designed to allow you to implement custom matching logic, but as it provides access to the req: IncomingRequest object before the proxy request is created it is a convenient way to manipulate the headers before they even reach the http-proxy code.

const filter = (_pathname: string, req: http.IncomingMessage): boolean => {
  req.headers['TEST'] = 'hello world';

  return true;
};

server.use('*', createProxyMiddleware(filter, {
  agent,
  target: apiHost,
  changeOrigin: true
}));

I think this is basically the same as what you’ve done but it keeps all the logic in the same middleware.

I guess one drawback of this approach (and your workaround) is that because the headers are changed on the original incoming request rather than on the proxied request, other downstream routers that rely on “untampered” headers might be compromised. In that case then you’d need to keep a copy of the original set and copy them back after onProxyRes completes. I don’t need to do that for my application though.

[JounQin]

I also meet this issue when using fixRequestBody from this middleware.