res.write not sending chunks until res.end() is called

I’m using node v.0.12.0 with express (v4.4.1) and compression (v1.6.0)

I’m sending back about 80MB of dynamically generated data (not from FS or DB) in multiple res.write() calls. When I add the compression middle ware (with no options passed), I don’t see any traffic (using WireShark) from the Node server until the res.end() is called. When res.end() is called, there is a sudden burst of chunked data.

But when the compression module is not used, I see chunked responses going on the wire. The size of each of these chunked fragments seems to be between 10K and 16K (based on WireShark).

The only header information I am setting happens before the res.write() calls and is:

res.setHeader('Content-Type', 'application/json');

Any reason why the data is getting buffered before the res.end() call?

[bijoythomas]

So I figured out what is happening. The same issue exists in the last version of Node too. The way I am sending data is something like below

for (i-0; i<some_large_number; i++) {
 poppulate_data_in_array(arr);
 if (arr.length === 500) {
   res.write(SON.stringify(arr));
   res.flush();
   arr = [];
}

// Finally if there is anything in array, send it across too

A look at the zlib.js source https://github.com/nodejs/node/blob/master/lib/zlib.js line 448 shows that the flush() call is not synchronous. Instead, it sets up a ‘drain’ listener to execute later. But with my for-loop consuming the thread, this listener has no chance to run. In fact, the loop will keep adding the listeners for ’ drain’ and that explains why I was getting the error below: (node) warning: possible EventEmitter memory leak detected. 11 drain listeners added

I changed my code to allow the execution of the listener by making use of the callback that the Zlib.flush() takes and trampoline between data generation and sending it. So it looks something like below:

function populateArray() {
    if (iterCount < maxCount) {
      arr.push(generate_some_data());
    }
    send();
}

function send() {
    if (iterCount === maxCount) {
      if (arr.length > 0) {
        res.write(JSON.stringify(arr));
      }
      res.end();
      return;
   }

    if (arr.length === 500) {
      res.write(JSON.stringify(arr));
      res.flush(function() { // <--------- callback to flush which on invocation resumes the array population
        arr = [];
        populateArray(++iterCount);

      });
    } else {
      populateArray(++iterCount);
    }
}

In order to get this to work, I had to change the flush implementation to take a callback

res.flush = function flush(cb) {
      if (stream) {
        stream.flush(opts.flush, cb);
      }   
}

Where opts is the variable that captures the options passed into the compression function. The Zlib flush implementation takes the flush mode but I’m not aware of a practical use case of using multiple flush modes for a single compression run. Or if there is one, the function could take the addition flush argument but the user has to make sure the right mode is passed in.

[dougwilson]

This is because it’s how compression works with gzip: to get small sizes, gzip needs to accumulate the payload so it can do substring search and replacements.

If you want to stream, you can use the res.flush() command between writes, but the compression will be much less efficient.

An example can be found at the bottom of the readme.