Response.json() never resolves

I have an interesting issue with request that essentially never finishes. It is tricky to reproduce, but everything boils down to proper timing such that there are response headers, but not response body yet.

In such circumstances I’ve uncovered 2 issues:

1. Timeouts are not propagated to cloned responses, so those never time out (fixed by https://github.com/bitinn/node-fetch/pull/664)
2. Sometimes even though complete body was received (accumBytes is the same as Content-Length), body.on('end', ...) is never fired

I’m not sure how to fix the second one, one reason may be that response.size is 0 for some reason, looks like it is not read from headers or something. Hopefully maintainers can figure that out.

But at very least with mentioned PR merged it will handle timeouts on cloned response properly.

Reduced example of the code I’m dealing with is like this:

import fetch from "node-fetch";

(async () => {
    const response = await fetch(
        'http://domain.tld/path',
        {
            timeout: 2000,
        },
    );

    console.log(response.status, response.headers);
    console.log(await response.clone().json());
    console.log('This may not be printed ever, without any errors');
})();

[bitinn]

The correct solution is to resolve cloned response in parallel. See our test cases for an example (search for Promise.all).

The lazy solution is to set a higher water mark during clone, this has been introduced in v3 also.

(From my phone)

On Oct 7, 2019, at 20:52, Nazar Mokrynskyi [e-mail] wrote:

So the issue is buffer determined by highWaterMark parameter of PassThrough stream. When 2 of those streams are created and only one is read, the buffer of second stream is filled till the limit, after which streaming to both stops until someone consumes data from the second stream.

For instance, this modification works:

const streamBody = new PassThrough(); body.pipe(streamBody); const streamBody2 = new PassThrough(); body.pipe(streamBody2); body = streamBody; streamBody2.on(‘data’, () => {});

return new Body.Promise(function (resolve, reject) { Not sure how to fix it though besides specifying large highWaterMark on PassThrough such that it will fit the whole response body in it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nazar-pc]

Have to reopen, turns out it is still an issue and is related to chunked encoding apparently. So this works fine:

import fetch from "node-fetch";

(async () => {
    const response = await fetch(
        'http://domain.tld/path',
        {
            timeout: 2000,
        },
    );

    console.log(await response.json());
})();

This will timeout if response headers contain Transfer-Encoding: chunked (in my case response from Discord that contains a lot of guild channels, like GET https://discordapp.com/api/v6/guilds/X/channels):

import fetch from "node-fetch";

(async () => {
    const response = await fetch(
        'http://domain.tld/path',
        {
            timeout: 2000,
        },
    );

    response.clone();
    console.log(await response.json());
})();

Still investigating why exactly.