Bug: node-fetch doesn't throw an error when downloading a big file fails

When downloading a file bigger than 4 GiB (= 2³² bytes) with http in a 32-bit environment, Node.js fails and throws the HPE_INVALID_CONSTANT error. With node-fetch, the download also fails (the file is truncated), but no error is raised.

Reproduction

• server.js

const http = require("http");
const stream = require("stream");

const ZeroFillReadable = class extends stream.Readable {
    constructor(length) {
        super();
        this.length = length;
    }

    _read() {
        if (this.length === 0) {
            this.push(null);
        } else {
            const size = Math.min(this.length, 1024);
            this.push(Buffer.alloc(size));
            this.length -= size;
        }
    }
};

const server = http.createServer((req, res) => {
    const length = Number.parseInt(req.url.slice(1));

    res.setHeader("Content-Type", "application/octet-stream");
    res.setHeader("Content-Disposition", "attachment; filename=data.rnd");
    res.setHeader("Content-Length", length);

    new ZeroFillReadable(length).pipe(res);
}).listen(8000, () => {
    console.log("Listening on 8000");
});

• client_fetch.js

const fs = require("fs");
const stream = require("stream");
const util = require("util");
const fetch = require("node-fetch");
const pipeline = util.promisify(stream.pipeline);

const INPUT = "http://localhost:8000/";
const OUTPUT = "/tmp/output_fetch.data";

const length = Math.pow(2, 32) + 1024 * 1024; // 4 Gio + 1 Mio.
fetch(INPUT + length).then(async (response) => {
    try {
        await pipeline(response.body, fs.createWriteStream(OUTPUT));
    } catch (err) {
        console.log("pipeline", err);
    }
}).catch((err) => {
    console.log("fetch", err);
});

• client_http.js

const fs = require("fs");
const http = require("http");

const INPUT = "http://localhost:8000/";
const OUTPUT = "/tmp/output_http.data";

try {
    const length = Math.pow(2, 32) + 1024 * 1024; // 4 Gio + 1 Mio.
    const request = http.request(INPUT + length, (response) => {
        console.log(`request.on("response")`);

        const file = fs.createWriteStream(OUTPUT);
        file.on("error", (err) => console.log(`file.on("error")`, err))
            .on("finish", () => console.log(`file.on("finish")`))
            .on("close", () => console.log(`file.on("close")`))
            .on("pipe", () => console.log(`file.on("pipe")`))
            .on("unpipe", () => console.log(`file.on("unpipe")`));

        response.pipe(file)
                .on("error", (err) => console.log(`response.on("error")`, err))
                .on("finish", () => console.log(`response.on("finish")`))
                .on("close", () => console.log(`response.on("close")`))
                .on("pipe", () => console.log(`response.on("pipe")`))
                .on("unpipe", () => console.log(`response.on("unpipe")`));
    });

    request.on("error", (err) => console.log(`request.on("error")`, err))
           .on("finish", () => console.log(`request.on("finish")`))
           .on("close", () => console.log(`request.on("close")`))
           .on("pipe", () => console.log(`request.on("pipe")`))
           .on("unpipe", () => console.log(`request.on("unpipe")`))
           .end();
} catch (err) {
    console.trace("catch (err)", err);
}

Steps to reproduce the behavior:

1. In a 32-bit environment.
2. node server.js
3. node client_fetch.js

Expected behavior

Running node client_fetch.js should throw an error. With node client_http.js we get the following error:

Error: Parse Error: Expected HTTP/
    at Socket.socketOnData (_http_client.js:509:22)
    at Socket.emit (events.js:314:20)
    at addChunk (_stream_readable.js:307:12)
    at readableAddChunk (_stream_readable.js:282:9)
    at Socket.Readable.push (_stream_readable.js:221:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
  bytesParsed: 16384,
  code: 'HPE_INVALID_CONSTANT',
  reason: 'Expected HTTP/',
  rawPacket: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 65433 more bytes>
}

With client_http, you can see that the error has been threw by request.on("error", ...). I think that node-fetch doesn’t listen anymore to the errors associated with the request when it has received the response.

Your Environment

software	version
node-fetch	3.0.0-beta.9 / 2.6.1
node	v14.13.0
npm	6.14.8
Operating System	OSMC 2020.06-1 (Debian 9.13)
Computer	Raspberry Pi 2 Model B Rev 1.1
Processor	ARMv7 Processor rev 5 (v7l)

Additional context

I’m not asking to fix large file downloads, but I would like node-fetch to throw an error when it happens.

[regseb]

I retested the issue in the environment:

software	version
node-fetch	3.0.0
node	v16.6.0
Computer	Raspberry Pi 2 Model B Rev 1.1
Processor	ARMv7 Processor rev 5 (v7l)

The download with client_http.js works. I think the issue https://github.com/nodejs/node/issues/37053 fixed the problem of downloading big files in Node.

So the test case client_fetch.js no longer shows the problem in node-fetch because the download works. I haven’t found any other case to test if the problem is still present in node-fetch.

[tekwiz]

@regseb @silviuburceadev Please take a look at #1064 and see if it solves this problem. You should be able to try it in your project with:

npm install 'github:tekwiz/node-fetch#fix-chunked-transfer-premature-close-2'