clone() hangs with large response in Node

I hit this bug today, it was caused by https://github.com/bitinn/node-fetch/issues/386

The bug makes ky-universal not useable at all in node. (You don’t know when the code will hang as you don’t know the response size ahead of time)

Is there a way to remove clone() from the source code?

---

IssueHunt Summary

xxczaki has been rewarded.

Backers (Total: $80.00)

• issuehunt ($80.00)

Submitted pull Requests

• #19 Use node-fetch v3 beta

---

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[xxczaki]

@gutenye @sindresorhus In version 3.0.0 of node-fetch you will be able to manually set highWaterMark and prevent this issue from happening 😄

fetch(url, {highWaterMark: 10}).then(res => res.clone().buffer());

See this PR

[sholladay]

It would be pretty challenging to re-implement .clone() ourselves. In doing so, we would probably end up creating more new bugs than we solved. The only practical way forward that I see here would be to just not clone the response at all.

We currently clone in three places:

1. When calling afterResponse hooks
2. When streaming the response to provide progress events
3. When body method shortcuts are used (e.g. ky().json())

I’m not sure which, if any, of these places would be safe to remove. None of them seem particularly important to me, though I like having them for safety.

It’s worth noting that we only call .clone() if you use one of the above mentioned features. So you can actually avoid this problem if you can live without those features.

For example, if .clone() is being called because you are using the .json() method like this:

const users = await ky.get('users').json();

It’s perfectly fine to re-write that to:

const response = await ky.get('users');
const users = await response.json();

… in which case .clone() is not called. The only caveat, beyond it being more verbose, is that you will also have to set the Accept: application/json header yourself, if you need that.

Other than those workarounds, I’m not sure there’s much we can do, since we are using the API correctly and this is really an upstream issue. We could explore alternative fetch polyfills, but I suspect that would again introduce its own set of problems. Let’s keep an eye on that issue you referenced.

In the meantime, should we consider removing .clone() internally and, if so, where? Removing 3 seems the safest to me and it’s probably the most common case as well.