Do not call ".end()" in the callback of ".write"

Hey 👋

I’ve got a question about the way this library calls the original .end() method on the ClientRequest. It does so in the callback of .write() (given there is data to send so that the callback gets called):

https://github.com/follow-redirects/follow-redirects/blob/13136e95bbe23cabbeaeb74bd0c933aa98dd9b96/index.js#L127-L130

I think I understand the intention here: send this chunk, when done (regardless of whether it’s successful or not—because you’re not checking the error callback argument) mark the request as ready.

Question

Can we execute .write and .end in parallel?

this.write(data, encoding, function () {
  self._ended = true;
});

// This call is moved out of the write's callback.
currentRequest.end(null, null, callback);

this._ending = true;

Context

I’m a maintainer of the API mocking library called Mock Service Worker. We intercept requests in Node.js using the @mswjs/interceptors library, which augments the native ClientRequest class. We use class augmentation instead of stubbing it to allow as much of the Node.js default behavior to execute without having to polyfill it. Since we are an API mocking tool, we cannot really write request body chunks until we know there’s (or isn’t) a request handler for this particular request. That’s why we only collect body chunks in the .write() method:

https://github.com/mswjs/interceptors/blob/219f9f3cd82f5baf3297f7c6095caa356c148d31/src/interceptors/ClientRequest/NodeClientRequest.ts#L85-L97

We then write those chunks only for passthrough requests when the request handler is known. That is possible to check only in the .end method:

https://github.com/mswjs/interceptors/blob/219f9f3cd82f5baf3297f7c6095caa356c148d31/src/interceptors/ClientRequest/NodeClientRequest.ts#L170-L179

Now since RedirectableRequest only calls .end in the callback of .write, when our library is applied the .end is never called. That happens because we do not call the .write callbacks immediately (not semantic, no chunks are being written just yet so it feels wrong to confirm the result by calling a callback), but instead are referred to the .end. Note that this difference is purely an implementation detail which should have no effect on the end-consumer. For them, the expected result is intercepted request and mocked response.

• The original reported issue: https://github.com/mswjs/msw/issues/1125

Common usage of .end

I’ve also noticed that commonly the .write and .end are called in parallel:

const req = http.get()
req.write('hello world')
req.end()

That is the way to send request body chunk and end a request in about any example I can find. I’ve never seen the write(chunk, () => this.end()) usage before but I understand it must have reasons behind it.

What I’d like to know

1. What are the reasons to call .end only in the callback of .write?
2. What are the implications of calling these two methods in parallel?
3. Do you have any recommendations on how to handle your flow in our “interceptors” library?

Thank you for looking into this.

[RubenVerborgh]

• Write callback(s) have already been called. Now it’s either we call them again upon the actual chunk write, or ignore them entirely to prevent duplicate calls.

You’ve fulfilled the contract; the write callbacks have been called.

• If we choose to ignore them, we don’t know whether writing them to the actual socket connection succeeded (the server might’ve rejected a chunk). And it’s only in the .write() call where we are exposed the error argument of the callback to know that.

There will be no errors here, because none occur at that stage. Any errors will be emitted as an error event on the stream or through the end callback, and that’s fine.

[RubenVerborgh]

Do you see any potential issues if we call write callbacks while the request hasn’t actually written anything yet?

No; from the perspective of the direct consumer, it is written: it’s just that your writing is a memory buffer. It’s success, so the callback should be invoked.

What matters is finding the proper solution

I think both work, but I think changing the interceptor is the conceptually correct and safest one.