question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

encode stream does not proactively send data to its output

See original GitHub issue

If you create an encode stream and pipe it somewhere (for example, the network), the data you send to the stream won’t be forwarded automatically. It only seems to be sent when the server end()'s the writable stream. This makes the transform stream basically unusable for streaming realtime data.

There’s a workaround of calling stream.encoder.flush(), but that seems brittle.

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:1
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
rjeczalikcommented, Mar 2, 2018

@kawanet WebSocket is message-oriented, so msgpack.encode() will work. How to handle streaming msgpack e.g. over raw tcp socket? Like this simple echo server:

const echo = (value, enc) => {
  enc.write(value)
  enc._flush() // required, otherwise client won't get the the response
}

net.createServer(socket => {
  const enc = msgpack.createEncodeStream()
  const dec = msgpack.createDecodeStream()

  enc.pipe(socket)
  socket.pipe(dec)

  dec.on('data', value => echo(value, enc))
}).listen(3300)
0reactions
kawanetcommented, Mar 1, 2018

Right.

msgpack.createEncodeStream() instance does not manage Stream buffer for each item. One item might be splitted for multiple chunks encoded or buffered for while. Multiple items might be joined for a single chunk.

Using it with fs Stream would work great because memory copy operations reduced. Using it with net Stream may not cause trouble in some cases when chunks splitted.

I rather use the simple msgpack.encode() interface for WebSocket messaging applications.

Read more comments on GitHub >

github_iconTop Results From Across the Web

What Is Stream Output? - Alibaba Cloud Community
To sum up, the stream output refers to the process when the server returns streaming data after the client proactively initiates a request....
Read more >
Stream | Node.js v19.3.0 Documentation
Calling the writable.end() method signals that no more data will be written to the Writable . The optional chunk and encoding arguments allow...
Read more >
Consuming streaming data | Docs
Streaming API connections will be encoded using chunked transfer encoding, as indicated by the presence of a Transfer-Encoding: chunked HTTP header in the...
Read more >
Encode Stage
The Encode stage is a processing stage. It encodes a data set using a UNIX encoding command, such as gzip, that you supply....
Read more >
Write string to output stream - java
Streams ( InputStream and OutputStream ) transfer binary data. If you want to write a string to a stream, you must first convert...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found