Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Implement async Netty pipeline

See original GitHub issue


Netty API is solely callback based, and a next ChannelHandler should be added to the pipeline synchronously and immediately right after the handler complete some intermediate task (i.e. inside a ChannelHandler callback when the last event is received). Sometimes the next handler should be added even before the old handler is removed, since upon remove old handler may forward cached but unprocessed bytes toward pipeline and the next handler should already be in place to pick up those bytes. This design causes quite inconvenient UX especially when we trying to deliver another API (libp2p in our case) based on Netty architecture. This limitation can’t be workaround except with some dirty solutions like blocking ChannelHandler callbacks.

What we want

We want to add a next handler asynchronously when a previous one completes its task. For example the following code should be possible:

Future<Connection> connF = transport.dial(addr);
Future<SecureConnection> secConnF = connF
    .thenApply(conn -> SecurityHandler()));
Future<MplexedConnection> mplexConnF = secConnF
    .thenApply(secConn -> secConn.multiplex(new MultiplexHandler()));
Future<Stream> streamF = mplexConnF
streamF.thenAccept(stream -> stream.getChannel().pipeline().addLast(applicatioHandler))

Possible solution

The main reason is that default Netty pipeline just drops unprocessed events when reaching the end of pipeline. Thus we can implement pipeline which doesn’t drop events but queues them and when a new handler added replays all events to it. To prevent unbound data receiving the pipeline should setup AUTO_READ flag from the point when first unhandled data queued and the point when next handler added

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:11 (10 by maintainers)

github_iconTop GitHub Comments

jezhigginscommented, Oct 21, 2019

By finding the appropriate place for Netty to sit in our stack, and with the judicious application of new interfaces I believe we can solve, or certainly, ease the problems Anton identifies, make working with libp2p more straightforward, and make the implementation less complex and therefore more robust.

I’m going to continue on the branch linked above, and I’d welcome your comments, suggestions, or alternative proposals. Thanks.

jezhigginscommented, Oct 22, 2019

Thanks @shahankhatch. I think you’ve captured what I’m trying to say very well. We should be trying to find the appropriate abstractions at each level, making the common cases as straightforward as we can, without precluding the difficult ones.

I’ll continue exploring on the refactor/push-down-netty branch, with a view to raising a PR against the develop branch when that’s settled down. Might help us to have actual examples to discuss.

Equally, I’m open to alternative approaches to explore or particular places in the code it might be worth paying particular attention to.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to use initPipeline method in org.asynchttpclient.netty ...
Parsing JSON documents to java classes using gson · setScale (BigDecimal) · onRequestPermissionsResult (Fragment) · getOriginalFilename (MultipartFile). Return the ...
Read more >
HTTP Server with Netty - Baeldung
In this tutorial, we're going to implement a simple upper-casing server over HTTP with Netty, an asynchronous framework that gives us the ...
Read more >
ChannelPipeline (Netty API Reference (4.0.56.Final))
ChannelPipeline implements an advanced form of the Intercepting Filter pattern to ... If your business logic is fully asynchronous or finished very quickly, ......
Read more >
Netty - Skip the rest of the handlers in the pipeline
I tried using and ctx.fireChannelInactive() to no avail. It seems to still pass the request onto the next handler (HttpObjectAggregator in ...
Read more >
"Netty - The async event-driven network application framework ...
Netty - The async event-driven network application framework The internet and network is getting more important with every single day.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found