Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Streaming API modification proposal

See original GitHub issue

Hi, first of all, thanks for building this library. It’s awesome. With that said, I have a number of problems with the streaming API. I’m trying to build a service that is supposed to consume tweets 24/7, and the design of the API makes it really hard to handle any issues.

Twitter describes a pretty large collection of errors and reasons for shutting down a stream. The user of the API has no reasonable way to react to that. Let’s say we use the method filterStatuses. It returns a Future[TwitterStream]that returns Success as soon as the connection is established. All events are processed within a partial function. Let’s say the stream receives a DisconnectMessage - the application can only respond to that by using side effects. What is more, type of the error received impacts the retry operation that should be performed. Some errors require immediate retry, others should be handled with backoff strategy. I found about all of this after seeing that my stream has been running for days but has not been receiving any new messages.

Therefore I would like to propose one of two solutions:

  • Changing the result of streaming functions to something like ([Future[TwitterStreamHandle], Future[StreamResult]]). This would allow the end user to manually close the stream, as it is right now, as well react to errors that occur in a functional manner. I would also supply an example with proper retry strategies that would keep the stream working.
  • Better yet, I believe that the retry behaviour should be integrated into the library itself. The backoff/instant retry based on error messages will require a bit of work to get right. I don’t see a need for the end user to implement this on his own. In that case we could just add a simple flag parameter autoHandleStreamErrors that would enable the retry mechanism in case of any issues.

Please let me know what you think about this, I’m happy to implement either of these solutions.

Issue Analytics

  • State:open
  • Created 4 years ago
  • Reactions:1
  • Comments:9 (9 by maintainers)

github_iconTop GitHub Comments

lmyslinskicommented, Jun 12, 2019

#243 I’ve been messing with the code a bit. I’ve created a duplicate implementation side by side which exposed the stream to the end user. I’ve also had to simplify lots of implicit stuff just not to have to figure out what parameters are required where etc. at this point.

This implementation still misses any of the added-value stuff like retries and message filtering, but it is working. Lots of stuff is patch-worked just to make it compile at this point, but the core flow is there.

In short, the top-level functions would return Future[Source[StreamingMessage, NotUsed]]. The Future success or failure is based on establishing a connection. The end user can map upon success to start processing incoming events. All of the event mapping and wrapping in appropriate types still needs to be done.

DanielaSfregolacommented, Jun 11, 2019

A few things people have been requested that could be interesting for our discussion:

Read more comments on GitHub >

github_iconTop Results From Across the Web

Proposal: Media Capture and Streams Settings API v6 - W3C
This proposal describes additions and suggested changes to the Media Capture and Streams specification in order to support the goal of device ...
Read more >
Use Streaming API Unit | Salesforce Trailhead
Streaming API lets you push real-time notifications to clients using the publish-subscribe model. Some types of notifications are based on data changes in ......
Read more >
Proposal for a simple streaming API · Issue #154 - GitHub
This proposal is introducing new commands for data uploads/downloads. They can be implemented by the browser with the most efficient method of uploads/downloads ......
Read more >
Why and how to transform a REST API into a Data Streaming ...
We know interactivity is the key to keep our user's interest alive but we can't reduce animation to UI anymore. Twitter, Waze, Slack…...
Read more >
A Refresher on the Four Streaming APIs and What's New
Whether you work on custom integrations or build near real-time apps, the Streaming APIs are there for you. Join my session to get...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found