Streaming API modification proposal
See original GitHub issueHi, first of all, thanks for building this library. It’s awesome. With that said, I have a number of problems with the streaming API. I’m trying to build a service that is supposed to consume tweets 24/7, and the design of the API makes it really hard to handle any issues.
Twitter describes a pretty large collection of errors and reasons for shutting down a stream. The user of the API has no reasonable way to react to that. Let’s say we use the method filterStatuses
. It returns a Future[TwitterStream]
that returns Success
as soon as the connection is established. All events are processed within a partial function. Let’s say the stream receives a DisconnectMessage
- the application can only respond to that by using side effects. What is more, type of the error received impacts the retry operation that should be performed. Some errors require immediate retry, others should be handled with backoff strategy. I found about all of this after seeing that my stream has been running for days but has not been receiving any new messages.
Therefore I would like to propose one of two solutions:
- Changing the result of streaming functions to something like
([Future[TwitterStreamHandle], Future[StreamResult]])
. This would allow the end user to manually close the stream, as it is right now, as well react to errors that occur in a functional manner. I would also supply an example with proper retry strategies that would keep the stream working. - Better yet, I believe that the retry behaviour should be integrated into the library itself. The backoff/instant retry based on error messages will require a bit of work to get right. I don’t see a need for the end user to implement this on his own. In that case we could just add a simple flag parameter
autoHandleStreamErrors
that would enable the retry mechanism in case of any issues.
Please let me know what you think about this, I’m happy to implement either of these solutions.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:9 (9 by maintainers)
#243 I’ve been messing with the code a bit. I’ve created a duplicate implementation side by side which exposed the stream to the end user. I’ve also had to simplify lots of implicit stuff just not to have to figure out what parameters are required where etc. at this point.
This implementation still misses any of the added-value stuff like retries and message filtering, but it is working. Lots of stuff is patch-worked just to make it compile at this point, but the core flow is there.
In short, the top-level functions would return
Future[Source[StreamingMessage, NotUsed]]
. TheFuture
success or failure is based on establishing a connection. The end user can map upon success to start processing incoming events. All of the event mapping and wrapping in appropriate types still needs to be done.A few things people have been requested that could be interesting for our discussion: