Streaming API modification proposal

Hi, first of all, thanks for building this library. It’s awesome. With that said, I have a number of problems with the streaming API. I’m trying to build a service that is supposed to consume tweets 24/7, and the design of the API makes it really hard to handle any issues.

Twitter describes a pretty large collection of errors and reasons for shutting down a stream. The user of the API has no reasonable way to react to that. Let’s say we use the method filterStatuses. It returns a Future[TwitterStream]that returns Success as soon as the connection is established. All events are processed within a partial function. Let’s say the stream receives a DisconnectMessage - the application can only respond to that by using side effects. What is more, type of the error received impacts the retry operation that should be performed. Some errors require immediate retry, others should be handled with backoff strategy. I found about all of this after seeing that my stream has been running for days but has not been receiving any new messages.

Therefore I would like to propose one of two solutions:

• Changing the result of streaming functions to something like ([Future[TwitterStreamHandle], Future[StreamResult]]). This would allow the end user to manually close the stream, as it is right now, as well react to errors that occur in a functional manner. I would also supply an example with proper retry strategies that would keep the stream working.
• Better yet, I believe that the retry behaviour should be integrated into the library itself. The backoff/instant retry based on error messages will require a bit of work to get right. I don’t see a need for the end user to implement this on his own. In that case we could just add a simple flag parameter autoHandleStreamErrors that would enable the retry mechanism in case of any issues.

Please let me know what you think about this, I’m happy to implement either of these solutions.

[lmyslinski]

#243 I’ve been messing with the code a bit. I’ve created a duplicate implementation side by side which exposed the stream to the end user. I’ve also had to simplify lots of implicit stuff just not to have to figure out what parameters are required where etc. at this point.

This implementation still misses any of the added-value stuff like retries and message filtering, but it is working. Lots of stuff is patch-worked just to make it compile at this point, but the core flow is there.

In short, the top-level functions would return Future[Source[StreamingMessage, NotUsed]]. The Future success or failure is based on establishing a connection. The end user can map upon success to start processing incoming events. All of the event mapping and wrapping in appropriate types still needs to be done.

[DanielaSfregola]

A few things people have been requested that could be interesting for our discussion:

• A user may want to handle stream unrelated errors differently (see https://github.com/DanielaSfregola/twitter4s/issues/142)
• A user may want to “listen” for a stream closure (see https://github.com/DanielaSfregola/twitter4s/issues/144)
• A user may want to access the stream directly (see https://github.com/DanielaSfregola/twitter4s/issues/168)
• I also have people requesting if it was possible to inject their own streaming library of choosing (for example fs2 rather than akka-streams)