Streams unreliable for collecting tweets from slow tweeting accounts

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn’t already filed: [open issues][open-issues]
  • Checked the [bug report guidelines][bug-report-guidelines]

Description

The streams part of twitter-v2 is unreliable. I’ve set it to monitor five twitter accounts that tweet about once every six hours.

Steps to Reproduce

1. Watch several accounts in a filtered stream that don’t post that often.
2. Start streaming tweets
3. Streams randomly quit within 30-45 minutes or less.

Expected behaviour:

Should be able to run indefinitely. twitter-v2 should not disconnect the stream just because it thinks the stream is unresponsive. The sample code by Twitter themselves does not have this behaviour.

Actual behaviour:

Streams randomly hit this._closeWithError.

    at Timeout.<anonymous> (/home/rctgamer3/project/node_modules/twitter-v2/src/TwitterStream.js:50:30)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7)

Stream timeout’s, and throws a TwitterError that the stream is unresponsive. Reproduces how often: Always, the only variable is the time until it closes the stream.

Versions

1.0.8

Additional Information

If you need any further information to triage things, let me know.

[gustawdaniel]

This is code to reproduce:

import TwitterStream from "twitter-v2/build/TwitterStream";

const Twitter = require('twitter-v2');

const client = new Twitter({
    bearer_token: process.env.TWITTER_API_BEARER_TOKEN,
});

// client.get('tweets/search/stream/rules').then(console.log);

const listenForever = async (
    streamFactory: () => TwitterStream,
    dataConsumer: (data: any) => void
): Promise<any> => {
    try {
        for await (const {data} of streamFactory()) {
            dataConsumer(data);
        }
        // The stream has been closed by Twitter. It is usually safe to reconnect.
        console.log('Stream disconnected healthily. Reconnecting.');
        return listenForever(streamFactory, dataConsumer);
    } catch (error) {
        // An error occurred so we reconnect to the stream. Note that we should
        // probably have retry logic here to prevent reconnection after a number of
        // closely timed failures (may indicate a problem that is not downstream).
        console.warn('Stream disconnected with error. Retrying.', error);
        return listenForever(streamFactory, dataConsumer);
    }
}

listenForever(
    () => client.stream('tweets/search/stream'),
    (data) => console.log(data)
).catch(console.error);

And this is results:

{
  id: '1407654882488823808',
  text: 'GET https://t.co/qnn5a0VaXe for check rules of filtering messages on twitter.'
}
Stream disconnected with error. Retrying. TwitterError: Stream unresponsive
    at Timeout.<anonymous> (/media/nvme/pro/crypto/twitter-producer/node_modules/twitter-v2/build/TwitterStream.js:43:38)
    at listOnTimeout (node:internal/timers:556:17)
    at processTimers (node:internal/timers:499:7)
Stream disconnected with error. Retrying. TwitterError: Stream unresponsive
    at Timeout.<anonymous> (/media/nvme/pro/crypto/twitter-producer/node_modules/twitter-v2/build/TwitterStream.js:43:38)
    at listOnTimeout (node:internal/timers:556:17)
    at processTimers (node:internal/timers:499:7)

First json when I posted tweet:

https://twitter.com/PreciseLabPL/status/1407654882488823808

Next error some minutes after my tweet and next error some minutes after.

There is connected topic:

https://twittercommunity.com/t/filtered-stream-disconnects-every-5-minutes-on-nodejs/153866/7

My rules:

https://api.twitter.com/2/tweets/search/stream/rules

{
    "data": [
        {
            "id": "1407650287389712386",
            "value": "from:PreciseLabPL OR from:elonmusk"
        }
    ],
    "meta": {
        "sent": "2021-06-23T11:00:28.629Z"
    }
}

[gustawdaniel]

There is connected issue:

https://github.com/twitterdev/Twitter-API-v2-sample-code/issues/34

I finally suggest write in README that this package can be used ONLY with node 12 until mentioned issue will be solved and that twitter developers are aware of this bug and works to fix it.

---

I confirming that

nvm use 12

fixed problem.