Python and Javascript stream examples break after 5 minutes

Describe the bug Both Python and Javascript example code for streaming break after 5 minutes of retrieving tweets. Javascript just hangs/ends without message, Python throws the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 543, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 598, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 547, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 432, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 626, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 320, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "twitter-sample-python.py", line 93, in <module>
    main()
  File "twitter-sample-python.py", line 89, in main
    get_stream(headers, set, bearer_token)
  File "twitter-sample-python.py", line 76, in get_stream
    for response_line in response.iter_lines():
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 797, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

To Reproduce Download example code, set bearer token, run example code. Each time the code is executed, it will stop receiving tweets after the same amount of time, i.e. ~5 minutes. I could reproduce this on two different machines, os (ubuntu+osx), twitter accounts, and three different internet connections.

Expected behavior Stream should not stop.

[andypiper]

Thanks for all the information here and the workarounds to reconnect. We’ll see what we can do to integrate those changes into future versions of the sample code as needed.

[Da1ne]

Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn’t receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.

try:
    for response_line in response.iter_lines():
        if response_line:
            json_response = json.loads(response_line)
            print(json_response)
except requests.exceptions.ChunkedEncodingError:
    encoding_count += 1

A crude patch until Twitter Devs figure out what’s happening on their end?