Error connecting to CertStream - Connection is already closed

I implemented this tool, following the given examples. I found out that sometimes ( usally every 2 minutes, more or less) the connection drops and it takes almost a minute to reconnect to the stream.

I don’t know why this happens and this could probably lead to missing some hits i’m looking for.

Here’s an example of log ( integrated with python logging module )

[ERROR:certstream] 2020-09-02 16:20:31,002 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:20:36,097 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:23:10,796 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:23:15,908 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:25:11,015 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:25:16,182 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:27:17,051 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:27:22,194 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:29:01,343 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:29:06,522 - Connection established to CertStream! Listening for events...

Any suggestion about how to fix this?

[mikenor]

Argh, well this is disappointing. 😦 I am hitting this problem too and can only reach the same conclusion as @kres20 that the certstream.calidog.io service is just basically unusable.

I noticed certstream.calidog.io is actually Cloudflare. In Wireshark packet capture on my client, I can see the Cloudflare frontend server is just suddenly closing the TCP connection for some reason. No shutting down of the Websocket session or the TLS session, but a (mostly) normal 4-way close handshake of the outer TCP connection. (Abnormally, the Cloudflare server waits for exactly 10 seconds before sending its very last ACK, despite my client dutifully attempting several retransmissions of its last FIN that the server is waiting for.)

When I add sslopt={'suppress_ragged_eofs': False} to the certstream.listen_for_events parameters, the issue causes an exception that matches what I saw in packet capture: ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2633).

Traceback (most recent call last):
  File "C:\Users\m\dev\watchcertstream\watchcertstream.py", line 72, in <module>
    main()
  File "C:\Users\m\dev\watchcertstream\watchcertstream.py", line 68, in main
    certstream.listen_for_events(certstream_message_callback, url='wss://certstream.calidog.io/', on_error=certstream_error_callback, sslopt={"suppress_ragged_eofs": False})
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 54, in listen_for_events
    c.run_forever(ping_interval=15, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 371, in run_forever
    self._callback(self.on_error, e)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 412, in _callback
    self.on_error(self, e)
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 47, in _on_error
    self.on_error_handler(ex)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 407, in _callback
    callback(self, *args)
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 47, in _on_error
    self.on_error_handler(ex)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 369, in run_forever
    dispatcher.read(self.sock.sock, read, check)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 70, in read
    if not read_callback():
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 335, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 396, in recv_data_frame
    frame = self.recv_frame()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 435, in recv_frame
    return self.frame_buffer.recv_frame()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_abnf.py", line 351, in recv_frame
    payload = self.recv_strict(length)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_abnf.py", line 372, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 519, in _recv
    return recv(self.sock, bufsize)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_socket.py", line 113, in recv
    bytes_ = _recv()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_socket.py", line 90, in _recv
    return sock.recv(bufsize)
  File "C:\Program Files\Python39\lib\ssl.py", line 1226, in recv
    return self.read(buflen)
  File "C:\Program Files\Python39\lib\ssl.py", line 1101, in read
    return self._sslobj.read(len)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2633)

When I use my preferred public DNS resolver (149.112.121.10), certstream.calidog.io resolves to 172.64.80.1 for me. This is a Cloudflare IP. Traceroute from my client and from a couple other places suggests that it’s an anycast IP that exists at/near multiple different Internet exchanges.

When I use other public DNS resolvers, certstream.calidog.io resolves to different Cloudflare IPs for me - 172.67.151.243 and 104.21.40.132. When I force my client to connect to these IPs, the problem still happens, but anecdotally seems to occur far less frequently than with 172.64.80.1.

I’m skeptical that IP anycast is a suitable CDN strategy for services requiring permanent TCP connections, such as the Calidog Certstream service. IMO this problem is likely a completely normal side-effect of the chosen CDN strategy that the Calidog service is using. 😦

[kres20]

@katesaikishore @Fitblip I set up the certstream server too (https://github.com/CaliDog/certstream-server), and connect there like certstream.listen_for_events(print_callback, url='ws://localhost:4000/'), and it works fine without any errors. Therefore this above error is caused by certstream.calidog.io.