[BUG] Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal])

Describe the bug Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal])

To Reproduce Steps to reproduce the behavior:

1. npm run electron &
2. Wait about 3~5 hours
3. chia daemon exit
4. See error

Expected behavior Working fine

Screenshots

Desktop (please complete the following information):

• OS: Linux
• OS Version/Flavor: Ubuntu 20.04
• CPU: 5950X

Additional context

debug.log

  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia-miner/chia-blockchain/chia/daemon/server.py", line 182, in safe_handle
    await socket.send(response)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
    await self.ensure_open()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-04-26T22:49:52.481 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 846, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia-miner/chia-blockchain/chia/daemon/server.py", line 167, in safe_handle
    async for message in websocket:
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-04-26T22:49:52.481 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with Unknown code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia-miner/chia-blockchain/chia/daemon/server.py", line 167, in safe_handle
    async for message in websocket:
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/chia-miner/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-04-26T22:49:52.482 daemon chia.daemon.server         : INFO     Daemon WebSocketServer closed

[noisymime]

But will it affect farming in anyway?

I’ve been playing with this and as best as I can tell, the answer is Yes. When this starts occurring in the logs I see my response times grow dramatically, going from around 1000ms at most to averages of 17000ms and maximums well above 50000ms. I can’t confirm for certain that it’s impacting farming, but that seems worrying.

Just for reference, you don’t need to restart your whole machine to resolve this, you can do the following (Note the -d as you must restart the daemon itself):

chia stop -d all
chia start harvester

The issue here is in daemon/server.py and seems to be as a result of the high value used for ping_interval on the websocket (500). From everything I’ve read 500 seconds (Just over 8 minutes) is far too high and can result in these broken sockets, particularly with some NAT setups. I’ve adjusted the value to 20 and am not seeing the issue reoccurring yet.

20s seems to be a much more sensible value and will not impose any significant overhead on either the client or server.

@ArrayZoneYour I think this issue needs to be looked at more seriously. Could you please reopen this issue? (Or if not I’ll raise a new one with these findings as there still seems to be an issue).

[dorofino]

It appears to be a case of your operating system and python collectively not cleaning up after themselves, and leaving certain sockets in use after the programs terminate processes uncleanly. For the time being the only practical solution is to simply first try to restart your chia daemon and if that fails to address it then reboot your system.

Great, that makes sense. Does this affect farming?