[Bug] Ubuntu harvester terminates itself after any disruption

What happened?

I have two remote harvesters, and if I restart my farmer, temporarily lose network connection, use a different Ethernet port, etc., one of them (Ubuntu) goes offline and disappears from my farmer’s remote harvesters section. The other one (Windows) stays on and reconnects without any issues. Both are on same network, connected to the same icx6610 switch, etc. This has been going on the time I updated everything to 1.3.5 or so and has yet to resolve even now on 1.5.1. I have had this continue to occur after installing the gui client and cli-only client and using different installation methods. Everything runs just fine asides from the permanent termination of the harvester. Only solution is to manually restart the harvester.

Version

1.5.1, 1.6.0

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2022-09-02T19:12:36.058 harvester chia.harvester.harvester: INFO     16 plots were eligible for farming 7f7d2a6216... Found 0 proofs. Time: 0.47556 s. Total 6171 plots
2022-09-02T19:12:40.273 harvester harvester_server        : INFO     Connection closed: 192.168.1.22, node id: b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96
2022-09-02T19:12:40.274 harvester chia.harvester.harvester: INFO     peer disconnected {'host': '192.168.1.22’, 'port': 8447}
2022-09-02T19:13:23.045 daemon chia.daemon.server         : ERROR    Ping error
Traceback (most recent call last):
  File "/home/user/chia-blockchain/chia/daemon/server.py", line 275, in ping_task
    await socket.ping()
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/web_ws.py", line 295, in ping
    await self._writer.ping(message)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 676, in ping
    await self._send_frame(message, WSMsgType.PING)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 653, in _send_frame
    self._write(header + message)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 663, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2022-09-02T19:13:23.047 daemon chia.daemon.server         : ERROR    Ping failed, connection closed. 

<the above is the last thing that was logged after I had restarted the farmer. Below is what was logged after restarting the harvester> 

2022-09-02T19:16:05.489 daemon chia.daemon.server         : INFO     sending term signal to chia_harvester
2022-09-02T19:16:20.512 daemon chia.daemon.server         : INFO     sending kill signal to chia_harvester
2022-09-02T19:16:20.959 daemon chia.daemon.server         : INFO     process chia_harvester returned -9
2022-09-02T19:16:20.961 daemon chia.daemon.server         : INFO     Daemon Server stopping, Services stopped: ['chia_harvester']
2022-09-02T19:16:20.964 daemon chia.daemon.server         : INFO     chia daemon exiting
2022-09-02T19:16:20.965 daemon chia.daemon.server         : INFO     Daemon WebSocketServer closed 
2022-09-02T19:16:23.479 daemon chia.daemon.server         : INFO     chia-blockchain version: 1.5.1
2022-09-02T19:16:23.532 daemon chia.daemon.server         : INFO     Starting Daemon Server
2022-09-02T19:16:25.528 daemon chia.daemon.server         : INFO     ConnectionClosed. Closing websocket with Unknown
2022-09-02T19:16:26.242 harvester chia.harvester.harvester: INFO     Using plots_refresh_parameter: {'batch_size': 300, 'batch_sleep_milliseconds': 1, 'interval_seconds': 120, 'retry_invalid_seconds': 1200}
2022-09-02T19:16:26.242 harvester harvester               : INFO     chia-blockchain version: 1.5.1
2022-09-02T19:16:26.243 harvester harvester_server        : INFO     Service capabilities: [(1, '1'), (2, '1'), (3, '1')]
2022-09-02T19:16:26.245 harvester harvester               : INFO     Started harvester service on network_id: mainnet
2022-09-02T19:16:26.245 harvester harvester               : INFO     Reconnecting to peer {'host': '192.168.1.22', 'port': 8447}
2022-09-02T19:16:26.270 harvester harvester_server        : INFO     Connected with farmer {'host': '192.168.1.22', 'port': 8447}
2022-09-02T19:16:26.284 harvester chia.plotting.cache     : INFO     Loaded 970540 bytes of cached data
2022-09-02T19:16:26.317 daemon chia.daemon.server         : INFO     Register service {'service': 'chia_harvester'}
2022-09-02T19:16:26.318 daemon chia.daemon.server         : INFO     registered for service chia_harvester
2022-09-02T19:16:26.319 daemon chia.daemon.server         : INFO     {'success': True}
2022-09-02T19:16:26.604 harvester chia.plotting.manager   : INFO     Found plot /media/chia/1/…..

[benjfranklin100]

FYI, testing this possible solution from 13403 . Looks ok so far

Doesn’t seem to be an incongruence in python version, all machines are on 3.10.4 I’m not entirely familiar with the process of safely rolling back python. I know completely removing python just absolutely shreds the system, however, I’m getting mixed answers regarding rolling it back and most posts I see recommend doing so in a virtual environment.

From what I saw in the other thread, the UI is built with 3.9 – I ended up dumping the UI and using CLI for the actual farmer and turned the UI farmer into a wallet. All good 24h now. I still see it failing pings but it just continues on.

How’s yours running? Hopefully still well? I noticed another user was having the same traceback after farmer restart/disconnection in #13480

[benjfranklin100]

Anyone here able to send me a full debug.log with this issue and DEBUG log level while #13508 is running on the harvester + the log from the farmer? Just send it on keybase http://keybase.io/dustinface

I can do this within the next new days.