Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
[Bug] After upgrade from 1.3.3 to 1.3.4 farmer stopped farming tonight

See original GitHub issue
What happened?

Yesterday I upgraded to version 1.3.4 this night the farmer stopped farming for no reason. My log level is only set to warning.
Version

1.3.4.
What platform are you using?

Linux
What ui mode are you using?

CLI
Relevant log output

2022-04-21T00:04:38.006 full_node full_node_server        : WARNING  Cannot write to closing transport 2.147.124.25
2022-04-21T00:13:00.104 full_node full_node_server        : WARNING  Cannot write to closing transport 46.188.45.186
2022-04-21T00:14:43.485 full_node chia.full_node.full_node: WARNING  Block validation time: 2.32 seconds, pre_validation time: 1.70 seconds, cost: 5414972420, percent full: 49.227% header_hash: 9b18f05967698d6dc5eff3cbbb342c5cdc3fc88d9fcbf5d6c94d762254495c2f height: 1865937
2022-04-21T00:17:00.222 full_node chia.full_node.full_node: WARNING  Block validation time: 2.36 seconds, pre_validation time: 1.66 seconds, cost: 4416469522, percent full: 40.15% header_hash: 24e7cae97483f667e7854e45a8567c4d902a21b24c178efc868b94051e1992aa height: 1865945
2022-04-21T00:19:55.385 full_node chia.full_node.full_node: WARNING  Block validation time: 2.46 seconds, pre_validation time: 0.24 seconds, cost: 3778045444, percent full: 34.346% header_hash: b9ad4c26ee0ef38ee139bb57f440ee777d954fe0d99137f605076b8de3b2d75e height: 1865956
2022-04-21T00:25:46.356 full_node chia.full_node.full_node: WARNING  Block validation time: 2.50 seconds, pre_validation time: 0.33 seconds, cost: 5002773628, percent full: 45.48% header_hash: 17f4dd037ce3d5351fd63d6d4d46a02734ca025b712cc46b1ca06e2a48a086c6 height: 1865980
2022-04-21T01:06:38.282 full_node chia.full_node.full_node: WARNING  Block validation time: 2.49 seconds, pre_validation time: 0.28 seconds, cost: 3980750271, percent full: 36.189% header_hash: e69fd91dcfd3ecf79e326276c442336b7c4698726ff1dd98c08b465209344e0b height: 1866101
2022-04-21T01:28:00.821 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 114.82.34.221
2022-04-21T01:40:26.062 full_node chia.full_node.full_node: WARNING  Block validation time: 2.54 seconds, pre_validation time: 0.23 seconds, cost: 5494610841, percent full: 49.951% header_hash: 47019366d64728e65e33947a84c106dc5bd5664834332a0fe5d93bb8a4f586e2 height: 1866200
2022-04-21T01:59:21.675 full_node chia.full_node.full_node: WARNING  Block validation time: 2.63 seconds, pre_validation time: 1.70 seconds, cost: 5295982522, percent full: 48.145% header_hash: 0e23b6c3a4da256e0ae10dc500c96bafcd1a89a3e9db27e9c953fce06683a38f height: 1866249
2022-04-21T02:09:50.468 full_node chia.full_node.full_node: WARNING  Block validation time: 2.41 seconds, pre_validation time: 0.29 seconds, cost: 4126188246, percent full: 37.511% header_hash: 1c876b7cb3ee5d8f96d98743499b49337ac1590dae1e74b20dc79131d43dd112 height: 1866272
2022-04-21T02:18:56.828 full_node full_node_server        : WARNING  Cannot write to closing transport 188.71.200.162
2022-04-21T02:38:15.388 full_node chia.full_node.full_node: WARNING  Block validation time: 2.79 seconds, pre_validation time: 0.28 seconds, cost: 5489709176, percent full: 49.906% header_hash: 359d696c7c87fd334c6deebad028651d63fd18f98d9043119da785a2825d721b height: 1866362
2022-04-21T02:58:37.004 full_node chia.full_node.full_node: WARNING  Block validation time: 2.96 seconds, pre_validation time: 0.24 seconds, cost: 5296409820, percent full: 48.149% header_hash: fd083023ee73b18be22a5713c028f5ef3afc2932c3f75c240d8fc7b5a858f529 height: 1866411
2022-04-21T03:06:03.988 full_node chia.full_node.full_node: WARNING  Block validation time: 2.92 seconds, pre_validation time: 0.23 seconds, cost: 5493410295, percent full: 49.94% header_hash: c891e1fb3fe8857024db927639a71ab8d5855dfaf510fdad82a2155d5cedaa89 height: 1866433
2022-04-21T03:08:30.626 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 114.82.34.221
2022-04-21T03:15:43.746 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 175.160.137.88
2022-04-21T03:20:49.098 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 139.180.28.206
2022-04-21T03:25:56.435 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 61.160.221.250
2022-04-21T03:49:10.578 full_node chia.full_node.full_node: WARNING  Block validation time: 2.75 seconds, pre_validation time: 0.27 seconds, cost: 5472037770, percent full: 49.746% header_hash: 8c6f2082ed1af053f3b1ff62580c8105ab0a1928a05f3c502f7df016685ea1c0 height: 1866555
2022-04-21T04:23:08.384 full_node chia.full_node.full_node: WARNING  Block validation time: 3.21 seconds, pre_validation time: 0.36 seconds, cost: 5445020476, percent full: 49.5% header_hash: 8bafc7f1e9180aaa8228d6dd785ff0a5fde4da2f7124ca2d8d1985338d65e814 height: 1866660
2022-04-21T04:50:40.361 full_node chia.full_node.full_node: WARNING  Block validation time: 2.18 seconds, pre_validation time: 0.20 seconds, cost: 4864816111, percent full: 44.226% header_hash: cfa440682913dc0b2da85e72868c943c02e2fecd17c97aa77d70ebb4ef521bc6 height: 1866763
2022-04-21T04:53:30.341 full_node chia.full_node.full_node: WARNING  Block validation time: 2.10 seconds, pre_validation time: 0.20 seconds, cost: 0, percent full: 0.0% header_hash: c83d49d82d3ccd8ef9d7ede27bd523609fcc7e7bda99e9cbf8d52260c1610ba6 height: 1866773
2022-04-21T04:57:44.145 full_node chia.full_node.full_node: WARNING  Block validation time: 2.37 seconds, pre_validation time: 0.22 seconds, cost: 5257300139, percent full: 47.794% header_hash: 0376c6b543fa91c714ea095990440e35a4addba89a5da00254901af29671d76d height: 1866790
2022-04-21T05:26:20.897 full_node chia.full_node.full_node: WARNING  Block validation time: 2.59 seconds, pre_validation time: 0.27 seconds, cost: 5457471055, percent full: 49.613% header_hash: dcf7f0fcadd8082b70f3e1e714fac6ebccbc5cdae0b0cfd34b8821021144857c height: 1866875
2022-04-21T05:29:41.738 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 101.80.168.204
2022-04-21T05:29:54.282 full_node chia.full_node.full_node: WARNING  Block validation time: 2.72 seconds, pre_validation time: 0.22 seconds, cost: 3667539152, percent full: 33.341% header_hash: f11406bd30404bc426028b10f18ef207f2ae09abd41e38e47dd3ecebb3a6f40d height: 1866887
2022-04-21T05:37:22.270 full_node chia.full_node.full_node: WARNING  Block validation time: 2.91 seconds, pre_validation time: 0.25 seconds, cost: 5412807643, percent full: 49.207% header_hash: 73bc2e1ca739d1e81c810527c5164777e6509e8971080a4e11563f9134e858ee height: 1866907
2022-04-21T05:40:37.484 full_node chia.full_node.full_node: WARNING  Block validation time: 2.07 seconds, pre_validation time: 0.28 seconds, cost: 3839406926, percent full: 34.904% header_hash: e4f26a89140cb20fc4bf3273f5e3e8866df3ad63a15fac5503997c2694aea4f1 height: 1866921
2022-04-21T05:42:25.246 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 43.154.42.182
2022-04-21T05:55:48.938 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 58.20.75.229
2022-04-21T05:57:09.580 full_node chia.full_node.full_node: WARNING  Block validation time: 2.31 seconds, pre_validation time: 0.21 seconds, cost: 3724271466, percent full: 33.857% header_hash: 37f7bfc0bfbabf50cc2524084f8bf6a7b06533eb74f806d7628aeb1846f3fd89 height: 1866978
2022-04-21T06:13:11.799 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 218.31.18.120
2022-04-21T06:19:28.165 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 201.140.15.11
2022-04-21T06:20:35.225 full_node full_node_server        : WARNING  Cannot write to closing transport 95.24.12.44
2022-04-21T06:40:58.210 full_node chia.full_node.full_node: WARNING  Block validation time: 2.38 seconds, pre_validation time: 0.32 seconds, cost: 3707202103, percent full: 33.702% header_hash: e18320b9c1e564c7e7eceddb92a10cfbed7fa1a302832af4481cb8c21ccf89f6 height: 1867131
2022-04-21T06:42:28.332 full_node chia.full_node.full_node: WARNING  Block validation time: 2.50 seconds, pre_validation time: 0.30 seconds, cost: 5421815399, percent full: 49.289% header_hash: a03b0665c8c7c83fe4f71b2c51510fce74ff416c564f5baa19ce7ca5c28b33da height: 1867138
2022-04-21T06:42:36.936 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 82.157.26.106
2022-04-21T06:58:14.054 full_node chia.full_node.full_node: WARNING  Block validation time: 2.28 seconds, pre_validation time: 0.21 seconds, cost: 5205992869, percent full: 47.327% header_hash: ee84f6dde329bd8ae760fec1b969594f758d20b54faa3e23cd5e896458bf16c5 height: 1867186
2022-04-21T07:05:30.505 full_node chia.full_node.full_node: WARNING  Block validation time: 2.23 seconds, pre_validation time: 0.20 seconds, cost: 3836151399, percent full: 34.874% header_hash: 86b55af43aacf963e7f84778b36dc67c85f3e81e2e0ee0944eb1a0bf50486c4e height: 1867205
2022-04-21T07:25:35.102 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 119.147.177.1
2022-04-21T07:58:43.252 full_node full_node_server        : WARNING  Banning 102.65.92.56 for 10 seconds
2022-04-21T07:58:43.320 full_node full_node_server        : WARNING  Invalid handshake with peer. Maybe the peer is running old software.
2022-04-21T08:00:31.339 full_node chia.full_node.full_node: WARNING  Block validation time: 2.95 seconds, pre_validation time: 0.81 seconds, cost: 3922141747, percent full: 35.656% header_hash: 0140978bcdb2c7544df2905ec159d1f7b5344bdf32c33eee19e32284493fd9b0 height: 1867389
2022-04-21T08:04:01.486 full_node chia.full_node.full_node: WARNING  Block validation time: 2.35 seconds, pre_validation time: 0.20 seconds, cost: 5450242778, percent full: 49.548% header_hash: ad7ccad4e62b72153597b564d2bcf6e54c14c72286c37700511149aea60f6373 height: 1867398
2022-04-21T08:07:12.029 full_node chia.full_node.full_node: WARNING  Block validation time: 2.69 seconds, pre_validation time: 0.21 seconds, cost: 0, percent full: 0.0% header_hash: a880bfd56e84262916bf295dc99cce47b6b9775af800772b5b4f504332df42aa height: 1867414
2022-04-21T08:10:24.133 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 221.197.94.70
2022-04-21T08:13:54.350 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 122.237.77.248
2022-04-21T08:13:57.085 full_node chia.full_node.full_node: WARNING  Block validation time: 2.46 seconds, pre_validation time: 0.22 seconds, cost: 5474311545, percent full: 49.766% header_hash: 453dbea280ca6b95bf4c176c1a1fffb0d95942e33087cce9c426c357437004d8 height: 1867446
2022-04-21T08:15:52.620 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 183.159.67.29
2022-04-21T08:20:09.041 full_node full_node_server        : WARNING  Cannot write to closing transport 85.140.9.37
2022-04-21T08:29:59.635 full_node chia.full_node.full_node: WARNING  Block validation time: 2.28 seconds, pre_validation time: 0.38 seconds, cost: 3914776039, percent full: 35.589% header_hash: 769088ddbaf3ff3f8656fa10f55218ecd9ead4b2270f8faaf60f8d3b9e5be624 height: 1867489
2022-04-21T08:35:47.922 full_node full_node_server        : ERROR    Exception: [Errno 110] Connection timed out with 211.155.228.10
2022-04-21T08:35:47.922 full_node full_node_server        : ERROR    Exception Stack: Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/server/ws_connection.py", line 249, in outbound_handler
    await self._send_message(msg)
  File "/home/chia/chia-blockchain/chia/server/ws_connection.py", line 402, in _send_message
    await self.ws.send_bytes(encoded)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/web_ws.py", line 307, in send_bytes
    await self._writer.send(data, binary=True, compress=compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 685, in send
    await self._send_frame(message, WSMsgType.BINARY, compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 656, in _send_frame
    await self.protocol._drain_helper()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
    await waiter
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 935, in _write_ready
    n = self._sock.send(self._buffer)
TimeoutError: [Errno 110] Connection timed out

2022-04-21T08:36:47.107 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 175.4.212.100
2022-04-21T08:52:40.588 full_node chia.full_node.full_node: WARNING  Block validation time: 2.61 seconds, pre_validation time: 0.21 seconds, cost: 3898499917, percent full: 35.441% header_hash: 1b4a26e7872842388725bf485354e6ae2a5a78ff3bae9d69bce71c2f53627ce4 height: 1867552
2022-04-21T08:53:43.245 full_node chia.full_node.full_node: WARNING  Block validation time: 2.59 seconds, pre_validation time: 0.21 seconds, cost: 4396690324, percent full: 39.97% header_hash: 06ab85d11a2294a73f12f8957154f029d4f445b8b1a5326ebd655f882da7dfae height: 1867555
2022-04-21T08:58:38.420 full_node chia.full_node.full_node: WARNING  Block validation time: 2.12 seconds, pre_validation time: 0.18 seconds, cost: None header_hash: 86670845264e8ba933b062097cf40412b800db95cf85f1715fc2a2bed822a13f height: 1867570
2022-04-21T09:00:12.503 full_node full_node_server        : WARNING  Cannot write to closing transport 216.147.124.13
2022-04-21T09:24:07.079 full_node chia.full_node.full_node: WARNING  Block validation time: 2.44 seconds, pre_validation time: 0.28 seconds, cost: 5256684781, percent full: 47.788% header_hash: 849e541087eb0413064d7d069fd213157230765b52ce1277facd07c7f68b28bc height: 1867644
2022-04-21T09:27:45.347 full_node chia.full_node.full_node: WARNING  Block validation time: 2.55 seconds, pre_validation time: 0.22 seconds, cost: 4175687322, percent full: 37.961% header_hash: c3e3f6b49d980fdee81d742b09f5d4923499e31063cb896db5b63efd5d73011b height: 1867658
2022-04-21T09:58:30.148 full_node chia.full_node.full_node: WARNING  Block validation time: 2.61 seconds, pre_validation time: 1.93 seconds, cost: 3898816903, percent full: 35.444% header_hash: 47dfef92ab6f5144225127bafe7f47c98ad37421ecb2d8159e53577d4b8a7de4 height: 1867744
2022-04-21T10:12:43.303 full_node chia.full_node.full_node: WARNING  Block validation time: 2.56 seconds, pre_validation time: 0.24 seconds, cost: 5408100085, percent full: 49.165% header_hash: 775dd992050b99f244a81093a3f7bed1ab8269e83962988b3509c506bb5272cf height: 1867781
2022-04-21T10:16:01.165 full_node chia.full_node.full_node: WARNING  Block validation time: 2.38 seconds, pre_validation time: 0.23 seconds, cost: 3802767490, percent full: 34.571% header_hash: b972b0c6d6938a5b028afee7d07ca9fb5477e325a88521c1d696e7e174ea0ebf height: 1867794
2022-04-21T10:16:55.720 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 49.66.183.34
2022-04-21T10:47:19.231 full_node chia.full_node.full_node: WARNING  Block validation time: 2.07 seconds, pre_validation time: 1.88 seconds, cost: 5404742540, percent full: 49.134% header_hash: 7bec5865a82a89a9015b2544ab2325903b67005d2eca6d9041081743a79bea3f height: 1867892
2022-04-21T11:00:17.846 full_node chia.full_node.full_node: WARNING  Block validation time: 2.47 seconds, pre_validation time: 0.22 seconds, cost: 5497426443, percent full: 49.977% header_hash: 10baacc354fbeb3c6be83f8e3a7e84809c00edd753d02bff45ece15ac2446bec height: 1867932
2022-04-21T11:06:11.026 full_node full_node_server        : WARNING  Cannot write to closing transport 5.94.157.33
2022-04-21T11:15:13.086 full_node chia.full_node.full_node: WARNING  Block validation time: 2.55 seconds, pre_validation time: 0.21 seconds, cost: 4047413713, percent full: 36.795% header_hash: 1c9a2638191f2a222ad28734542025b91428f4f7a800eab9c09e21c6ce5655ab height: 1867974
2022-04-21T11:17:59.854 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 82.157.26.106
2022-04-21T11:24:29.523 full_node chia.full_node.full_node: WARNING  Block validation time: 2.87 seconds, pre_validation time: 0.77 seconds, cost: 5357313548, percent full: 48.703% header_hash: 9a29056fc5171f26448201a5df2d16d9991815627c655db54716712dbf7790e8 height: 1868003
2022-04-21T11:41:07.117 full_node chia.full_node.full_node: WARNING  Block validation time: 2.60 seconds, pre_validation time: 0.37 seconds, cost: 4740080212, percent full: 43.092% header_hash: e92699d91aabfd89f411277d13ca3f9cb062499a8b72bdeed2fde8c9fcdacf55 height: 1868055
2022-04-21T12:14:31.425 full_node chia.full_node.full_node: WARNING  Block validation time: 2.40 seconds, pre_validation time: 0.29 seconds, cost: 3721984547, percent full: 33.836% header_hash: 65f993b0142c1054a2e5916aa5f687c63e8807e0fe0ef4731ecf858c35e93973 height: 1868170
2022-04-21T12:25:40.918 full_node chia.full_node.full_node: WARNING  Block validation time: 2.06 seconds, pre_validation time: 0.23 seconds, cost: 3995513274, percent full: 36.323% header_hash: 8fb253852408fe3074e5a8d9ed5d664a041bfa23de625df96e514c1312652e4f height: 1868205
2022-04-21T12:43:13.559 full_node full_node_server        : WARNING  [Errno 104] Connection reset by peer 95.82.125.52
2022-04-21T12:43:41.961 full_node full_node_server        : WARNING  Cannot write to closing transport 51.91.178.138
2022-04-21T12:44:11.315 full_node chia.full_node.full_node: WARNING  Block validation time: 2.50 seconds, pre_validation time: 0.23 seconds, cost: 3819424228, percent full: 34.722% header_hash: 431d442fb8b0184f12140e1a8ce4f822c52f726d7e3179c75ea20aef7e8a75c9 height: 1868265
2022-04-21T12:48:23.603 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 116.112.94.172
2022-04-21T13:01:59.065 full_node full_node_server        : WARNING  [Errno 104] Connection reset by peer 112.50.52.27
2022-04-21T13:04:14.009 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 60.209.250.114
2022-04-21T13:13:24.731 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 183.159.60.227
2022-04-21T13:15:23.798 full_node chia.full_node.full_node: WARNING  Block validation time: 2.24 seconds, pre_validation time: 0.21 seconds, cost: 3836122646, percent full: 34.874% header_hash: 0a6ea4d3b9c3809a63a0b290e3cf6d9e0452927dfed1266fae155aeff53da458 height: 1868352
2022-04-21T13:17:32.611 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 114.246.35.113
2022-04-21T13:19:34.761 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 117.150.56.99
2022-04-21T13:55:42.816 full_node chia.full_node.full_node: WARNING  Block validation time: 2.55 seconds, pre_validation time: 0.23 seconds, cost: 5378488947, percent full: 48.895% header_hash: 200bcff68914b577acbfa6cc83f9214747ba1df7490c499e76ff1aea48409014 height: 1868481
2022-04-21T14:20:40.285 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 183.11.39.226
2022-04-21T14:24:48.034 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 116.112.94.172
2022-04-21T14:25:43.635 full_node full_node_server        : WARNING  [Errno 104] Connection reset by peer 171.107.66.150
2022-04-21T14:28:17.267 full_node chia.full_node.full_node: WARNING  Block validation time: 2.43 seconds, pre_validation time: 0.24 seconds, cost: 3934510863, percent full: 35.768% header_hash: b80972fb93172ff53933e14d3410c8e1f85bc7216a6546b8d658681db6d04c0e height: 1868565
2022-04-21T14:43:49.085 full_node full_node_server        : WARNING  Was not able to get peername from 95.24.3.165 - closing connection
2022-04-21T14:46:44.360 full_node chia.full_node.full_node: WARNING  Block validation time: 2.26 seconds, pre_validation time: 0.22 seconds, cost: 4061672410, percent full: 36.924% header_hash: 93ca8d402f07c294fb2697efbbfc7373203e4b5709100348aa983a40ee504de6 height: 1868629
2022-04-21T15:16:31.964 full_node chia.full_node.full_node: WARNING  Block validation time: 2.29 seconds, pre_validation time: 0.22 seconds, cost: 3963808447, percent full: 36.035% header_hash: 869d5516c4d29b26154bfabbc1b44f237088da75f6db39b60833788c19d2df50 height: 1868732
2022-04-21T15:22:14.508 full_node chia.full_node.full_node: WARNING  Block validation time: 2.46 seconds, pre_validation time: 0.22 seconds, cost: 4608237507, percent full: 41.893% header_hash: a687dc8a6d2cbe3f0304911b53ed1f886f22ce5aaa7188e79e05b23d1e941a49 height: 1868746
2022-04-21T15:37:32.497 full_node chia.full_node.full_node: WARNING  Block validation time: 2.50 seconds, pre_validation time: 2.02 seconds, cost: 4116358262, percent full: 37.421% header_hash: 303a32241e50b34a3b86fe40d7631770886fe9f5b643942898eae1e252d384ca height: 1868802
2022-04-21T15:39:08.051 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 113.225.22.53
2022-04-21T16:01:28.697 full_node chia.full_node.full_node: WARNING  Block validation time: 2.43 seconds, pre_validation time: 0.26 seconds, cost: 4004346293, percent full: 36.403% header_hash: f6a7c560d49069c9754b8b1266c570df9dc6553efbfcd8376b01ed8b9d623283 height: 1868890
2022-04-21T16:04:53.140 full_node full_node_server        : WARNING  Cannot write to closing transport 93.125.121.127
2022-04-21T16:14:36.418 full_node full_node_server        : WARNING  Cannot write to closing transport 81.146.33.34
2022-04-21T16:41:43.148 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 27.43.162.238
2022-04-21T16:57:05.149 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 144.255.83.203
2022-04-21T17:01:05.299 full_node chia.full_node.full_node: WARNING  Block validation time: 2.35 seconds, pre_validation time: 0.20 seconds, cost: 3899663510, percent full: 35.451% header_hash: 6ed2ebbd3d28918a73995a875be6756d91a0940107a074b512d6f2ccbcf2ee63 height: 1869101
2022-04-21T17:22:41.359 full_node chia.full_node.full_node: WARNING  Block validation time: 2.57 seconds, pre_validation time: 0.27 seconds, cost: 5308762267, percent full: 48.261% header_hash: e8b053a431e03f7e14487f5bc62b4e9435b2b8b3b8acaf9647bacc83bdd748c0 height: 1869170
2022-04-21T17:48:08.282 full_node chia.full_node.full_node: WARNING  Block validation time: 2.02 seconds, pre_validation time: 0.28 seconds, cost: None header_hash: 6bcc49159cef28dd0427550c7e03485ee4df37437609983ccac5e8b039c7c0c7 height: 1869254
2022-04-21T18:08:38.875 full_node chia.full_node.full_node: WARNING  Block validation time: 3.06 seconds, pre_validation time: 0.25 seconds, cost: 23869738, percent full: 0.217% header_hash: 90670841d9982baa7e154925c35bd3c30ba50423908e55421459fa1e3dad5ba9 height: 1869307
2022-04-21T18:16:40.203 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 112.224.65.21
2022-04-21T18:54:15.823 full_node chia.full_node.full_node: WARNING  Block validation time: 2.63 seconds, pre_validation time: 1.88 seconds, cost: 5409091753, percent full: 49.174% header_hash: 0246e4c145fc90cd46923e83ea760ea75122d21e31f106ca7790f75b150f2148 height: 1869451
2022-04-21T19:00:00.824 full_node chia.full_node.full_node: WARNING  Block validation time: 2.04 seconds, pre_validation time: 1.76 seconds, cost: None header_hash: 7c2fb80d3a401fe9bce6fbf6468e84905149d8cc9fff64134e35f1182ccdef95 height: 1869473
2022-04-21T19:23:40.166 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 113.89.247.246
2022-04-21T19:47:33.903 full_node chia.full_node.full_node: WARNING  Block validation time: 2.44 seconds, pre_validation time: 0.28 seconds, cost: 3537131312, percent full: 32.156% header_hash: af3daf435576b7a5e23b145bcbf6a3daa38b0d31caf82dc5e068dee5b97fc603 height: 1869616
2022-04-21T19:56:53.115 full_node chia.full_node.full_node: WARNING  Block validation time: 2.67 seconds, pre_validation time: 0.25 seconds, cost: 5164640418, percent full: 46.951% header_hash: e6d520358ebefbbb639ae46073254b6de70139460f399d4510b4309597d8c112 height: 1869643
2022-04-21T20:01:22.601 full_node chia.full_node.full_node: WARNING  Block validation time: 2.46 seconds, pre_validation time: 0.23 seconds, cost: 3810812267, percent full: 34.644% header_hash: 280b03f421fac1e7c493b1965f5043a53a0403f29aab4ded635bb559a2e314a7 height: 1869652
2022-04-21T20:03:08.638 full_node full_node_server        : ERROR    Exception: Failed to fetch block 1869655 from {'host': '112.3.77.200', 'port': 8444}, timed out, {'host': '112.3.77.200', 'port': 8444}. Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/server/server.py", line 595, in wrapped_coroutine
    result = await coroutine
  File "/home/chia/chia-blockchain/chia/full_node/full_node_api.py", line 124, in new_peak
    return await self.full_node.new_peak(request, peer)
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 557, in new_peak
    if await self.short_sync_backtrack(
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 491, in short_sync_backtrack
    raise e
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 476, in short_sync_backtrack
    raise ValueError(f"Failed to fetch block {curr_height} from {peer.get_peer_logging()}, timed out")
ValueError: Failed to fetch block 1869655 from {'host': '112.3.77.200', 'port': 8444}, timed out

2022-04-21T20:03:08.641 full_node full_node_server        : ERROR    Exception: Failed to fetch block 1869655 from {'host': '112.3.77.200', 'port': 8444}, timed out <class 'ValueError'>, closing connection {'host': '112.3.77.200', 'port': 8444}. Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/server/server.py", line 605, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/home/chia/chia-blockchain/chia/server/server.py", line 602, in wrapped_coroutine
    raise e
  File "/home/chia/chia-blockchain/chia/server/server.py", line 595, in wrapped_coroutine
    result = await coroutine
  File "/home/chia/chia-blockchain/chia/full_node/full_node_api.py", line 124, in new_peak
    return await self.full_node.new_peak(request, peer)
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 557, in new_peak
    if await self.short_sync_backtrack(
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 491, in short_sync_backtrack
    raise e
  File "/home/chia/chia-blockchain/chia/full_node/full_node.py", line 476, in short_sync_backtrack
    raise ValueError(f"Failed to fetch block {curr_height} from {peer.get_peer_logging()}, timed out")
ValueError: Failed to fetch block 1869655 from {'host': '112.3.77.200', 'port': 8444}, timed out

2022-04-21T20:03:08.643 full_node full_node_server        : WARNING  Banning 112.3.77.200 for 10 seconds
2022-04-21T20:25:16.827 full_node chia.full_node.full_node: WARNING  Block validation time: 2.59 seconds, pre_validation time: 0.24 seconds, cost: 4097741555, percent full: 37.252% header_hash: a636c641406edfef75020034a49f6784114a3bb7b4afa4ce892d60882bc49ba5 height: 1869738
2022-04-21T21:11:39.251 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 180.162.211.226
2022-04-21T21:43:55.924 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 23.92.25.192
2022-04-21T21:47:23.313 full_node chia.full_node.full_node: WARNING  Block validation time: 2.15 seconds, pre_validation time: 0.27 seconds, cost: 5302146754, percent full: 48.201% header_hash: b4c11780728065030cc444fe031d3520248e4e383ddbea669f790d2b3f6d065b height: 1869998
2022-04-21T21:57:20.460 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 177.52.80.122
2022-04-21T22:30:21.929 full_node full_node_server        : WARNING  Cannot write to closing transport 109.234.28.155
2022-04-21T23:07:52.139 full_node full_node_server        : WARNING  Cannot write to closing transport 58.136.20.128
2022-04-21T23:26:20.139 full_node full_node_server        : WARNING  Was not able to get peername from 50.112.41.83 - closing connection
2022-04-21T23:32:54.189 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 79.40.120.211
2022-04-21T23:38:15.648 full_node full_node_server        : WARNING  Cannot write to closing transport 218.80.72.64
2022-04-21T23:46:15.743 full_node chia.full_node.full_node: WARNING  Block validation time: 2.52 seconds, pre_validation time: 0.31 seconds, cost: 0, percent full: 0.0% header_hash: 58093b0719cbde422edb45ba40831d520eec65fe09539c20d89478f37f82fe6c height: 1870383
2022-04-21T23:50:11.196 full_node chia.full_node.full_node: WARNING  Block validation time: 2.04 seconds, pre_validation time: 1.52 seconds, cost: 4096538131, percent full: 37.241% header_hash: 858ad2a64c06314cdf372766a17c5949745ddf964b79e8454211053801829e98 height: 1870396
2022-04-22T00:02:58.200 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 79.40.120.211
2022-04-22T00:05:29.822 full_node full_node_server        : WARNING  Banning 187.190.39.123 for 10 seconds
2022-04-22T00:05:29.825 full_node full_node_server        : WARNING  Invalid handshake with peer. Maybe the peer is running old software.
2022-04-22T00:15:40.162 full_node chia.full_node.full_node: WARNING  Block validation time: 2.60 seconds, pre_validation time: 0.30 seconds, cost: 16117750, percent full: 0.147% header_hash: a9247243ab3570d0820ef3db6a6e9007e7f457bb2356380605b95505515cfe74 height: 1870463
2022-04-22T00:49:53.786 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 101.86.85.53
2022-04-22T00:51:41.269 full_node chia.full_node.full_node: WARNING  Block validation time: 2.09 seconds, pre_validation time: 0.31 seconds, cost: 5341744868, percent full: 48.561% header_hash: b571e49aa0f667d620cc72881a7591d5ebe078aa21d4477746b92d0f4f18833a height: 1870569
2022-04-22T01:19:54.709 full_node chia.full_node.full_node: WARNING  Block validation time: 2.11 seconds, pre_validation time: 0.24 seconds, cost: 3844079597, percent full: 34.946% header_hash: 1070987795edcc13dac25f414eb8cb0e6ba2198a90c2c16863d9f3215ffe4c74 height: 1870662
2022-04-22T01:50:44.228 full_node full_node_server        : WARNING  Cannot write to closing transport 52.88.178.237
2022-04-22T02:10:01.954 full_node full_node_server        : WARNING  [Errno 32] Broken pipe 114.246.35.113
2022-04-22T02:13:39.203 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-363414' coro=<RpcServer._state_changed() done, defined at /home/chia/chia-blockchain/chia/rpc/rpc_server.py:53> exception=ValueError('Newer block not found')>
Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/rpc_server.py", line 56, in _state_changed
    payloads: List[Dict] = await self.rpc_api._state_changed(*args)
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 72, in _state_changed
    data = await self.get_blockchain_state({})
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 165, in get_blockchain_state
    space = await self.get_network_space(
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 473, in get_network_space
    raise ValueError("Newer block not found")
ValueError: Newer block not found
2022-04-22T02:36:38.089 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x05d99862287ca80c07a47a7ead8255e6dbe12835e00df66f296ea552a378a3e9'}
2022-04-22T02:36:38.091 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x1fc81eaee052583497824564e06c18196720caed22d17720da55eb4aff28fb4e'}
2022-04-22T02:36:38.094 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x405fbfe7337694bcb6a7c4d8352a04b4466345b912b3bc041e30c54870e9d18e'}
2022-04-22T02:36:38.193 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x12a4f7150ade737e86267fcb2670b9f189ce712c0909facbfc846be9353a4cec'}
2022-04-22T02:36:38.195 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x9a4412dda7e17e7acec9937857bb383a5cdff71b7c220e57ee8903bac503f8e4'}
2022-04-22T02:36:38.204 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0x05d99862287ca80c07a47a7ead8255e6dbe12835e00df66f296ea552a378a3e9'}
2022-04-22T02:36:38.206 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0x1fc81eaee052583497824564e06c18196720caed22d17720da55eb4aff28fb4e'}
2022-04-22T02:36:38.207 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0x12a4f7150ade737e86267fcb2670b9f189ce712c0909facbfc846be9353a4cec'}
2022-04-22T02:36:38.209 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x405fbfe7337694bcb6a7c4d8352a04b4466345b912b3bc041e30c54870e9d18e'}
2022-04-22T02:36:38.211 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0x9a4412dda7e17e7acec9937857bb383a5cdff71b7c220e57ee8903bac503f8e4'}
2022-04-22T02:36:38.217 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x05d99862287ca80c07a47a7ead8255e6dbe12835e00df66f296ea552a378a3e9'}
2022-04-22T02:44:27.230 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T03:06:35.582 wallet chia.wallet.wallet_node    : WARNING  Peer {'host': '96.232.138.197', 'port': 8444} did not respond in time.
2022-04-22T05:21:06.144 wallet chia.wallet.wallet_node    : WARNING  Peer {'host': '68.9.48.23', 'port': 8444} did not respond in time.
2022-04-22T06:52:32.715 wallet chia.wallet.wallet_node    : WARNING  Peer {'host': '126.207.31.148', 'port': 8444} did not respond in time.
2022-04-22T08:06:05.005 wallet chia.wallet.wallet_node    : WARNING  Peer {'host': '82.113.205.48', 'port': 9444} did not respond in time.
2022-04-22T09:01:32.027 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'NO_TRANSACTIONS_WHILE_SYNCING',
 'status': 3,
 'txid': '0x12a4f7150ade737e86267fcb2670b9f189ce712c0909facbfc846be9353a4cec'}
2022-04-22T09:01:32.028 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'NO_TRANSACTIONS_WHILE_SYNCING',
 'status': 3,
 'txid': '0x405fbfe7337694bcb6a7c4d8352a04b4466345b912b3bc041e30c54870e9d18e'}
2022-04-22T09:01:32.028 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'NO_TRANSACTIONS_WHILE_SYNCING',
 'status': 3,
 'txid': '0x9a4412dda7e17e7acec9937857bb383a5cdff71b7c220e57ee8903bac503f8e4'}
2022-04-22T09:01:32.242 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x12a4f7150ade737e86267fcb2670b9f189ce712c0909facbfc846be9353a4cec'}
2022-04-22T09:01:32.243 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x405fbfe7337694bcb6a7c4d8352a04b4466345b912b3bc041e30c54870e9d18e'}
2022-04-22T09:01:32.243 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x9a4412dda7e17e7acec9937857bb383a5cdff71b7c220e57ee8903bac503f8e4'}
2022-04-22T09:01:32.385 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x12a4f7150ade737e86267fcb2670b9f189ce712c0909facbfc846be9353a4cec'}
2022-04-22T09:01:32.385 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x405fbfe7337694bcb6a7c4d8352a04b4466345b912b3bc041e30c54870e9d18e'}
2022-04-22T09:01:32.386 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x9a4412dda7e17e7acec9937857bb383a5cdff71b7c220e57ee8903bac503f8e4'}
2022-04-22T09:02:08.545 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-639' coro=<RpcServer._state_changed() done, defined at /home/chia/chia-blockchain/chia/rpc/rpc_server.py:53> exception=ValueError('Newer block not found')>
Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/rpc_server.py", line 56, in _state_changed
    payloads: List[Dict] = await self.rpc_api._state_changed(*args)
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 72, in _state_changed
    data = await self.get_blockchain_state({})
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 165, in get_blockchain_state
    space = await self.get_network_space(
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 473, in get_network_space
    raise ValueError("Newer block not found")
ValueError: Newer block not found
2022-04-22T09:02:20.641 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-683' coro=<RpcServer._state_changed() done, defined at /home/chia/chia-blockchain/chia/rpc/rpc_server.py:53> exception=ValueError('Newer block not found')>
Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/rpc_server.py", line 56, in _state_changed
    payloads: List[Dict] = await self.rpc_api._state_changed(*args)
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 72, in _state_changed
    data = await self.get_blockchain_state({})
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 165, in get_blockchain_state
    space = await self.get_network_space(
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 473, in get_network_space
    raise ValueError("Newer block not found")
ValueError: Newer block not found
2022-04-22T09:02:42.647 full_node chia.rpc.util           : WARNING  Error while handling message: Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/util.py", line 16, in inner
    res_object = await f(request_data)
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 165, in get_blockchain_state
    space = await self.get_network_space(
  File "/home/chia/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 473, in get_network_space
    raise ValueError("Newer block not found")
ValueError: Newer block not found

2022-04-22T09:06:59.085 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.086 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.219 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.450 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.451 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.674 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.675 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.675 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.845 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.846 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:06:59.846 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.094 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.094 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.225 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.225 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.225 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:07:00.226 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2022-04-22T09:10:57.992 full_node full_node_server        : WARNING  Cannot write to closing transport 24.132.64.35
Issue Analytics

State:
Created a year ago
Comments:9 (1 by maintainers)
Top GitHub Comments

2reactions
emlowecommented, Apr 25, 2022
If anyone here installed from Git - can you pull main and update the log output. We made a change here (https://github.com/Chia-Network/chia-blockchain/pull/11130/) to fix up the error here to include the block hash which would allow for some better debugging.
0reactions
Wmikebcommented, May 28, 2022
Same here happens almost every 3 hours, delays my proof lockups. They take 6sec each harvester. I have 4 they all get the same lookup time 6sec although no qualities warning comes upon their logs. Waiting for 1.3.6 hope will get fixed.