1.3.4 Farming Performance (400 second lookup time) [Bug]

What happened?

I have a large monolithic farm with several JBODs attached to a single farmer.

New releases occasionally lead to very slow farming performance, whilst 1.3.3 was generally stable.

1.3.4 sets a new record at 437 seconds for a lookup.

I have full DEBUG level logs available and am happy to work with team over keybase (reythia).

A grep of logs suggests there is a single disk is associated with the majority of slow lookups since 1.3.4, so the issue may well be hardware on my end. I will check this and report back.

Even if it is a failing disk, it shouldn’t bring down the whole farm.

Wouldn’t it make more sense for lookups to timeout with a warning after ~30 seconds rather than skipping several signage points? And respond with what is available in time? ie “700 of 800 plots”

Previous farming issues:

1.3.0 https://github.com/Chia-Network/chia-blockchain/issues/10668 1.2.10 https://github.com/Chia-Network/chia-blockchain/issues/8972

Version

1.3.4

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2022-04-23T06:19:30.698 harvester chia.harvester.harvester: WARNING  Looking up qualities on ... took: 437.51397252082825. This should be below 5 seconds to minimize risk of losing rewards.

2022-04-23T06:19:30.698 harvester chia.harvester.harvester: INFO farming a71d9acfe9... Found 0 proofs. Time: 437.51397 s. Total...

[xdustinface]

This should be fixed already by https://github.com/Chia-Network/chia-blockchain/pull/11204.

[emlowe]

Note, long lookup times due to a single disk issue is a long standing issue (see #7788 ), however, it isn’t simple to solve and hasn’t been a high priority.