Teku + Besu, attestations randomly missed since the merge

besu 22.7.2 teku 22.9.0

Sep 15 15:06:00 eth-consensus teku[522445]: 2022-09-15 15:06:00.673 INFO - Attestation performance: epoch 146952, expected 6, produced 6, included 5 (83%), distance 1 / 1.00 / 1>

teku log https://termbin.com/573u besu log https://termbin.com/41ic

It’s not clear why produced is 6 and included 5.

The end result is 80% performance.

[ajsutton]

@totocz A 2 second arrival is relatively slow - ideally it should be under 1s, but it’s not generally the end of the world. The main problem with those numbers is execution_payload_result_received which is the time that Teku is waiting on besu to respond and that’s what’s caused those blocks to complete importing after the 4s mark. Some of them would have imported late even if they’d been received quickly.

To give an idea of block arrival times. In the last day, 11% of blocks were received within 500ms, 50% of blocks were received between 500ms and 1s, 25% between 1s and 1.5s and 8% between 1.5s and 2s. Then about 5% of blocks were received later than that.

So with the time besu is taking most blocks would probably be imported just in time, but quite frequently blocks would be imported late and you’d see an empty slot message when really the block was received with enough time that it should have been imported in time. This is the common pattern we’re seeing with 22.9.1 where it’s the execution client that is now dominating the block import time and there’s not much more Teku can do to improve things.

[iamkoa]

Having the same issue with Teku + Geth. Attestation performance has gone from 95%+ pre-merge to around 80% post-merge.