Invalidated terminal block hash (by EL)

Description

I stuck how should we handle this, so explaining the issue. We reach the terminal block and are sending FCU to the EL:

2022-08-08 13:56:28.120 INFO  - Loaded initial state at epoch 0 (state root = 0x71ef5939afd6e13ef0bf496e51144e4a95381ecdd7eba875aee705adfd43a31a, block root = 0x9a13008edfae88d4c53c4a48d8ca1bc3a207e0facb37b175dc4096df4684f40c, block slot = 0).
2022-08-08 13:56:28.132 INFO  - Genesis Event *** 
Genesis state root: 0x71ef5939afd6e13ef0bf496e51144e4a95381ecdd7eba875aee705adfd43a31a 
Genesis block root: 0x9a13008edfae88d4c53c4a48d8ca1bc3a207e0facb37b175dc4096df4684f40c 
Genesis time: 2022-08-08 13:56:34 GMT
2022-08-08 13:56:48.086 TRACE  - engineForkChoiceUpdated(forkChoiceState=ForkChoiceState{headBlockRoot=0x9a13008edfae88d4c53c4a48d8ca1bc3a207e0facb37b175dc4096df4684f40c, headBlockSlot=0, headExecutionBlockHash=0x463bb01dc74aaf70ea402298240a43e40f323f535b00e9218da2df8a3a77552c, safeExecutionBlockHash=0x0000000000000000000000000000000000000000000000000000000000000000, finalizedExecutionBlockHash=0x0000000000000000000000000000000000000000000000000000000000000000, isHeadOptimistic=false}, payloadAttributes=Optional.empty) -> ForkChoiceUpdatedResult{payloadStatus=PayloadStatus{status=Optional[INVALID], latestValidHash=Optional[0x0fea94893de4066de64c5f3ad817b3370607cb1d95876f2e71b44ea7ad741aa1], validationError=Optional.empty, failureCause=Optional.empty}, payloadId=Optional.empty}

It looks like a correct answer by EL as it invalidates exact terminal block hash. But what we are doing next looks incorrect: we are trying to invalidate corresponding BeaconBlock which is not incorrect and getting an Exception:

2022-08-08 13:56:48.088 INFO  - node ProtoNode{blockSlot=0, stateRoot=0x71ef5939afd6e13ef0bf496e51144e4a95381ecdd7eba875aee705adfd43a31a, blockRoot=0x9a13008edfae88d4c53c4a48d8ca1bc3a207e0facb37b175dc4096df4684f40c, parentRoot=0x0000000000000000000000000000000000000000000000000000000000000000, justifiedCheckpoint=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, finalizedCheckpoint=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, unrealizedJustifiedCheckpoint=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, unrealizedFinalizedCheckpoint=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, executionBlockHash=0x0000000000000000000000000000000000000000000000000000000000000000, weight=0, parentIndex=Optional.empty, bestChildIndex=Optional.empty, bestDescendantIndex=Optional.empty, validationStatus=VALID} marking as invalid, 0x9a13008edfae88d4c53c4a48d8ca1bc3a207e0facb37b175dc4096df4684f40c, Optional[0x0fea94893de4066de64c5f3ad817b3370607cb1d95876f2e71b44ea7ad741aa1], true
2022-08-08 13:56:48.089 ERROR - PLEASE FIX OR REPORT | Unexpected exception thrown for forkchoice-async-0
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Cannot change node validity from VALID to INVALID

From my view invalid terminal block hash should be handled here in the same way like in #5887 not invalidating transition BeaconBlock by default because it doesn’t have any reference to the terminal block (and is not invalid btw). Do I understand it correctly?

[ajsutton]

but what if, say ttd = 1000

pre-terminal (999) - terminal 1 (1001)
               \- terminal 2 (1002) - arrives a bit later

And we are sending fcU with terminal 1? It looks like a correct EL behavior in this case

No, it is entirely within the CL’s prerogative to choose any terminal block that meets the conditions. It would normally pick the EL chain head which would have the highest total difficulty, but there is no requirement for that to be true and the block isn’t invalid just because there’s an alternative block.

[zilm13]

yeah, CL could have requirement for hash, for example, right