External validator client should retrieve network information from the beacon node config endpoint

Description

Rather than having to be explicitly told the network to use, the external validator should retrieve the network config from the beacon chain config endpoint. Shouldn’t need to specify --network for the external validator client at all.

[ajsutton]

The issue here might be that the --network option is already optional, and defaults to “mainnet”, when not specified. This means that users already have an expectation of how this option behaves even when they are not using it explicitly. I don’t imagine we want to break this expectation right away either, and deprecating the default behavior (or implicit usage) of something generally feels a bit trickier (i.e. eventually removing --network might not be something such users notice)

Good point. I was thinking backward compatibility in terms of not failing to start if you have --network specified (because if it’s removed it becomes an invalid option). Changing behaviour unexpectedly is probably not ideal though.

The saving grace here is that if you have a beacon node on prater and a validator client on mainnet, it’s not going to work anyway. So while we probably want to provide some notice about the upcoming change, ultimately when we do switch over, the behaviour should either be unchanged or go from not working to working.

• is our desired end-state that fetching the network spec from the beacon node is the only option (i.e. a user can’t override this), or that we continue to support a way for the user to manually specify a network or custom spec file if they choose?

Yes, I think ultimately this is where we want to get to.

• in the short term, fetching the network spec from the beacon node API will have to be something a user explicitly enables (since not specifying anything already uses “mainnet”). How should we do this? This is of course influenced by the question above. Some of the concerns are, of course, whatever we use (a new boolean flag, augmenting the --network option, etc.) will have to a) eventually transition to an option that is implicitly enabled by default b) potentially become deprecated, and require phasing out if it isn’t something one can disable in the end-state (question above)

I’d suggest that we do this in three steps.

1. Support --network auto which means load the config from the beacon node. Default would remain --network mainnet. This then acts as a really handy feature toggle so people can test out the new approach without it being the default and applying to everyone all at once
2. Switch the default to --network auto. Probably at this point we’d also hide the --network option, update docs etc so new users just expect it to get the config automatically, but teku still starts for users who have the --network option. We’d add removal of --network to the list of upcoming breaking changes in the changelog.
3. Probably after a fair bit of time, we’d fully remove the --network option.

[ermyas]

Sorry for the delayed response @ajsutton. This sounds like a sensible approach. I’ve pushed a draft PR #4324, for feedback.