AlluxioMaster leader switch unexpectedly after manual leader switch with "quorum elect" command

Alluxio Version: v2.6.2

Describe the bug With embedded journal system, once using ‘fsadmin journal quorum elect -address <xxxx>’, nodes priorities were changed and not reset, hence, if the given node temporarily offline and back online sometime later, the leader was switched again immediately after the given node back online.

To Reproduce

1. manual switch leader to NodeA ./bin/alluxio fsadmin journal quorum elect -address <node_a>:19200
2. check the priority info of nodes by ./bin/alluxio fsadmin journal quorum info -domain MASTER
3. Take down AlluxioMaster process on NodeA, wait for leader role fail overs to one of remain two nodes.
4. Once the remain two nodes reaches final state as one LEADER, one FOLLOWER, take NodeA online again
5. A leader switch happens again once NodeA is online.

Expected behavior in step5, the leader switch is an unexpected operation.

Urgency Describe the impact and urgency of the bug.

Additional context

Should be a side effect of fsadmin journal quorum elect -address <xxxx> who sets priorities to master nodes, but doesn’t clean the priority after operation done.

Suggest fix:

• restore or reset the priority info by calling resetPriorities in QuorumElectCommand or calling resetPriorities in RaftJournalSystem::transferLeadership after leadership is transferred.

[jenoudet]

I was able to reproduce and confirm this behavior. Looking into it.

[codings-dan]

@jenoudet @kevincai The change of node priority will indeed affect the result of the next election. When the quroum elect command is designed at the beginning, the resetPriority command will be called after the switch is completed. But I think the resetPriority command also needs to monitor whether the priority is reset successfully. If this process is all implemented in resetPriorities in QuorumElectCommand or RaftJournalSystem::transferLeadership, the switching time may be too long. A compromise solution is to manually reset the priority after the switch is completed. This has been implemented in #14031 , for reference, thx.