Feature Request: Support for Leader Election

I noticed a pattern we use occasionally at Netflix (Leader Election) wasn’t represented in this framework so I thought I’d raise an issue and outline some of the nice features that a good Spring wrapper around the Curator recipe could offer.

• Enabling the feature
  • Suggest something like @EnableLeaderElection(zkPath=“/some/zookeeper/path”)
  • Extra nice if we can make this Repeatable or or take an array argument for zkPath to multiple leadership for multiple topics
• Discovery/Eureka integration
  • If an application is marked out of service or unhealthy in Eureka (or just unhealthy in terms of Spring Boot HealthIndicators possibly), the application should give up its LeaderLatch so that another instance can take over. This helps to support Red/Black pushes.
  • Implementation note, when coming back into service or healthy again, we would want the application to again request leadership. This may require re-creating the Curator LeaderLatch, so that bean may not be a good fit for Singleton scope and instead may need some proxy.
• Event Integration
  • A LeadershipEvent should be fired whenever status changes. This would be used to support batch processes.
  • Common patterns that may be useful to expose via Config are to shut down framework components such as MessagingContainers (JMS, SQS, RabbitMQ) doing queue consumption, Spring Integration channel adapters, Spring Batch jobs, @Scheduled pollers, etc. Obviously users can subscribe to the events themselves to manage each of these frameworks, but I’m considering a config pattern as follows

spring.leadership.manage.messaging=aContainerName //some specific container bean name
spring.leadership.manage.messaging=* //all the containers
spring.leadership.manage.integration=aChannel,anotherChannel,*Channel //bean names that support start/stop
spring.leadership.manage.batch=aJob, anotherJob

• Actuator
  • An actuator for leadership that can indicate if the current instance is a leader (Helpful if status code also reflects this, 2xx for yes, something else for no)
  • Actuator should indicate what beans/channels are being governed by Leadership as in the above point
  • Actuator should accept POST requests to force an instance to relinquish leadership (a handy testing util)

[jkschneider]

WDYT about @LeaderOnly for things like @Scheduled to use instead of a prop where possible?

[daniellavoie]

I went a little further with an implementation and realized that using Spring EventListener to react to a LeaderElection composed with a @Scheduled is something really dangerous for race conditions. The task that we wish to be executed by a single instance needs to run in a context where the lock has the highest guarantees of being active.

This is why I am proposing something like this :

  @Autowired
  private LockClient lockClient;

  @Scheduled(fixedDelay=60000)
  public void  executeSomethingImportant(){
    // Will executes the Runnable if lock is active.
    lockClient.executeIfOwned("lock-key", () -> importantService.doStuff());

    // This variant uses spring.application.name as the lock key.
    lockClient.executeIfOwned(() -> importantService.doStuff());
  }

With a LockClient providing a “safe” context to execute our tasks, we could then support an extension of @Scheduled like @LeaderScheduled. But this is tricky since it would represents rewriting a ScheduledAnnotationBeanPostProcessor specially for the @LeaderScheduled. I am not a fan. I think a simple wrap with lockClient.executeIfOwned by end users is less messy than having to fork the processor.

Edit : A working POC is available here