[PIP-139] Support Broker send command to close producer/consumer.

Relation pull request: #13337 Authors: @Technoboy- @mattisonchao

Motivation

• When there are no user-created topics under a namespace, Namespace should be deleted. But currently, the system topic existed and the reader/producer could auto-create the system which may cause the namespace deletion to fail.
• When enabling geo-replication under a namespace, inactive topics could not be deleted by GC, and if the broker enables auto-create topic, the topic will be created(replicator) even deleted with force=true.

For the above reasons, we need to close the topics readers/producers first, then delete the relative topics.

Following this way, we need to close consumers/producers(not disconnect, which means do not allow them to reconnect). So we write this PIP to discuss to resolve this problem.

Related email : https://lists.apache.org/thread/p4tmgk23l8j554bjr4h4ofk97bll5gp4

Goal

We can add a new field(reconnect) in command CommandCloseProducer/ CommandCloseConsumer to achieve this goal.

API Changes

• Add reconnect to CommandCloseProducer;

**Before**

message CommandCloseProducer {
    required uint64 producer_id = 1;
    required uint64 request_id = 2;
}

**After**
message CommandCloseProducer {
    required uint64 producer_id = 1;
    required uint64 request_id = 2;
    optional bool reconnect = 3 [default = true];
}

• Add reconnect to CommandCloseConsumer

**Before**

message CommandCloseConsumer {
    required uint64 consumer_id = 1;
    required uint64 request_id = 2;
}

**After**
message CommandCloseConsumer {
    required uint64 consumer_id = 1;
    required uint64 request_id = 2;
    optional bool reconnect = 3 [default = true];
}

Implementation

ClientCnx - Producer:

Before

@Override
protected void handleCloseProducer(CommandCloseProducer closeProducer) {
   log.info("[{}] Broker notification of Closed producer: {}", remoteAddress, closeProducer.getProducerId());
   final long producerId = closeProducer.getProducerId();
   ProducerImpl<?> producer = producers.get(producerId);
   if (producer != null) {
       producer.connectionClosed(this);
   } else {
       log.warn("Producer with id {} not found while closing producer ", producerId);
   }
}  

After:

@Override
protected void handleCloseProducer(CommandCloseProducer closeProducer) {
   log.info("[{}] Broker notification of Closed producer: {}", remoteAddress, closeProducer.getProducerId());
   final long producerId = closeProducer.getProducerId();
   ProducerImpl<?> producer = producers.get(producerId);
   if (producer != null) {
       if (closeProducer.isReconnect()) {
           producer.connectionClosed(this);
       } else {
           producer.closeAsync();
       }
   } else {
       log.warn("Producer with id {} not found while closing producer ", producerId);
   }
}

ClientCnx - Consumer:

Before

@Override
protected void handleCloseConsumer(CommandCloseConsumer closeConsumer) {
   log.info("[{}] Broker notification of Closed consumer: {}", remoteAddress, closeConsumer.getConsumerId());
   final long consumerId = closeConsumer.getConsumerId();
   ConsumerImpl<?> consumer = consumers.get(consumerId);
   if (consumer != null) {
       consumer.connectionClosed(this);
   } else {
       log.warn("Consumer with id {} not found while closing consumer ", consumerId);
   }
}    

After

@Override
protected void handleCloseConsumer(CommandCloseConsumer closeConsumer) {
   log.info("[{}] Broker notification of Closed consumer: {}", remoteAddress, closeConsumer.getConsumerId());
   final long consumerId = closeConsumer.getConsumerId();
   ConsumerImpl<?> consumer = consumers.get(consumerId);
   if (consumer != null) {
       if (closeConsumer.isReconnect()) {
           consumer.connectionClosed(this);
       } else {
           consumer.closeAsync();
       }
   } else {
       log.warn("Consumer with id {} not found while closing consumer ", consumerId);
   }
}   

Reject Alternatives

none.

[Technoboy-]

@merlimat @lhotari @eolivelli @codelipenghui @gaoran10 Could you help give some suggestions?

[Technoboy-]

More details and discussions are in the email thread. and after diving into the code, we push #14730 to resolve the current problem. So now we decide to close this PIP.