[discussion] Pulsar release 2.6.0

This issue is used for collecting the features planned for release 2.6.0.

Features

• [PIP-37] Large message size support #4400 approved doc-added
• [PIP-39] Namespace change events(System Topic) #4955 merged no-need-doc
• [PIP-45] Switch ManagedLedger to use MetadataStore interface #5358 merged
• [PIP 53] Contribute DotPulsar to Apache Pulsar
• [PIP-54] Support acknowledgment at batch index level #6052 merged doc-added
• [PIP 57] Improve Zookeeper Session Timeout Handling #6347 review-required
• [PIP-60] Support SNI routing to support various proxy-server #6566 approved
• [PIP-61] Advertise multiple addresses #6903 review-required doc-added
• [PIP-65] Adapting Pulsar IO Sources to support Batch Sources #7090 review-required
• [Load balancer] Add ThresholdShedder Strategy for loadbalancer #6772 merged
• [Load balancer] Expose loadbalance metric to Prometheus #6772 merged
• • [Key Shared] Ordering dispatch when adding consumer #7106 #7108 #7188
• [Key Shared] Add support for key hash range reading #5928 merged
• [Schema] Allow for schema reader and writer registration on SchemaDefinition #6905 merged
• Use pure-java Air-Compressor instead of JNI based libraries #5390 approved
• Support multiple pulsar clusters to use the same bk cluster #5985 merged
• Support delete inactive topic when subscriptions caught up #6077 merged
• Support consumers set custom message retry delay #6449 merged doc-added
• Add configuration to disable auto-creation of subscriptions #6456 merged
• Add a flag to skip broker shutdown on transient OOM #6634 merged
• Make zookeeper cache expiry time configurable #6668 merged doc-added
• Check replicator periodically to avoid issue due to zookeeper missing watch #6674 merged
• Expose managedLedgerCache, managedLedger, loadBalance metrics to Prometheus #6705 merged
• Optimize consumer fetch messages in case of batch message #6719 approved
• Add config to limit max partitions for a partiitoned topic #6794 merged
• Add options for the GetBookieInfo check #6800 merged
• Change default FlushEntryLogBytes to 256MB to improve bookie io throughput #6915 merged
• Make Nar Extraction Directory configurable #6933 merged
• Custom functions runtime
• Introduce precise topic publish rate limiting #7078 merged doc-required
• • Expose new entries check delay in the broker.conf #7154
• • Add broker interceptor for Intercepting all Pulsar command and REST API requests #7143
• • Only close active consumer for Failover subscription when seek() #7141
• • [Schema] Support use null key and null value in KeyValue Schema #7139
• • Allow to delete topics that are failing to recover #7131
• • Support set netty max frame size in bookkeeper.conf #7116
• • Trigger rollover when meeting maxLedgerRolloverTimeMinutes #7111
• • Cap the dispatcher batch size in bytes to fixed max #7097
• • Support specify managedLedgerMaxSizePerLedgerMbytes in broker.conf #7085
• • Allow to grant permissions when the authorization is disabled #7074
• • Add messages and bytes counter stats to broker-stats/topics #7045

Pulsar Proxy

• Add REST API to get connection and topic stats #6473 merged doc-required-it is generated automatically
• Add advertised address option #6942 merged doc-added
• Add proxyLogLevel into config #6948 merged

Admin

• Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies #5936 merged
• Support get a message by message ID in pulsar-admin #6331 merged
• Support delete subscription forcefully #6383 merged
• Add subscribe initial position for consumer CLI #6442 merged
• Support to get managed ledger info of a partitioned topic #6532 merged
• Support compact all partitions of a partitioned topic #6537 merged
• Support multi-hosts in PulsarAdmin #6547 merged
• Support to get internal stats for a partitioned topic #6624 merged
• Support enable or disable subscription auto-creation at namespace level #6637 merged
• Enable to set the subscription expiration time for each namespace #6851 merged doc-added

Functions

• Built-in Functions #6895 approved
• [Function-Go] Add Go Function heartbeat (and gRPC service) for production usage #6031 merged
• Add custom property option to functions #6348 merged
• Separate TLS configuration of function worker and broker #6602 merged
• Added ability to build consumers in functions and sources #6954 merged
• • Support DLQ on sources and sinks #7032

Pulsar SQL

• KeyValue schema support #6325 merged
• Multiple Avro schema version #4847 merged
• • Fix presto SQL does not start metrics service before queue execute #7030

Pulsar IO

• [RabbitMQ] Allow routing key per message to RabbitMQ sink connector #5890 merged
• [RabbitMQ] Add passive config options #6679 merged
• [debezium] Upgrad from v0.10.0-Final to v1.0.0-Final #5972 merged
• [debezium] Support avro schema for debezium connector #6034 merged
• [influxdb2] Add support for influxdb2 in pulsar-influxdb-sink #6601 merged
• [jdbc] Add jdbc sinks: postgres, mariadb, clickhouse #6835 merged
• Added ability for sources to publish messages on their own #6941 merged

Java Client

• Change the time unit of patternAutoDiscoveryPeriod to seconds #5950 merged
• Support waiting for inflight messages while closing producer #6648 merged
• Add support to load TLS certs/key dynamically from input stream #6760 merged
• Support return sequence ID when throwing Exception for async send message #6825 merged
• • Change the default value of maxLookupRedirects of Java client to 20 #7126
• • Limit the number of times lookup requests are redirected #7096
• • Fix duplicated messages sent to dead letter topic #7021

Python Client

• Add deliver_at and deliver_after for the producer #6737 merged

CPP Client

• • Support seek by time on partitioned topic #7198
• • Refresh authentication credentials #7070
• • Fix Result can’t be serialized to string inside the library #7034
• • Supo zstd and Snappy compression to C API #7014

Bug Fixes

• [Websocket] Fix incorrect topic URL parse #6630 merged
• [Broker] Make ZkBookieRackAffinityMapping work as expected #6917 merged
• [Broker] Fix backlog and backlog size stats keeps growing #7082
• [Java Client] Fix connection leak #6524 merged
• [Java Client] Fix message id compare between MessageId and BatchMessageId #6621 merged
• [Pulsar SQL] Fix problem with multiple zookeeper address #6947 merged
• • [CPP Client] Fix deadlock of consumer for topics auto discovery #7206
• • [Managed Ledger] Fix NPE on opening non-durable cursors on an empty managed ledger #7133
• • [Java Client] Fix memory leak when create producer with not exsits topic #7120 #7124
• • [Broker] Fixed increasing number of partitions with attached readers #7077

Others

• Work on JDK 11

[Jennifer88huang-zz]

Some users are expecting Pulsar transaction and KOP in the new release.

[codelipenghui]

@lhotari I have started the 2.6.0 candidate, we can plan #6869 in 2.6.1.