Release note 0.15.0

Highlight

Multi-tenant & Resource isolation

Now users can divide BE nodes in a Doris cluster into multiple resource groups by means of resource tags, so that online and offline business can be unified management and node-level resource isolation. At the same time, the resource overhead of a single query can be controlled by limiting the CPU, memory overhead, and complexity of a single query task, thereby reducing the problem of resource preemption between different queries. For details, please refer to Usage Document.

#6159 #6203 #6443

Performance optimization

Added Runtime Filter (#6121) and Join Reorder (#6226) functions.

The Runtime Filter function filters the data of the left table by using the join key column condition of the right table in the join operator, which can significantly improve query efficiency in most join scenarios. For example, under the Star Schema Benchmark (TPCH’s condensed test set), a 2-10 times performance improvement can be obtained. For details, please refer to Usage Document.

The Join Reorder function can automatically help adjust the order of joins in SQL through the cost model to help obtain the best join efficiency. It can be turned on through the conference return variable set enable_cost_based_join_reorder=true.

New features

1. Support direct connection to Canal Server to synchronize MySQL binlog data. For details, please refer to Use Document. (#6289)
2. Support String column type, length range 1-2GB (#6391)
3. Support the List partition function, which can create partitions for enumerated values. (#5529)
4. Support transactional insert statement function. You can import data in batches by begin; insert;,insert;,…; commit; (#6245)
5. Support the update statement function on the Unique Key model. The update set where statement can be executed on the Unique Key model table (#6230)
6. Support SQL blocking list function. The execution of some SQL can be prevented through regular, hash value matching, etc. (#6192)
7. Support LDAP login authentication (#6333)

Extensions

1. Support Flink-Doris-Connector (#5375)

   For details, please refer to Usage Document

2. Support DataX doriswriter plug-in (#6107)

   For details, please refer to Usage Document

3. Spark-Doris-Connector supports data writing to Doris. (#6256)

Optimization

Query

Supports the calculation of all constant expressions by using BE’s function calculation ability in the SQL query planning stage (#6233).

Load

1. When importing text format files, specify multi-byte row and column separators or invisible separators (#5462 #5524)
2. Support for importing compressed format files through StreamLoad (#5463)
3. Stream Load supports importing json data in multi-line format (5774)

Export

1. Support the Export function to specify where filter conditions. Support exporting files to use multi-byte row and column separators. Support export to local file (#5445)
2. The Export export function supports exporting only the specified columns. (#5689)
3. Support to export the result set to local disk through outfile statement, and support to write the successfully exported mark file after export (#5489)

Utilities

1. The dynamic partition function supports the creation and retention of designated historical partitions, and supports automatic cold and hot data migration settings. (#5703 #5877 #6554)
2. Support the use of a visual tree structure to display queries, imported plans and profiles on the command line. (#5475 #6214)
3. Support recording and viewing stream load operation logs. (#5452 #5488)
4. When consuming Kafka data through Routine Load, you can specify the time point for consumption. (#5832)
5. Support to export the creation statement of routine load through the show create routine load function. (#6110)
6. Support to start and stop all routine load jobs with one key via the pause/resume all routine load command. (#6394)
7. Support to modify routine load broker list and topic through alter routine load statement (#6335)
8. Support create table as select function. (#6102)
9. Support to modify column comments and table comments through the alter table command. (#6387)
10. show tablet status adds table creation time and data update time (#6117)
11. Support the show data skew command to view the data distribution of the table to troubleshoot data skew problems. (#6219)
12. Support the show/clean trash command to view the disk occupancy of the BE file recycle bin and actively clear it (#6247 #6323)
13. Support showing which views a table is referenced by the show view statement. (#5813)

New builtin function

1. bitmap_min, bit_length (#5581 #6140)
2. yearweek, week, makedate (#6000)
3. Percentile exact percentile function (#6410)
4. json_array, json_object, json_quote (#6506)
5. Supports the creation of custom public keys for the AES_ENCRYPT and AES_DECRYPT functions. (6115)
6. Support creating function alias through create alias function to combine multiple functions. (#6261)

Others

1. Support to access the ES appearance of the SSL connection protocol (#5325)
2. Support to specify the number of hot partitions in the dynamic partition properties, and the hot partitions will be stored in the SSD disk. (#5877)
3. Supports importing json format data through Broker Load. (#5845)
4. Supports directly accessing hdfs through the libhdfs3 library for data import and export without the need for a broker process. (#5686)
5. The select into outfile function supports export parquet file format, and supports parallel export (#5938 #6539)
6. ODBC external table supports SQLServer. (6223)

Contributors

Thanks for all contributors who contrubute to this release:

@924060929 @acelyc111 @Aimiyoo @amosbird @arthur-zhang @azurenake @BiteTheDDDDt @caiconghui @caneGuy @caoliang-web @ccoffline @chaplinthink @chovy-3012 @ChPi @copperybean @crazyleeyang @dh-cloud @DinoZhang @dixingxing0 @dohongdayi @e0c9 @EmmyMiao87 @eyesmoons @francisoliverlee @Gabriel39 @gaodayue @GoGoWen @HappenLee @harveyyue @Henry2SS @hf200012 @huangmengbin @huozhanfeng @huzk8 @hxianshun @ikaruga4600 @JameyWoo @Jennifer88huang @JinLiOnline @jinyuanlu @JNSimba @killxdcj @kuncle @liutang123 @luozenglin @luzhijing @MarsXDM @mh-boy @mk8310 @morningman @Myasuka @nimuyuhan @pan3793 @PatrickNicholas @pengxiangyu @pierre94 @qidaye @qzsee @shiyi23 @smallhibiscus @songenjie @spaces-X @stalary @stdpain @Stephen-Robin @Sunt-ing @Taaang @tarepanda1024 @tianhui5 @tinkerrrr @TobKed @ucasfl @Userwhite @vinson0526 @wangbo @wangliansong @wangshuo128 @weajun @weihongkai2008 @weizuo93 @WindyGao @wunan1210 @wuyunfeng @xhmz @xiaokangguo @xiaoxiaopan118 @xinghuayu007 @xinyiZzz @xuliuzhe @xxiao2018 @xy720 @yangzhg @yx91490 @zbtzbtzbt @zenoyang @zh0122 @zhangboya1 @zhangstar333 @zuochunwei

[htyoung]

请问大概什么时候 Release？比较期待多租户与资源隔离的功能

[morningman]

@morningman Will vectorization query optimization show up in this release?

No, it is still WIP. But you can try part of this feature by using Palo(powered by Doris) here: palo.baidu.com, version 0.15.1-rc09