Avoid unnecessary exchange and partial aggregation in correlated scalar subquery and correlated IN predicate

For a correlated scalar subquery like

select name, (select max(name) from region where regionkey = nation.regionkey) from nation;

we currently produce the following plan (i’m using set session push_aggregation_through_join = false; to have the plan more readable)

 - Output[name, _col1] => [name:varchar(25), max:varchar(25)]
     _col1 := max
   - RemoteExchange[GATHER] => name:varchar(25), max:varchar(25)
     - Project[] => [name:varchar(25), max:varchar(25)]
x      - Aggregate(FINAL)[name, regionkey, unique][$hashvalue] => [name:varchar(25), regionkey:bigint, unique:bigint, $hashvalue:bigint, max:varchar(25)]
x          max := "max"("max_17")
x        - LocalExchange[HASH][$hashvalue] ("name", "regionkey", "unique") => name:varchar(25), regionkey:bigint, unique:bigint, max_17:varbinary, $hashvalue:bigint
x          - Aggregate(PARTIAL)[name, regionkey, unique][$hashvalue_23] => [name:varchar(25), regionkey:bigint, unique:bigint, $hashvalue_23:bigint, max_17:varbinary]
x              max_17 := "max"("name_1")
             - Project[] => [name:varchar(25), regionkey:bigint, unique:bigint, name_1:varchar(25), $hashvalue_23:bigint]
                 $hashvalue_23 := "combine_hash"("combine_hash"("combine_hash"(BIGINT '0', COALESCE("$operator$hash_code"("name"), 0)), COALESCE("$operator$hash_code"("regionkey"), 0)), COALESCE("$operator$hash_code"("unique"), 0))
               - LeftJoin[("regionkey" = "regionkey_0")][$hashvalue_18, $hashvalue_20] => [name:varchar(25), regionkey:bigint, unique:bigint, name_1:varchar(25)]
                 - RemoteExchange[REPARTITION][$hashvalue_18] => name:varchar(25), regionkey:bigint, unique:bigint, $hashvalue_18:bigint
                   - Project[] => [name:varchar(25), regionkey:bigint, unique:bigint, $hashvalue_19:bigint]
                       $hashvalue_19 := "combine_hash"(BIGINT '0', COALESCE("$operator$hash_code"("regionkey"), 0))
                     - AssignUniqueId => [name:varchar(25), regionkey:bigint, unique:bigint]
                       - TableScan[tpch:tpch:nation:sf0.01, originalConstraint = true] => [name:varchar(25), regionkey:bigint]
                           name := tpch:name
                           regionkey := tpch:regionkey
                 - LocalExchange[HASH][$hashvalue_20] ("regionkey_0") => regionkey_0:bigint, name_1:varchar(25), $hashvalue_20:bigint
                   - RemoteExchange[REPARTITION][$hashvalue_21] => regionkey_0:bigint, name_1:varchar(25), $hashvalue_21:bigint
                     - ScanProject[table = tpch:tpch:region:sf0.01, originalConstraint = true] => [regionkey_0:bigint, name_1:varchar(25), $hashvalue_22:bigint]
                         $hashvalue_22 := "combine_hash"(BIGINT '0', COALESCE("$operator$hash_code"("regionkey_0"), 0))
                         regionkey_0 := tpch:regionkey
                         name_1 := tpch:name

the part marked has redundant local exchange and partial aggregation. Since the aggregation’s group contains unique symbol assigned before join then, at least as long the aggregation remains above the join, no exchange is needed. Values with same group key will be in the same split (pipeline) anyway. The marked group could be replaced with single step aggregation (and no exchange).

Same applies to correlated IN predicates.

cc @kokosing

[sopel39]

Unless… AssignUniqueId happens after probe rows have been split into local threads. Current implementation of AssignUniqueIdOperator doesn’t seem to support parallel local execution though.

It must support parallel local execution to work correctly, because it is executed in parallel within a task. There is: com.facebook.presto.operator.AssignUniqueIdOperator#rowIdPool.

Why do we need strict ordering in StreamProperties?

Strict order is not required (my mistake, corrected previous posts), just information that rows are ordered should be enough.

Can’t we add distinct or unique local property instead?

Probe rows that pass though LeftJoin won’t be unique, but they will be ordered on all columns via unique_id.

Why do we need both ActualProperties and StreamProperties?

Supporting AssignUniqueId in ActualProperties will allow to avoid remote exchange before aggregation if AssignUniqueId is in source stage. See your example: https://github.com/prestodb/presto/issues/8171#issuecomment-387549439 It is also needed so that SINGLE aggregation can be later converted into STREAMING when AssignUniqueId is in source stage.

Supporting AssignUniqueId in StreamProperties is needed so that you can convert SINGLE aggregation into STREAMING. Note that SINGLE->STREAMING conversion is a local decision on a pipeline level (stream needs to be partitioned and ordered)

Would it be simpler to modify AssignUniqueIdOperator to support parallel local execution?

It does support that by my knowledge, @kokosing ?

[mbasmanova]

@sopel39 @martint Karol, Martin, #10731 builds on top of #10681 and adds streaming support to HashAggregationOperator, updates [Stream]PropertyDerivations to set partitioned_on(unique) and grouped(unique) properties, and uses these in AddLocalExchanges to plan streaming aggregations.