PG17 compatibility: fix plan diffs in multi_explain (#7780)

Regress test `multi_explain` has two queries that have a different query
plan with PG17. Here is part of the plan diff for the query labelled
_Union and left join subquery pushdown_ in `multi_explain.sql` (for the
complete diff, search for `multi_explain`
[here](https://github.com/citusdata/citus/actions/runs/12158205599/attempts/1)):
```
                                       ->  Sort
                                             Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone, events.event_time
-                                            ->  Hash Left Join
-                                                  Hash Cond: (users.composite_id = subquery_2.composite_id)
-                                                  ->  HashAggregate
-                                                        Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
+                                            ->  Nested Loop Left Join
+                                                  Join Filter: (users.composite_id = subquery_2.composite_id)
+                                                  ->  Unique
+                                                        ->  Sort
+                                                              Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
                                                               ->  Append
```
The change is the same in both queries; a hash left join with subquery_1
on the outer and subquery_2 on the inner side of the join is now a
nested loop left join with subquery_1 on the outer and subquery_2 on the
inner; additionally, the chosen method of uniquifying the UNION in
subquery_1 has changed from hashed grouping to sort followed by unique,
as shown in the diff above.

The PG17 commit that caused this plan change is likely _[Fix MergeAppend
to more accurately compute the number of rows that need to be
sorted](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9d1a5354f)_
because it impacts the estimated rows counts of UNION paths. Comparing a
costed plan of the query between PG16 and PG17 I noticed that with PG16
the rows estimate for the UNION in subquery_1 is 4, whereas with PG17
the rows estimate is 2. A lower rows estimate in the outer side of the
join may result in nested loop looking cheaper than hash join for the
left outer join, hence the plan change in the two queries where there is
a UNION on the outer side of a left outer join.

The proposed fix achieves a consistent plan across all supported
postgres versions by temporarily disabling nested loop join and sort for
the two impacted queries; the postgres optimizer selects hash join for
the outer left join and hashed aggregation for the UNION operation. I
investigated tweaking the queries, but was not able to arrive at a
consistent plan, and I believe the SQL operator (e.g. join, group by,
union) implementations are orthogonal to the intent of the test, so this
should be a satisfactory solution, particularly as it avoids introducing
a second alternative output file for `multi_explain`.
pull/7922/head
Colm 2024-12-17 21:42:15 +00:00 committed by naisila
parent 592416250c
commit c3d21b807a
3 changed files with 172 additions and 144 deletions

View File

@ -671,6 +671,15 @@ Aggregate
-> Hash -> Hash
-> Seq Scan on events_1400285 events -> Seq Scan on events_1400285 events
Filter: ((event_type)::text = ANY ('{click,submit,pay}'::text[])) Filter: ((event_type)::text = ANY ('{click,submit,pay}'::text[]))
SELECT success FROM run_command_on_workers('alter system set enable_nestloop to off');
t
t
SELECT success FROM run_command_on_workers('alter system set enable_sort to off');
t
t
SELECT success FROM run_command_on_workers('select pg_reload_conf()');
t
t
-- Union and left join subquery pushdown -- Union and left join subquery pushdown
EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF)
SELECT SELECT
@ -741,41 +750,38 @@ HashAggregate
Tasks Shown: One of 4 Tasks Shown: One of 4
-> Task -> Task
Node: host=localhost port=xxxxx dbname=regression Node: host=localhost port=xxxxx dbname=regression
-> GroupAggregate -> HashAggregate
Group Key: subquery_top.hasdone Group Key: COALESCE(subquery_2.hasdone, 'Has not done paying'::text)
-> Sort -> GroupAggregate
Sort Key: subquery_top.hasdone Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone
-> Subquery Scan on subquery_top -> Sort
-> GroupAggregate Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone, events.event_time
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone -> Hash Left Join
-> Sort Hash Cond: (users.composite_id = subquery_2.composite_id)
Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone, events.event_time -> HashAggregate
-> Hash Left Join Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
Hash Cond: (users.composite_id = subquery_2.composite_id) -> Append
-> HashAggregate -> Hash Join
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time Hash Cond: (users.composite_id = events.composite_id)
-> Append -> Seq Scan on users_1400289 users
-> Hash Join Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
Hash Cond: (users.composite_id = events.composite_id) -> Hash
-> Seq Scan on users_1400289 users -> Seq Scan on events_1400285 events
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) Filter: ((event_type)::text = 'click'::text)
-> Hash -> Hash Join
-> Seq Scan on events_1400285 events Hash Cond: (users_1.composite_id = events_1.composite_id)
Filter: ((event_type)::text = 'click'::text) -> Seq Scan on users_1400289 users_1
-> Hash Join Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
Hash Cond: (users_1.composite_id = events_1.composite_id) -> Hash
-> Seq Scan on users_1400289 users_1 -> Seq Scan on events_1400285 events_1
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) Filter: ((event_type)::text = 'submit'::text)
-> Hash -> Hash
-> Seq Scan on events_1400285 events_1 -> Subquery Scan on subquery_2
Filter: ((event_type)::text = 'submit'::text) -> Unique
-> Hash -> Sort
-> Subquery Scan on subquery_2 Sort Key: ((events_2.composite_id).tenant_id), ((events_2.composite_id).user_id)
-> Unique -> Seq Scan on events_1400285 events_2
-> Sort Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
Sort Key: ((events_2.composite_id).tenant_id), ((events_2.composite_id).user_id)
-> Seq Scan on events_1400285 events_2
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-- Union, left join and having subquery pushdown -- Union, left join and having subquery pushdown
EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF)
SELECT SELECT
@ -856,44 +862,48 @@ Sort
Tasks Shown: One of 4 Tasks Shown: One of 4
-> Task -> Task
Node: host=localhost port=xxxxx dbname=regression Node: host=localhost port=xxxxx dbname=regression
-> GroupAggregate -> HashAggregate
Group Key: subquery_top.count_pay Group Key: COALESCE(subquery_2.count_pay, '0'::bigint)
-> Sort -> GroupAggregate
Sort Key: subquery_top.count_pay Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay
-> Subquery Scan on subquery_top Filter: (array_ndims(array_agg(('action=>1'::text) ORDER BY events.event_time)) > 0)
-> GroupAggregate -> Sort
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay, events.event_time
Filter: (array_ndims(array_agg(('action=>1'::text) ORDER BY events.event_time)) > 0) -> Hash Left Join
-> Sort Hash Cond: (users.composite_id = subquery_2.composite_id)
Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay, events.event_time -> HashAggregate
-> Hash Left Join Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
Hash Cond: (users.composite_id = subquery_2.composite_id) -> Append
-> Hash Join
Hash Cond: (users.composite_id = events.composite_id)
-> Seq Scan on users_1400289 users
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
-> Hash
-> Seq Scan on events_1400285 events
Filter: ((event_type)::text = 'click'::text)
-> Hash Join
Hash Cond: (users_1.composite_id = events_1.composite_id)
-> Seq Scan on users_1400289 users_1
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
-> Hash
-> Seq Scan on events_1400285 events_1
Filter: ((event_type)::text = 'submit'::text)
-> Hash
-> Subquery Scan on subquery_2
-> HashAggregate -> HashAggregate
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time Group Key: events_2.composite_id
-> Append Filter: (count(*) > 2)
-> Hash Join -> Seq Scan on events_1400285 events_2
Hash Cond: (users.composite_id = events.composite_id) Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-> Seq Scan on users_1400289 users SELECT success FROM run_command_on_workers('alter system reset enable_nestloop');
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) t
-> Hash t
-> Seq Scan on events_1400285 events SELECT success FROM run_command_on_workers('alter system reset enable_sort');
Filter: ((event_type)::text = 'click'::text) t
-> Hash Join t
Hash Cond: (users_1.composite_id = events_1.composite_id) SELECT success FROM run_command_on_workers('select pg_reload_conf()');
-> Seq Scan on users_1400289 users_1 t
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) t
-> Hash
-> Seq Scan on events_1400285 events_1
Filter: ((event_type)::text = 'submit'::text)
-> Hash
-> Subquery Scan on subquery_2
-> GroupAggregate
Group Key: events_2.composite_id
Filter: (count(*) > 2)
-> Sort
Sort Key: events_2.composite_id
-> Seq Scan on events_1400285 events_2
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-- Lateral join subquery pushdown -- Lateral join subquery pushdown
-- set subquery_pushdown due to limit in the query -- set subquery_pushdown due to limit in the query
SET citus.subquery_pushdown to ON; SET citus.subquery_pushdown to ON;

View File

@ -671,6 +671,15 @@ Aggregate
-> Hash -> Hash
-> Seq Scan on events_1400285 events -> Seq Scan on events_1400285 events
Filter: ((event_type)::text = ANY ('{click,submit,pay}'::text[])) Filter: ((event_type)::text = ANY ('{click,submit,pay}'::text[]))
SELECT success FROM run_command_on_workers('alter system set enable_nestloop to off');
t
t
SELECT success FROM run_command_on_workers('alter system set enable_sort to off');
t
t
SELECT success FROM run_command_on_workers('select pg_reload_conf()');
t
t
-- Union and left join subquery pushdown -- Union and left join subquery pushdown
EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF)
SELECT SELECT
@ -741,41 +750,38 @@ HashAggregate
Tasks Shown: One of 4 Tasks Shown: One of 4
-> Task -> Task
Node: host=localhost port=xxxxx dbname=regression Node: host=localhost port=xxxxx dbname=regression
-> GroupAggregate -> HashAggregate
Group Key: subquery_top.hasdone Group Key: COALESCE(subquery_2.hasdone, 'Has not done paying'::text)
-> Sort -> GroupAggregate
Sort Key: subquery_top.hasdone Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone
-> Subquery Scan on subquery_top -> Sort
-> GroupAggregate Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone -> Hash Left Join
-> Sort Hash Cond: (users.composite_id = subquery_2.composite_id)
Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone -> HashAggregate
-> Hash Left Join Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
Hash Cond: (users.composite_id = subquery_2.composite_id) -> Append
-> HashAggregate -> Hash Join
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time Hash Cond: (users.composite_id = events.composite_id)
-> Append -> Seq Scan on users_1400289 users
-> Hash Join Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
Hash Cond: (users.composite_id = events.composite_id) -> Hash
-> Seq Scan on users_1400289 users -> Seq Scan on events_1400285 events
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) Filter: ((event_type)::text = 'click'::text)
-> Hash -> Hash Join
-> Seq Scan on events_1400285 events Hash Cond: (users_1.composite_id = events_1.composite_id)
Filter: ((event_type)::text = 'click'::text) -> Seq Scan on users_1400289 users_1
-> Hash Join Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
Hash Cond: (users_1.composite_id = events_1.composite_id) -> Hash
-> Seq Scan on users_1400289 users_1 -> Seq Scan on events_1400285 events_1
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) Filter: ((event_type)::text = 'submit'::text)
-> Hash -> Hash
-> Seq Scan on events_1400285 events_1 -> Subquery Scan on subquery_2
Filter: ((event_type)::text = 'submit'::text) -> Unique
-> Hash -> Sort
-> Subquery Scan on subquery_2 Sort Key: ((events_2.composite_id).tenant_id), ((events_2.composite_id).user_id)
-> Unique -> Seq Scan on events_1400285 events_2
-> Sort Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
Sort Key: ((events_2.composite_id).tenant_id), ((events_2.composite_id).user_id)
-> Seq Scan on events_1400285 events_2
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-- Union, left join and having subquery pushdown -- Union, left join and having subquery pushdown
EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF)
SELECT SELECT
@ -856,44 +862,48 @@ Sort
Tasks Shown: One of 4 Tasks Shown: One of 4
-> Task -> Task
Node: host=localhost port=xxxxx dbname=regression Node: host=localhost port=xxxxx dbname=regression
-> GroupAggregate -> HashAggregate
Group Key: subquery_top.count_pay Group Key: COALESCE(subquery_2.count_pay, '0'::bigint)
-> Sort -> GroupAggregate
Sort Key: subquery_top.count_pay Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay
-> Subquery Scan on subquery_top Filter: (array_ndims(array_agg(('action=>1'::text) ORDER BY events.event_time)) > 0)
-> GroupAggregate -> Sort
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay
Filter: (array_ndims(array_agg(('action=>1'::text) ORDER BY events.event_time)) > 0) -> Hash Left Join
-> Sort Hash Cond: (users.composite_id = subquery_2.composite_id)
Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.count_pay -> HashAggregate
-> Hash Left Join Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
Hash Cond: (users.composite_id = subquery_2.composite_id) -> Append
-> Hash Join
Hash Cond: (users.composite_id = events.composite_id)
-> Seq Scan on users_1400289 users
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
-> Hash
-> Seq Scan on events_1400285 events
Filter: ((event_type)::text = 'click'::text)
-> Hash Join
Hash Cond: (users_1.composite_id = events_1.composite_id)
-> Seq Scan on users_1400289 users_1
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type))
-> Hash
-> Seq Scan on events_1400285 events_1
Filter: ((event_type)::text = 'submit'::text)
-> Hash
-> Subquery Scan on subquery_2
-> HashAggregate -> HashAggregate
Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time Group Key: events_2.composite_id
-> Append Filter: (count(*) > 2)
-> Hash Join -> Seq Scan on events_1400285 events_2
Hash Cond: (users.composite_id = events.composite_id) Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-> Seq Scan on users_1400289 users SELECT success FROM run_command_on_workers('alter system reset enable_nestloop');
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) t
-> Hash t
-> Seq Scan on events_1400285 events SELECT success FROM run_command_on_workers('alter system reset enable_sort');
Filter: ((event_type)::text = 'click'::text) t
-> Hash Join t
Hash Cond: (users_1.composite_id = events_1.composite_id) SELECT success FROM run_command_on_workers('select pg_reload_conf()');
-> Seq Scan on users_1400289 users_1 t
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type)) t
-> Hash
-> Seq Scan on events_1400285 events_1
Filter: ((event_type)::text = 'submit'::text)
-> Hash
-> Subquery Scan on subquery_2
-> GroupAggregate
Group Key: events_2.composite_id
Filter: (count(*) > 2)
-> Sort
Sort Key: events_2.composite_id
-> Seq Scan on events_1400285 events_2
Filter: ((composite_id >= '(1,-9223372036854775808)'::user_composite_type) AND (composite_id <= '(1,9223372036854775807)'::user_composite_type) AND ((event_type)::text = 'pay'::text))
-- Lateral join subquery pushdown -- Lateral join subquery pushdown
-- set subquery_pushdown due to limit in the query -- set subquery_pushdown due to limit in the query
SET citus.subquery_pushdown to ON; SET citus.subquery_pushdown to ON;

View File

@ -260,6 +260,10 @@ FROM
tenant_id, tenant_id,
user_id) AS subquery; user_id) AS subquery;
SELECT success FROM run_command_on_workers('alter system set enable_nestloop to off');
SELECT success FROM run_command_on_workers('alter system set enable_sort to off');
SELECT success FROM run_command_on_workers('select pg_reload_conf()');
-- Union and left join subquery pushdown -- Union and left join subquery pushdown
EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF)
SELECT SELECT
@ -396,6 +400,10 @@ GROUP BY
ORDER BY ORDER BY
count_pay; count_pay;
SELECT success FROM run_command_on_workers('alter system reset enable_nestloop');
SELECT success FROM run_command_on_workers('alter system reset enable_sort');
SELECT success FROM run_command_on_workers('select pg_reload_conf()');
-- Lateral join subquery pushdown -- Lateral join subquery pushdown
-- set subquery_pushdown due to limit in the query -- set subquery_pushdown due to limit in the query
SET citus.subquery_pushdown to ON; SET citus.subquery_pushdown to ON;