Commit Graph

7126 Commits (onur-leftjoin_push-improvements)

Author SHA1 Message Date
Onur Tirtir 7534345984 suggestions to improve debug messages 2025-08-18 12:55:01 +03:00
Onur Tirtir b84cc4b825 add a comment 2025-08-18 10:47:56 +03:00
Onur Tirtir 2cbea6528f rename a few more functions 2025-08-18 10:33:54 +03:00
eaydingol 08ae68e5db review 2025-08-16 00:08:50 +03:00
eaydingol f1181db55c check query plan during physical planning for outer joins 2025-08-16 00:08:50 +03:00
eaydingol c2c6f7b910 fix style 2025-08-16 00:08:50 +03:00
eaydingol a79b4b31c9 Improve lateral check and add test cases 2025-08-16 00:08:50 +03:00
ebruaydingol 7a5f5c1c08 Add more tests 2025-08-16 00:08:50 +03:00
eaydingol a964d45323 Update comments, checks 2025-08-16 00:08:50 +03:00
eaydingol 13405e7871 Run update quals code only for queries with outer joins, update planners for error handling, update test cases. 2025-08-16 00:08:50 +03:00
eaydingol b8fd1e64ff Fix tests for pg15 2025-08-16 00:08:50 +03:00
eaydingol 4fbfff9cc1 Clean up local checks 2025-08-16 00:08:50 +03:00
eaydingol 0cf34388e2 Update test cases w.r.to 8113 2025-08-16 00:08:50 +03:00
eaydingol 55bbf08c81 Test dependency and reindent 2025-08-16 00:08:50 +03:00
eaydingol c465be2a30 Update test case 2025-08-16 00:08:50 +03:00
eaydingol 13a0939ca4 Update test cases 2025-08-16 00:08:50 +03:00
eaydingol efbf6a7bc2 Fix test dependencies 2025-08-16 00:08:50 +03:00
eaydingol e58d730e64 Extend the logic to right joins 2025-08-16 00:08:50 +03:00
eaydingol 49d163fb63 Update test cases 2025-08-16 00:08:50 +03:00
eaydingol 4f618a355d Disable recurring/non-recurring outer join push down for lateral joins, update test cases 2025-08-16 00:08:50 +03:00
eaydingol a850c3e3a5 Refactor the update part 2025-08-16 00:08:50 +03:00
eaydingol 7a89dad904 Fix task computation 2025-08-16 00:08:50 +03:00
eaydingol 188043c5e7 Add a guc for the feature 2025-08-16 00:08:50 +03:00
eaydingol fc109f408b reindent 2025-08-16 00:08:50 +03:00
eaydingol 665ae75a65 test 2025-08-16 00:08:50 +03:00
eaydingol bca534cb3e test 2025-08-16 00:08:50 +03:00
eaydingol afd3bd921d Revert ununsed helper functions after refactor 2025-08-16 00:08:50 +03:00
eaydingol 245b9534e2 More tests 2025-08-16 00:08:50 +03:00
eaydingol 9f72067c12 Use quals for both on and using syntax. 2025-08-16 00:08:50 +03:00
eaydingol 96a7d70fa9 update tests 2025-08-16 00:08:50 +03:00
eaydingol cd9f6ae313 Enable push down only for reference table 2025-08-16 00:08:50 +03:00
eaydingol c08c1eb299 Introduce a method to check if the left join is safe to push-down, use the same method to compute constraints for push 2025-08-16 00:08:50 +03:00
eaydingol 1060de98d3 Refactor the code, add checks for the join clause, add basic test cases. 2025-08-16 00:08:50 +03:00
eaydingol fccc4324a6 Check if the outer table has distribution column, wip, still need to check the using clause or join constraint 2025-08-16 00:08:50 +03:00
eaydingol f42c8edd43 list case 2025-08-16 00:08:50 +03:00
eaydingol ef362f403b Do not push down the left join for nested cases. 2025-08-16 00:08:50 +03:00
eaydingol 5e57908f5b Check outer entry in left join. 2025-08-16 00:08:50 +03:00
eaydingol 53f2506901 push down left join, wip 2025-08-16 00:08:50 +03:00
ibrahim halatci 649050c676
results of extension compatibility testing (#8048)
List of extensions that are verified to be working with Citus, and some
special cases that needs attention. Thanks to the efforts of @emelsimsek
, @m3hm3t , @alperkocatas , @eaydingol
2025-08-15 15:32:29 +03:00
ibrahim halatci cf9a4476e0
Merge branch 'main' into ihalatci-extension-compat-test-report 2025-08-13 19:27:45 +03:00
ibrahim halatci f73da1ed40
Refactor background worker setup for security improvements (#8078)
Enhance security by addressing a code scanning alert and refactoring the
background worker setup code for better maintainability and clarity.

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-08-13 19:25:31 +03:00
Mehmet YILMAZ 41883cea38
PG18 - unify psql headings to ‘List of relations’ (#8119)
fixes #8110 

This patch updates the `normalize.sed` script used in pg18 psql
regression tests:

- Replaces the headings “List of tables”, “List of indexes”, and “List
of sequences” with a single, uniform heading: “List of relations”.
2025-08-13 12:22:23 +03:00
Mehmet YILMAZ bfc6d1f440
PG18 - Adjust EXPLAIN's output for disabled nodes (#8108)
fixes #8097
2025-08-12 12:38:19 +03:00
Mehmet YILMAZ a6161f5a21
Fix CTE traversal for outer Vars in FindReferencedTableColumn (remove assert; correct parentQueryList handling) (#8106)
fixes #8105 

This change lets `FindReferencedTableColumn()` correctly resolve columns
through a CTE even when the expression comes from an outer query level
(`varlevelsup > 0`, `skipOuterVars = false`). Before, we hit an
`Assert(skipOuterVars)` in this path.

**Problem**

* Hitting a CTE after walking outer Vars triggered
`Assert(skipOuterVars)`.
* Cause: we modified `parentQueryList` in place and didn’t rebuild the
correct parent chain before recursing into the CTE, so the path was
considered unsafe.

**Fix**

* Remove the `Assert(skipOuterVars)` in the `RTE_CTE` branch.
* Find the CTE’s owning level via `ctelevelsup` and compute
`cteParentListIndex`.
* Rebuild a private parent list for recursion: `list_copy` →
`list_truncate` → `lappend(current query)`.
* Add a bounds check before indexing the CTE’s `targetList`.

**Why it works**


```diff
-parentQueryList = lappend(parentQueryList, query);
-FindReferencedTableColumn(targetEntry->expr, parentQueryList,
-                          cteQuery, column, rteContainingReferencedColumn,
-                          skipOuterVars);
+    /* hand a private, bounded parent list to the recursion */
+    List *newParent = list_copy(parentQueryList);
+    newParent = list_truncate(newParent, cteParentListIndex + 1);
+    newParent = lappend(newParent, query);
+
+    FindReferencedTableColumn(targetEntry->expr,
+                              newParent,
+                              cteQuery,
+                              column,
+                              rteContainingReferencedColumn,
+                              skipOuterVars);
+}


```
**Before:** We changed `parentQueryList` in place (`parentQueryList =
lappend(...)`) and didn’t trim it to the CTE’s owner level.

**After:** We copy the list, trim it to the CTE’s owner level, then
append the current query. This keeps the parent list accurate for the
current recursion and safe when following outer Vars.


**Example: Nested subquery referencing the CTE (two levels down)**

```
WITH c AS MATERIALIZED (SELECT user_id FROM raw_events_first)
SELECT 1
FROM raw_events_first t
WHERE EXISTS (
  SELECT 1
  FROM (SELECT user_id FROM c) c2
  WHERE c2.user_id = t.user_id
);
```

Levels:
Q0 = top SELECT
Q1 = EXISTS subquery
Q2 = inner (SELECT user_id FROM c)

When resolving c2.user_id inside Q2:

- parentQueryList is [Q0, Q1, Q2].
- `ctelevelsup`: 2


`cteParentListIndex = length(parentQueryList) - ctelevelsup - 1`

- Recurse into the CTE’s query with [Q0, Q2].


**Tests (added in `multi_insert_select`)**

* **T1:** Correlated subquery that references a CTE (one level down) 
Verifies that resolving through `RTE_CTE` after following an outer `Var`
succeeds, row count matches source table.
* **T2:** Nested subquery that references a CTE (two levels down) 
Exercises deeper recursion and confirms identical to T1.
* **T3:** Scalar subquery in a target list that reads from the outer CTE
Checks expected row count and that no NULLs are inserted.

These tests cover the cases that previously hit `Assert(skipOuterVars)`
and confirm CTE references while following outer Vars.
2025-08-12 11:49:50 +03:00
Karina 71d6328378
Fix memory corruptions around pg_dist_background_task accessors after a Citus downgrade is followed by an upgrade (#8114)
DESCRIPTION: Fixes potential memory corruptions that could happen when
accessing pg_dist_background_task after a Citus downgrade is followed by
a Citus upgrade.

In case of Citus downgrade and further upgrade an undefined behavior may
be encountered. The reason is that Citus hardcoded the number of columns
in the extension's tables, but in case of downgrade and following update
some of these tables can have more columns, and some of them can be
marked as dropped.

This PR fixes all such tables using the approach introduced in #7950,
which solved the problem for the pg_dist_partition table.

See #7515 for a more thorough explanation.

---------

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
2025-08-11 18:34:06 +03:00
Mehmet YILMAZ 6b6d959fac
PG18 - pg17.sql Simplify step 10 verification to use COUNT(*) instead of SELECT * (#8111)
fixes #8096 

PostgreSQL 18 adds a `conenforced` flag allowing `CHECK` constraints to
be declared `NOT ENFORCED`.



ca87c415e2
```diff
@@ -1256,26 +1278,26 @@
  distributed_partitioned_table_id_partition_col_excl | x
 (2 rows)
 
 -- Step 9: Drop the exclusion constraints from both tables
 \c - - :master_host :master_port
 SET search_path TO pg17;
 ALTER TABLE distributed_partitioned_table DROP CONSTRAINT dist_exclude_named;
 ALTER TABLE local_partitioned_table DROP CONSTRAINT local_exclude_named;
 -- Step 10: Verify the constraints were dropped
 SELECT * FROM pg_constraint WHERE conname = 'dist_exclude_named' AND contype = 'x';
- oid | conname | connamespace | contype | condeferrable | condeferred | convalidated | conrelid | contypid | conindid | conparentid | confrelid | confupdtype | confdeltype | confmatchtype | conislocal | coninhcount | connoinherit | conkey | confkey | conpfeqop | conppeqop | conffeqop | confdelsetcols | conexclop | conbin
+ oid | conname | connamespace | contype | condeferrable | condeferred | conenforced | convalidated | conrelid | contypid | conindid | conparentid | confrelid | confupdtype | confdeltype | confmatchtype | conislocal | coninhcount | connoinherit | conperiod | conkey | confkey | conpfeqop | conppeqop | conffeqop | confdelsetcols | conexclop | conbin 
 -----+---------+--------------+---------+---------------+-------------+-------------+--------------+----------+----------+----------+-------------+-----------+-------------+-------------+---------------+------------+-------------+--------------+-----------+--------+---------+-----------+-----------+-----------+----------------+-----------+--------
 (0 rows)
 
 SELECT * FROM pg_constraint WHERE conname = 'local_exclude_named' AND contype = 'x';
- oid | conname | connamespace | contype | condeferrable | condeferred | convalidated | conrelid | contypid | conindid | conparentid | confrelid | confupdtype | confdeltype | confmatchtype | conislocal | coninhcount | connoinherit | conkey | confkey | conpfeqop | conppeqop | conffeqop | confdelsetcols | conexclop | conbin
+ oid | conname | connamespace | contype | condeferrable | condeferred | conenforced | convalidated | conrelid | contypid | conindid | conparentid | confrelid | confupdtype | confdeltype | confmatchtype | conislocal | coninhcount | connoinherit | conperiod | conkey | confkey | conpfeqop | conppeqop | conffeqop | confdelsetcols | conexclop | conbin 
 -----+---------+--------------+---------+---------------+-------------+-------------+--------------+----------+----------+----------+-------------+-----------+-------------+-------------+---------------+------------+-------------+--------------+-----------+--------+---------+-----------+-----------+-----------+----------------+-----------+--------
 (0 rows)
 
```

The purpose of step 10 is merely to confirm that the exclusion
constraints dist_exclude_named and local_exclude_named have been
dropped. There’s no need to pull back every column from pg_constraint—we
only care about whether any matching row remains.

- Reduces noise in the output
- Eliminates dependence on the full set of pg_constraint columns (which
can drift across Postgres versions)
- Resolves the pg18 regression diff without altering test expectations
2025-08-08 13:46:11 +03:00
ibrahim halatci 26409f6400
Update EXTENSION_COMPATIBILITY.md 2025-08-07 14:57:45 +03:00
ibrahim halatci dbf0e647a9
Merge branch 'main' into ihalatci-extension-compat-test-report 2025-08-07 14:35:34 +03:00
eaydingol 3d8fd337e5
Check outer table partition column (#8092)
DESCRIPTION: Introduce a new check to push down a query including union
and outer join to fix #8091 .

In "SafeToPushdownUnionSubquery", we check if the distribution column of
the outer relation is in the target list.
2025-08-06 16:13:14 +03:00
manaldush f0789bd388
Fix memory corruptions that could happen when a Citus downgrade is followed by an upgrade (#7950)
DESCRIPTION: Fixes potential memory corruptions that could happen when a
Citus downgrade is followed by a Citus upgrade.

In case of citus downgrade and further upgrade citus crash with core
dump.
The reason is that citus hardcoded number of columns in
pg_dist_partition table,
but in case of downgrade and following update table can have more
columns, and
some of then can be marked as dropped.

Patch suggest decision for this problem with using
tupleDescriptor->nattrs(postgres internal approach).

Fixes #7933.

---------

Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
2025-08-05 10:03:35 +00:00