Commit Graph

6836 Commits (naisila/test-upgrade)

Author SHA1 Message Date
naisila c4734450d1 Try citus upgrade 2025-01-24 17:35:56 +03:00
Naisila Puka 548395fd77
fix changelog date (#7859) 2025-01-22 14:28:46 +03:00
Naisila Puka cba8e57737
Changelog entries for 13.0.0 (#7858) 2025-01-22 13:17:35 +03:00
Naisila Puka 23d5207701
Fix pg17 test (#7857)
error merged in
ab7c3b7804
2025-01-22 12:54:52 +03:00
Mehmet YILMAZ ab7c3b7804
PG17 Compatibility - Fix crash when pg_class is used in MERGE (#7853)
This pull request addresses Issue #7846, where specific MERGE queries on
non-distributed and distributed tables can result in crashes in certain
scenarios. The issue stems from the usage of `pg_class` catalog table,
and the `FilterShardsFromPgclass` function in Citus. This function goes
through the query's jointree to hide the shards. However, in PG17,
MERGE's join quals are in a separate structure called
`mergeJoinCondition`. Therefore FilterShardsFromPgclass was not
filtering correctly in a `MERGE` command that involves `pg_class`. To
fix the issue, we handle `mergeJoinCondition` separately in PG17.

Relevant PG commit:

0294df2f1f

**Non-Distributed Tables:**
A MERGE query involving a non-distributed table using
`pg_catalog.pg_class` as the source may execute successfully but needs
testing to ensure stability.

**Distributed Tables:**
Performing a MERGE on a distributed table using `pg_catalog.pg_class` as
the source raises an error:
`ERROR: MERGE INTO a distributed table from Postgres table is not yet
supported`
However, in some cases, this can lead to a server crash if the
unsupported operation is not properly handled.

This is the test output from the same test conducted prior to the code
changes being implemented.

```
-- Issue #7846: Test crash scenarios with MERGE on non-distributed and distributed tables
-- Step 1: Connect to a worker node to verify shard visibility
\c postgresql://postgres@localhost::worker_1_port/regression?application_name=psql
SET search_path TO pg17;
-- Step 2: Create and test a non-distributed table
CREATE TABLE non_dist_table_12345 (id INTEGER);
-- Test MERGE on the non-distributed table
MERGE INTO non_dist_table_12345 AS target_0
USING pg_catalog.pg_class AS ref_0
ON target_0.id = ref_0.relpages
WHEN NOT MATCHED THEN DO NOTHING;
SSL SYSCALL error: EOF detected
connection to server was lost
```
2025-01-21 17:48:06 +03:00
Colm c2bc7aca4a
Update tdigest_aggregate_support output for PG15+ (#7849)
Regress test tdigest_aggregate_support has been failing since at least
Citus 12.0, when tdigest extension is installed in Postgres. This
appears to be because of an omission by commit 03832f3 and a change in
the implementation of Postgres random() function (pg commit
[d4f109e4a](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d4f109e4a)).
To reproduce the test diff:
- Checkout [tdigest ](https://github.com/tvondra/tdigest)and run `make;
make install`
- In citus regress directory run `make check-multi` or
`./citus_tests/run_test.py tdigest_aggregate_support`

There are two parts to this commit:

1. Revert `Output: xxxxx` in EXPLAIN VERBOSE. Citus commit fe4ac51
normalized EXPLAIN VERBOSE output because of a change between pg12 and
pg13. When pg12 support was no longer required, the rule was removed
from normalize.sed and `Output: xxxx` was reverted in the impacted
regress output files (03832f3), but `tdigest_aggregate_support` was
omitted.

2. Adjust the query results; the tdigest_aggregate_support test file has
a comment _verifying results - should be stable due to seed while
inserting the data, if failure due to data these queries could be
removed or check for certain ranges_ but the result values in this
commit are consistent across citus 12.0 (pg 15), citus 12.1 (pg 16) and
citus 13.0 (pg 17), or since the Postgres changed their [implementation
of
random](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d4f109e4a),
so proposing to go with these results.
2025-01-20 22:00:33 +00:00
Naisila Puka fa8e867662
Bump to latest PG minors 17.2, 16.6, 15.10, 14.15 (#7843)
Similar to
5ef2cd67ed,
we use the commit sha of a local build of the images, pushed.
2025-01-13 22:35:11 +03:00
Emel Şimşek c55bc8c669 Propagates SECURITY LABEL ON ROLE stmt (#7304) (#7735)
Propagates SECURITY LABEL ON ROLE stmt (https://github.com/citusdata/citus/pull/7304)
We propagate `SECURITY LABEL [for provider] ON ROLE rolename IS
labelname` to the worker nodes.
We also make sure to run the relevant `SecLabelStmt` commands on a
newly added node by looking at roles found in `pg_shseclabel`.

See official docs for explanation on how this command works:
https://www.postgresql.org/docs/current/sql-security-label.html
This command stores the role label in the `pg_shseclabel` catalog table.

This commit also fixes the regex string in
`check_gucs_are_alphabetically_sorted.sh` script such that it escapes
the dot. Previously it was looking for all strings starting with "citus"
instead of "citus." as it should.

To test this feature, I currently make use of a special GUC to control
label provider registration in PG_init when creating the Citus extension.

(cherry picked from commit 0d1f18862b)

Co-authored-by: Naisila Puka <37271756+naisila@users.noreply.github.com>
(cherry picked from commit 686d2b46ca)
2025-01-13 19:56:01 +03:00
Nils Dijk 7e316c90c4 Shard moves/isolate report LSN's in lsn format (#7227)
DESCRIPTION: Shard moves/isolate report LSN's in lsn format

While investigating an issue with our catchup mechanism on certain
postgres versions we noticed we print LSN's in the format of the native
long type. This is an uncommon representation for LSN's in postgres
logs.

This patch changes the output of our log message to go from the long
type representation to the native LSN type representation. Making it
easier for postgres users to recognize and compare LSN's with other
related reports.

example of new output:
```
2023-09-25 17:28:47.544 CEST [11345] LOG:  The LSN of the target subscriptions on node localhost:9701 have increased from 0/0 to 0/E1ED20F8 at 2023-09-25 17:28:47.544165+02 where the source LSN is 1/415DCAD0
```

(cherry picked from commit b87fbcbf79)
2025-01-13 17:47:47 +03:00
Teja Mupparti d2ca63fb8c For scenarios, such as, Bug 3697586: Server crashes when assigning distributed transaction: Raise an ERROR instead of a crash
(cherry picked from commit ab7c13beb5)
2025-01-13 17:47:18 +03:00
Onur Tirtir a19e180212 Avoid re-assigning the global pid for client backends and bg workers when the application_name changes (#7791)
DESCRIPTION: Fixes a crash that happens because of unsafe catalog access
when re-assigning the global pid after application_name changes.

When application_name changes, we don't actually need to
try re-assigning the global pid for external client backends because
application_name doesn't affect the global pid for such backends. Plus,
trying to re-assign the global pid for external client backends would
unnecessarily cause performing a catalog access when the cached local
node id is invalidated. However, accessing to the catalog tables is
dangerous in certain situations like when we're not in a transaction
block. And for the other types of backends, i.e., the Citus internal
backends, we need to re-assign the global pid when the application_name
changes because for such backends we simply extract the global pid
inherited from the originating backend from the application_name -that's
specified by originating backend when openning that connection- and this
doesn't require catalog access.

(cherry picked from commit 73411915a4)
2025-01-13 17:47:11 +03:00
Pavel Seleznev cdded256ef Remove warnings on some builds (#7680)
Co-authored-by: Pavel Seleznev <PNSeleznev@sberbank.ru>
(cherry picked from commit fe6d198ab2)
2025-01-13 15:22:13 +03:00
Colm McHugh 353033d3f0 [Bug Fix] [SEGFAULT] Querying distributed tables with window partition may cause segfault #7705
In function MasterAggregateMutator(), when the original Node is a Var node use makeVar() instead
of copyObject() when constructing the Var node for the target list of the combine query.
The varnullingrels field of the original Var node is ignored because it is not relevant for the
combine query; copying this cause the problem in issue 7705, where a coordinator query had
a Var with a reference to a non-existent join relation.

(cherry picked from commit c52f36019f)
2025-01-13 15:21:50 +03:00
Parag Jain 1893d9a900 [Bug Fix] : writing incorrect data to target Merge repartition Command (#7659)
We were writing incorrect data to target collection in some cases of merge command. In case of repartition when source query is RELATION. We were referring to incorrect attribute number that was resulting into
this incorrect behavior.

Example :

![image](https://github.com/user-attachments/assets/a101cb36-7976-459c-befb-96a55a5b3dc1)

![image](https://github.com/user-attachments/assets/e5c83b7b-5b8e-4d79-a927-95684dc9ba49)

I have added fixed tests as part of this PR , Thanks.

(cherry picked from commit 5bad6c6a1d)
2025-01-13 15:21:37 +03:00
Mehmet YILMAZ 063cff908e Fix race condition in citus_set_coordinator_host when adding multiple coordinator nodes concurrently (#7682)
When multiple sessions concurrently attempt to add the same coordinator
node using `citus_set_coordinator_host`, there is a potential race
condition. Both sessions may pass the initial metadata check
(`isCoordinatorInMetadata`), but only one will succeed in adding the
node. The other session will fail with an assertion error
(`Assert(!nodeAlreadyExists)`), causing the server to crash. Even though
the `AddNodeMetadata` function takes an exclusive lock, it appears that
the lock is not preventing the race condition before the initial
metadata check.

- **Issue**: The current logic allows concurrent sessions to pass the
check for existing coordinators, leading to an attempt to insert
duplicate nodes, which triggers the assertion failure.

- **Impact**: This race condition leads to crashes during operations
that involve concurrent coordinator additions, as seen in
https://github.com/citusdata/citus/issues/7646.

**Test Plan:**

- Isolation Test Limitation: An isolation test was added to simulate
concurrent additions of the same coordinator node, but due to the
behavior of PostgreSQL locking mechanisms, the test does not trigger the
edge case. The lock applied within the function serializes the
operations, preventing the race condition from occurring in the
isolation test environment.
While the edge case is difficult to reproduce in an isolation test, the
fix addresses the core issue by ensuring concurrency control through
proper locking.

- Existing Tests: All existing tests related to node metadata and
coordinator management have been run to ensure that no regressions were
introduced.

**After the Fix:**

- Concurrent attempts to add the same coordinator node will be
serialized. One session will succeed in adding the node, while the
others will skip the operation without crashing the server.

Co-authored-by: Mehmet YILMAZ <mehmet.yilmaz@microsoft.com>
(cherry picked from commit 4775715691)
2025-01-13 15:20:03 +03:00
eaydingol a115b3043a Check if the limit is null (#7665)
DESCRIPTION: Add a check to see if the given limit is null. 

Fixes a bug by checking if the limit given in the query is null when the
actual limit is computed with respect to the given offset.
Prior to this change, null is interpreted as 0 during the limit
calculation when both limit and offset are given.

Fixes #7663
2025-01-12 23:59:17 +03:00
Evgeny Nechayev 5eb037ac50 Use macro wrapper to access PGPROC data, which allow to improve compa… (#7607)
DESCRIPTION: Use macro wrapper to access PGPROC data, to improve compatibility with PostgreSQL forks.
2025-01-12 23:49:21 +03:00
Onur Tirtir ef946e44af Fix incorrect "VALID UNTIL" assumption made for roles in node activation (#7534)
Fixes https://github.com/citusdata/citus/issues/7533.

DESCRIPTION: Fixes incorrect `VALID UNTIL` setting assumption made for
roles when syncing them to new nodes
2025-01-12 23:46:20 +03:00
Karina 89ccafff50 Fix server crash when trying to execute activate_node_snapshot() on a single-node cluster (#7552)
This fixes #7551 reported by Egor Chindyaskin

Function activate_node_snapshot() is not meant to be called on a cluster
without worker nodes. This commit adds ERROR report for such case to
prevent server crash.
2025-01-12 23:44:13 +03:00
eaydingol e2a03ce6a0 Fix: store the previous shard cost for order verification (#7550)
Store the previous shard cost so that the invariant checking performs as
expected.
2025-01-12 23:43:53 +03:00
Filip Sedlák 657575114c Fail early when shard can't be safely moved to a new node (#7467)
DESCRIPTION: citus_move_shard_placement now fails early when shard
cannot be safely moved

The implementation is quite simplistic -
`citus_move_shard_placement(...)` will fail with an error if there's any
new node in the cluster that doesn't have reference tables yet.

It could have been finer-grained, i.e. erroring only when trying to move
a shard to an unitialized node. Looking at the related functions -
`replicate_reference_tables()` or `citus_rebalance_start()`, I think
it's acceptable behaviour. These other functions also treat "any"
unitialized node as a temporary anomaly.

Fixes #7426

---------

Co-authored-by: Jelte Fennema-Nio <jelte.fennema@microsoft.com>
2025-01-12 22:54:09 +03:00
eaydingol ddef97212a Generate qualified relation name (#7427)
This change refactors the code by using generate_qualified_relation_name
from id instead of using a sequence of functions to generate the
relation name.

Fixes #6602

(cherry picked from commit ee11492a0e)
2025-01-12 22:08:41 +03:00
zhjwpku f42e8556ec [performance improvement] remove duplicate LoadShardList call (#7380)
LoadShardList is called twice, which is not neccessary, and there is no
need to sort the shard placement list since we only want to know the list
length.

(cherry picked from commit 8e979f7ac6)
2025-01-12 22:08:03 +03:00
Karina 01945bd3a3 Fix getting heap tuple size (#7387)
This fixes #7230.

First of all, using HeapTupleHeaderGetDatumLength(heapTuple) is
definetly wrong, it gives a number that's 4 times less than the correct
tuple size (heapTuple.t_len). See

https://github.com/postgres/postgres/blob/REL_16_0/src/include/access/htup_details.h#L455-L456

https://github.com/postgres/postgres/blob/REL_16_0/src/include/varatt.h#L279

https://github.com/postgres/postgres/blob/REL_16_0/src/include/varatt.h#L225-L226

When I fixed it, the limit_intermediate_size test failed, so I tried to
understand what's going on there. In original commit fd546cf these
queries were supposed to fail. Then in b3af63c three of the queries that
were supposed to fail suddenly worked and tests were changed to pass
without understanding why the output had changed or how to keep test
testing what it had to test. Even comments saying that these queries
should fail were left untouched. Commit message gives no clue about why
exactly test has changed:

> It seems that when we use adaptive executor instead of task tracker,
we
> exceed the intermediate result size less in the test. Therefore
updated
> the tests accordingly.

Then 3fda2c3 also blindly raised the limit for one of the queries to
keep it working:

3fda2c3254 (diff-a9b7b617f9dfd345318cb8987d5897143ca1b723c87b81049bbadd94dcc86570R19)

When in fe3caf3 that HeapTupleHeaderGetDatumLength(heapTuple) call was
finally added, one of those test queries became failing again.

The other two of them now also failing after the fix. I don't understand
how exactly the calculation of "intermediate result size" that is
limited by citus.max_intermediate_result_size had changed through
b3af63c and fe3caf3, but these numbers are now closer to what
they originally were when this limitation was added in
fd546cf. So these queries should fail, like in the original
version of the limit_intermediate_size test.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 20dc58cf5d)
2025-01-12 22:06:46 +03:00
Cédric Villemain 1a8349aeaf Fix #7242, CALL(@0) crash backend (#7288)
When executing a prepared CALL, which is not pure SQL but available with
some drivers like npgsql and jpgdbc, Citus entered a code path where a
plan is not defined, while trying to increase its cost. Thus SIG11 when
plan is a NULL pointer.

Fix by only increasing plan cost when plan is not null.

However, it is a bit suspicious to get here with a NULL plan and maybe a
better change will be to not call
ShardPlacementForFunctionColocatedWithDistTable() with a NULL plan at
all (in call.c:134)

bug hit with for example:
```
CallableStatement proc = con.prepareCall("{CALL p(?)}");
proc.registerOutParameter(1, java.sql.Types.BIGINT);
proc.setInt(1, -100);
proc.execute();
```

where `p(bigint)` is a distributed "function" and the param the
distribution key (also in a distributed table), see #7242 for details

Fixes #7242

(cherry picked from commit 0678a2fd89)
2025-01-12 22:06:08 +03:00
Emel Şimşek f18c21c2b4 Send keepalive messages in split decoder periodically to avoid wal receiver timeouts during large shard splits. (#7229)
DESCRIPTION: Send keepalive messages during the logical replication
phase of large shard splits to avoid timeouts.

During the logical replication part of the shard split process, split
decoder filters out the wal records produced by the initial copy. If the
number of wal records is big, then split decoder ends up processing for
a long time before sending out any wal records through pgoutput. Hence
the wal receiver may time out and restarts repeatedly causing our split
driver code catch up logic to fail.

Notes:

1. If the wal_receiver_timeout is set to a very small number e.g. 600ms,
it may time out before receiving the keepalives. My tests show that this
code works best when the` wal_receiver_timeout `is set to 1minute, which
is the default value.

2. Once a logical replication worker time outs, a new one gets launched.
The new logical replication worker sets the pg_stat_subscription columns
to initial values. E.g. the latest_end_lsn is set to 0. Our driver logic
in `WaitForGroupedLogicalRepTargetsToCatchUp` can not handle LSN value
to go back. This is the main reason for it to get stuck in the infinite
loop.

(cherry picked from commit e9035f6d32)
2025-01-12 22:05:33 +03:00
Naisila Puka 3e924db90a
Propagate MERGE ... WHEN NOT MATCHED BY SOURCE (#7807)
DESCRIPTION: Propagates MERGE ... WHEN NOT MATCHED BY SOURCE

It seems like there is not much needed to be done here.
`get_merge_query_def` from `ruleutils_17` is updated with "WHEN NOT
MATCHED BY SOURCE" therefore `deparse_shard_query` parses the merge
query for execution on the shard correctly.

Relevant PG commit:
https://github.com/postgres/postgres/commit/0294df2f1
2025-01-09 00:03:06 +03:00
Naisila Puka 32c058e448
Remove Debian Buster support from packaging pipelines (#7828) (#7837)
Remove Debian Buster support from packaging-test-pipelines

Co-authored-by: Gürkan İndibay <gindibay@microsoft.com>
(cherry picked from commit 70f84e4aee)

Co-authored-by: Seda Gündoğdu <69769369+sedagundogdu@users.noreply.github.com>
2025-01-08 23:37:47 +03:00
Naisila Puka 9bad33f89d
PG17 - Propagate EXPLAIN options: MEMORY and SERIALIZE (#7802)
DESCRIPTION: Propagates MEMORY and SERIALIZE options of EXPLAIN

The options for `MEMORY` can be true or false. Default is false.
The options for `SERIALIZE` can be none, text or binary. Default is
none.

I referred to how we added support for WAL option in this PR [Support
EXPLAIN(ANALYZE, WAL)](https://github.com/citusdata/citus/pull/4196).
For the tests however, I used the same tests as Postgres, not like the
tests in the WAL PR. I used exactly the same tests as Postgres does, I
simply distributed the table beforehand. See below the relevant Postgres
commits from where you can see the tests added as well:
- [Add EXPLAIN
(MEMORY)](https://github.com/postgres/postgres/commit/5de890e36)
- [Invent SERIALIZE option for
EXPLAIN.](https://github.com/postgres/postgres/commit/06286709e)

This PR required a lot of copying of Postgres static functions regarding
how `EXPLAIN` works for `MEMORY` and `SERIALIZE` options. Specifically,
these copy-pastes were required for updating `ExplainWorkerPlan()`
function, which is in fact based on postgres' `ExplainOnePlan()`:
```C
/* copied from explain.c to update ExplainWorkerPlan() in citus according to ExplainOnePlan() in postgres */
#define BYTES_TO_KILOBYTES(b)
typedef struct SerializeMetrics
static bool peek_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_memory_counters(ExplainState *es, const MemoryContextCounters *mem_counters);
static void ExplainIndentText(ExplainState *es);
static void ExplainPrintSerialize(ExplainState *es, SerializeMetrics *metrics);
static SerializeMetrics GetSerializationMetrics(DestReceiver *dest);
```

_Note_: it looks like we were missing some `buffers` option details as
well. I put them together with the memory option, like the code in
Postgres explain.c, as I didn't want to change the copied code. However,
I tested locally and there is no big deal in previous Citus versions,
and you can also see that existing Citus tests with `buffers true`
didn't change. Therefore, I prefer not to backport "buffers" changes to
previous versions.
2025-01-02 12:32:36 +03:00
Mehmet YILMAZ 48849ff3c2
PG17 - Add Regression Test for REINDEX support in event triggers (#7819)
This PR adds regression tests to verify REINDEX support with event
triggers. Tests validates trigger execution, shard placement
consistency, and distributed index rebuilding without disruption.
2025-01-02 11:44:32 +03:00
Naisila Puka 6d2a329da6 EXPLAIN generic_plan NOT supported in Citus (#7825)
We thought we provided support for this in

b8c493f2c4

However the use of parameters in SQL is not supported in Citus. Since
generic plan queries use parameters, we can't support for now.

Relevant PG16 commit https://github.com/postgres/postgres/commit/3c05284

Fixes #7813 with proper error message

(cherry picked from commit 0a6adf4ccc)
2025-01-02 10:53:36 +03:00
Mehmet YILMAZ a5780c519f
PG17 - Add Regression Test for Access Method Behavior on Partitioned Tables (#7818)
This PR adds a regression test to verify the behavior of access methods
for partitioned and distributed tables, including:

- Creating partitioned tables with heap.
- Distributing tables using create_distributed_table.
- Switching access methods to columnar with ALTER TABLE.
- Validating access method inheritance for new partitions.

Relecant PG17 commit: https://github.com/postgres/postgres/commit/374c7a229
2024-12-31 22:51:43 +03:00
Naisila Puka 103125fa08
Add tests with xmltext() and random(min, max) (#7824)
xmltext() converts text into xml text nodes.
Test with columnar and citus tables.
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/526fe0d79

random(min, max) generates random numbers in a specified range Add tests
like the ones for random() in aggregate_support.sql References:

https://github.com/citusdata/citus/blob/main/src/test/regress/sql/aggregate_support.sql#L493-L532
https://github.com/citusdata/citus/pull/7183
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/e6341323a
2024-12-31 17:36:53 +03:00
Naisila Puka bee8e9cbb6
Fix foreign_key_to_reference_shard_rebalance test (#7826)
foreign_key_to_reference_shard_rebalance failed because partition of
2024 year does not exist, fixed by add default partition.

Replaces https://github.com/citusdata/citus/pull/7396 by adding a rule
that allows properly testing foreign_key_to_reference_shard_rebalance
via run_test.py.

Closes #7396

Co-authored-by: chuhx <148182736+cstarc1@users.noreply.github.com>
(cherry picked from commit 968ac74cde)

Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
2024-12-31 16:49:37 +03:00
Naisila Puka 80678bb07e
Allow configuring sslnegotiation using citus.node_conn_info (#7821)
Relevant PG commit:
https://github.com/postgres/postgres/commit/d39a49c1e

PR similar to https://github.com/citusdata/citus/pull/5203
2024-12-30 21:25:50 +03:00
Naisila Puka b3448a1661
Add pg17 jsonpath methods tests (#7820)
various jsonpath methods were added in PG17
Relevant PG commit:
https://github.com/postgres/postgres/commit/66ea94e8e
Here we add the same test as in pg15_jsonpath.sql
for the new additions
2024-12-30 21:06:30 +03:00
Naisila Puka 7dc97d3dc9
Disallow infinite values for partition interval in create_time_partitions udf (#7822)
PG17 added +/- infinity values for the interval data type
Relevant PG commit:
https://github.com/postgres/postgres/commit/519fc1bd9
2024-12-30 20:27:28 +03:00
Naisila Puka 9e3c3297fc
Adds JSON_TABLE() support, and SQL/JSON constructor/query functions tests (#7816)
DESCRIPTION: Adds JSON_TABLE() support

PG17 has added basic `JSON_TABLE()` functionality
`JSON_TABLE()` allows `JSON` data to be converted into a relational view
and thus used, for example, in a `FROM` clause, like other tabular data.

We treat `JSON_TABLE` the same as correlated functions (e.g., recurring
tuples). In the end, for multi-shard `JSON_TABLE` commands, we apply the
same restrictions as reference tables (e.g., cannot perform a lateral
outer join when a distributed subquery references a (reference
table)/(json table) etc.)

Relevant PG17 commits:
[basic JSON
table](https://github.com/postgres/postgres/commit/de3600452), [nested
paths in json
table](https://github.com/postgres/postgres/commit/bb766cde6)

Onder had previously added json table support for PG15BETA1, but we
reverted that commit because json table was reverted in PG15.
ce7f1a530f
Previous relevant PG15Beta1 commit:
https://github.com/postgres/postgres/commit/4e34747c8
Therefore, I referred to Onder's commit for this commit as well, with a
few changes due to some differences between PG15/PG17:

1) In PG15Beta1, we had also `PLAN` clauses for `JSON_TABLE`
https://github.com/postgres/postgres/commit/fadb48b00, and Onder's
commit includes tests for those as well. However, `PLAN` nodes are _not_
added in PG17. Therefore, I didn't include the `json_table_select_only`
test, which had mostly queries involving `PLAN`. I only included the
last query from that test.

2) In PG15 timeline (Citus 11.1), we didn't support outer joins where
the outer rel is a recurring one and the inner one is a non-recurring
one. However, [Onur added support for that one in Citus
11.2](https://github.com/citusdata/citus/pull/6512), therefore I updated
the tests from Onder's commit accordingly.

3) PG17 json table has nested paths and columns, therefore I added a
test
with a distributed table, which is exactly the same as the one in
sqljson_jsontable in PG17.
https://github.com/postgres/postgres/commit/bb766cde6

This pull request also adds some basic tests on validation of SQL/JSON
constructor functions JSON(), JSON_SCALAR(), and JSON_SERIALIZE(),
and also SQL/JSON query functions JSON_EXISTS(), JSON_QUERY(), and
JSON_VALUE(). The relevant PG commits are the following:
[JSON(), JSON_SCALAR(),
JSON_SERIALIZE()](https://github.com/postgres/postgres/commit/03734a7fe)
[JSON_EXISTS(), JSON_VALUE(),
JSON_QUERY()](https://github.com/postgres/postgres/commit/6185c9737)
2024-12-30 19:19:07 +03:00
Naisila Puka 8a8b2f9a6d
Add tests for inserting with AT LOCAL operator (#7815)
PG17 has added support for AT LOCAL operator
it converts the given time type to
time stamp with the session's TimeZone value as time zone. Here we add
tests that validate that we can use AT LOCAL at INSERT commands

Relevant PG commit:
https://github.com/postgres/postgres/commit/97957fdba

With the tests, we verify that we evaluate AT LOCAL at the coordinator
and then perform the insert remotely.
2024-12-30 12:54:21 +03:00
Mehmet YILMAZ 1a3316281c
Error out for ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION (#7814)
PG17 added support for
ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION.
Relevant PG commit: https://github.com/postgres/postgres/commit/5d06e99a3

We currently don't support propagating this command for Citus tables.
It is added to future work.

This PR disallows `ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION` on
all Citus table types (local, distributed, and partitioned distributed)
by adding an error check in `ErrorIfUnsupportedAlterTableStmt`. A new
regression test verifies that each table type fails with a consistent
error message when attempting to set an expression.
2024-12-27 16:02:12 +03:00
Mehmet YILMAZ 62a00b1b34
Error out for ALTER TABLE ... SET ACCESS METHOD DEFAULT (#7803)
PG17 introduced ALTER TABLE ... SET ACCESS METHOD DEFAULT

This PR introduces and enforces an error check preventing ALTER TABLE
... SET ACCESS METHOD DEFAULT on both Citus local tables (added via
citus_add_local_table_to_metadata) and distributed/partitioned
distributed tables. The regression tests now demonstrate that each table
type raises an error advising users to explicitly specify an access
method, rather than relying on DEFAULT. This ensures consistent behavior
across local and distributed environments in Citus.

The reason why we currently don't support this is that we can't simply
propagate the command as it is, because the default table access method
may be different across Citus cluster nodes.

Relevant PG commit:
https://github.com/postgres/postgres/commit/d61a6cad6
2024-12-27 15:07:38 +03:00
Naisila Puka b4cc7219ae
Add tests for FORCE_NULL * and FORCE_NOT_NULL * options for COPY FROM (#7812)
These options already existed in PG17, and we support them and have
tests for them in `multi_copy.sql`.

In PG17, their capability was extended to specify ALL columns at once
using *.
Citus performs the COPY correctly, as is validated by the added tests in
this PR.

Relevant PG commit:
https://github.com/postgres/postgres/commit/f6d4c9cf1

Copy-pasting from Postgres documentation what these options do, such
that the reviewer may better understand the tests added:

`FORCE_NOT_NULL`: Do not match the specified columns' values against the
null string. In the default case where the null string is empty, this
means that empty values will be read as zero-length strings rather than
nulls, even when they are not quoted. If * is specified, the option will
be applied to all columns. This option is allowed only in `COPY FROM`,
and only when using `CSV` format.

`FORCE_NULL`: Match the specified columns' values against the null
string, even if it has been quoted, and if a match is found set the
value to `NULL`. In the default case where the null string is empty,
this converts a quoted empty string into `NULL`. If * is specified, the
option will be applied to all columns. This option is allowed only in
`COPY FROM`, and only when using `CSV` format.

`FORCE_NULL` and `FORCE_NOT_NULL` can be used simultaneously on the same
column. This results in converting quoted null strings to null values
and unquoted null strings to empty strings.

Explain it to me like I'm a 5-year-old, for a text column:
`FORCE_NULL` looks for empty strings and registers them as `NULL`
`FORCE_NOT_NULL` looks for null values and registers them as empty
strings.
2024-12-26 16:52:42 +03:00
Naisila Puka d682bf06f7
Error for COPY FROM ... on_error, log_verbosity with Citus tables (#7811)
PG17 added the new ON_ERROR option for COPY FROM. When this option is
specified, COPY skips soft errors and
continues copying.
Relevant PG commits:
-- https://github.com/postgres/postgres/commit/9e2d87011
-- https://github.com/postgres/postgres/commit/b725b7eec

I tried it locally with Citus tables.
Without further implementation, it doesn't work correctly.
Therefore, we error out for now, and add it to future work.

PG17 also added log_verbosity option, which controls the
 amount of messages emitted during processing. This is
 currently used in COPY FROM when ON_ERROR option is set to
 ignore. Therefore, we error out for this option as well.
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/f5a227895
2024-12-26 15:29:44 +03:00
Naisila Puka af88c37b56
PG17: ALTER INDEX ALTER COLUMN SET STATISTICS DEFAULT (#7808)
DESCRIPTION: Propagates ALTER INDEX ALTER COLUMN SET STATISTICS DEFAULT

We automatically support this. Adding tests only.

We currently don't support ALTER TABLE ALTER COLUMN SET STATISTICS

Relevant PG commit:
https://github.com/postgres/postgres/commit/4f622503d
2024-12-25 16:58:51 +03:00
Naisila Puka 6fed917a53
Adds PG17.1 support - Regression tests sanity (#7661)
This is the final commit that adds
PG17 compatibility with Citus's current capabilities.

You can use Citus community, release-13.0 branch, with PG17.1.

---------

Specifically, this commit:

- Enables PG17 in the configure script.

- Adds PG17 tests to CI using test images that have 17.1

- Fixes an upgrade test: see below for details
In `citus_prepare_upgrade()`, don't drop any_value when upgrading from
PG16+, because PG16+ has its own any_value function. Attempting to do so
results in the error seen in [pg16-pg17
upgrade](https://github.com/citusdata/citus/actions/runs/11768444117/job/32778340003?pr=7661):
```
ERROR:  cannot drop function any_value(anyelement) because it is required by the database system
CONTEXT:  SQL statement "DROP AGGREGATE IF EXISTS pg_catalog.any_value(anyelement)"
```
When 16 becomes the minimum supported Postgres version, the drop
statements can be removed.

---------

Several PG17 Compatibility commits have been merged before this final one.
All these subtasks are done https://github.com/citusdata/citus/issues/7653

See the list below:

Compilation PR: https://github.com/citusdata/citus/pull/7699
Ruleutils PR: https://github.com/citusdata/citus/pull/7725
Sister PR for tests: https://github.com/citusdata/the-process/pull/159

Helpful smaller PRs:
- https://github.com/citusdata/citus/pull/7714
- https://github.com/citusdata/citus/pull/7726
- https://github.com/citusdata/citus/pull/7731
- https://github.com/citusdata/citus/pull/7732
- https://github.com/citusdata/citus/pull/7733
- https://github.com/citusdata/citus/pull/7738
- https://github.com/citusdata/citus/pull/7745
- https://github.com/citusdata/citus/pull/7747
- https://github.com/citusdata/citus/pull/7748
- https://github.com/citusdata/citus/pull/7749
- https://github.com/citusdata/citus/pull/7752
- https://github.com/citusdata/citus/pull/7755
- https://github.com/citusdata/citus/pull/7757
- https://github.com/citusdata/citus/pull/7759
- https://github.com/citusdata/citus/pull/7760
- https://github.com/citusdata/citus/pull/7761
- https://github.com/citusdata/citus/pull/7762
- https://github.com/citusdata/citus/pull/7765
- https://github.com/citusdata/citus/pull/7766
- https://github.com/citusdata/citus/pull/7768
- https://github.com/citusdata/citus/pull/7769
- https://github.com/citusdata/citus/pull/7771
- https://github.com/citusdata/citus/pull/7774
- https://github.com/citusdata/citus/pull/7776
- https://github.com/citusdata/citus/pull/7780
- https://github.com/citusdata/citus/pull/7781
- https://github.com/citusdata/citus/pull/7785
- https://github.com/citusdata/citus/pull/7788
- https://github.com/citusdata/citus/pull/7793
- https://github.com/citusdata/citus/pull/7796

---------

Co-authored-by: Colm <colmmchugh@microsoft.com>
2024-12-24 17:56:51 +03:00
Naisila Puka 743f0ebb75
Bump Citus version into 13.0.0 (#7792)
We are using `release-13.0` branch for both development and release, to
deliver PG17 support in Citus.

Afterwards, we will (probably) merge this branch into main.

Some potential changes for main branch, after we are done working on
release-13.0:
- Merge changes from `release-13.0` to `main`
- Figure out what changes were there on 12.2, move them to 13.1 version.
In a nutshell: rename `12.1--12.2` to `13.0--13.1` and fix issues.
- Set version to 13.1devel
2024-12-24 11:40:59 +03:00
Mehmet YILMAZ 351b1ca63c
PG17 compatibility: Fix Test Failure in multi_alter_table_add_const (#7733)
In earlier versions of PostgreSQL, exclusion constraints were not
allowed on partitioned tables. This is why the error in your regression
test (ERROR: exclusion constraints are not supported on partitioned
tables) was raised in PostgreSQL 16. In PostgreSQL 17, exclusion
constraints are now allowed on partitioned tables, which is why the
error no longer appears when you attempt to add an exclusion constraint.

The constraint exclusion mechanism, described in the documentation,
relies on CHECK constraints to decide which partitions or child tables
need to be queried.

[CHECK
constraints](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-CONSTRAINT-EXCLUSION)

```diff
 -- Check "ADD EXCLUDE" errors out for partitioned table since the postgres does not allow it
 ALTER TABLE AT_AddConstNoName.citus_local_partitioned_table ADD EXCLUDE(partition_col WITH =);
-ERROR:  exclusion constraints are not supported on partitioned tables
 -- Check "ADD CHECK"
 SET client_min_messages TO DEBUG1;
 ALTER TABLE AT_AddConstNoName.citus_local_partitioned_table ADD CHECK (dist_col > 0);
 DEBUG:  the constraint name on the shards of the partition is too long, switching to sequential and local execution mode to prevent self deadlocks: longlonglonglonglonglonglonglonglonglonglonglo_537570f5_5_check
 DEBUG:  verifying table "longlonglonglonglonglonglonglonglonglonglonglonglonglonglongabc"
 DEBUG:  verifying table "p1"
 RESET client_min_messages;
 SELECT con.conname
     FROM pg_catalog.pg_constraint con
       INNER JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid
       INNER JOIN pg_catalog.pg_namespace nsp ON nsp.oid = connamespace
           WHERE rel.relname = 'citus_local_partitioned_table';
                      conname                      
 --------------------------------------------------
+ citus_local_partitioned_table_partition_col_excl
  citus_local_partitioned_table_check
-(1 row)
+(2 rows)
```
2024-12-23 15:24:46 +03:00
Naisila Puka aa8ed4c60b Bump postgres versions in CI and dev: 14.14, 15.9, 16.5 (#7779)
Upgrade postgres versions to:
 - 14.14
 - 15.9
 - 16.5

Depends on https://github.com/citusdata/the-process/pull/163

We had some errors with the latest minors, so this is a 2-level bump for now.
2024-12-23 15:18:53 +03:00
Mehmet YILMAZ 2d843cb382
PG17 compatibility: Fix Isolation Test Failure in isolation_multiuser_locking (#7714)
This PR enhances `isolation_multiuser_locking.spec` test compatibility
across multiple PostgreSQL versions by handling differences in error
messages and behavior. Key updates include:

- **Error Message Handling:** Adjustments to manage version-specific
error messages, ensuring consistent test results.
  
- Modified to address variations in locking behavior across PostgreSQL
versions, ensuring test stability in multiuser scenarios.

- **REINDEX Behavior Adjustment**: This PR accounts for a behavioral
change introduced in PostgreSQL by commit ecb0fd337, which alters how
REINDEX interacts with system catalogs.


https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ecb0fd337

---------

Co-authored-by: Mehmet YILMAZ <mehmet.yilmaz@microsoft.com>
2024-12-23 13:30:54 +03:00
Naisila Puka f123632f85
Fix pg17 test (#7797)
Broken from this commit
e3db375149

https://github.com/citusdata/citus/actions/runs/12429202397/attempts/1#summary-34702334056
2024-12-20 20:13:48 +03:00