Compare commits

..

137 Commits

Author SHA1 Message Date
Naisila Puka 548395fd77
fix changelog date (#7859) 2025-01-22 14:28:46 +03:00
Naisila Puka cba8e57737
Changelog entries for 13.0.0 (#7858) 2025-01-22 13:17:35 +03:00
Naisila Puka 23d5207701
Fix pg17 test (#7857)
error merged in
ab7c3b7804
2025-01-22 12:54:52 +03:00
Mehmet YILMAZ ab7c3b7804
PG17 Compatibility - Fix crash when pg_class is used in MERGE (#7853)
This pull request addresses Issue #7846, where specific MERGE queries on
non-distributed and distributed tables can result in crashes in certain
scenarios. The issue stems from the usage of `pg_class` catalog table,
and the `FilterShardsFromPgclass` function in Citus. This function goes
through the query's jointree to hide the shards. However, in PG17,
MERGE's join quals are in a separate structure called
`mergeJoinCondition`. Therefore FilterShardsFromPgclass was not
filtering correctly in a `MERGE` command that involves `pg_class`. To
fix the issue, we handle `mergeJoinCondition` separately in PG17.

Relevant PG commit:

0294df2f1f

**Non-Distributed Tables:**
A MERGE query involving a non-distributed table using
`pg_catalog.pg_class` as the source may execute successfully but needs
testing to ensure stability.

**Distributed Tables:**
Performing a MERGE on a distributed table using `pg_catalog.pg_class` as
the source raises an error:
`ERROR: MERGE INTO a distributed table from Postgres table is not yet
supported`
However, in some cases, this can lead to a server crash if the
unsupported operation is not properly handled.

This is the test output from the same test conducted prior to the code
changes being implemented.

```
-- Issue #7846: Test crash scenarios with MERGE on non-distributed and distributed tables
-- Step 1: Connect to a worker node to verify shard visibility
\c postgresql://postgres@localhost::worker_1_port/regression?application_name=psql
SET search_path TO pg17;
-- Step 2: Create and test a non-distributed table
CREATE TABLE non_dist_table_12345 (id INTEGER);
-- Test MERGE on the non-distributed table
MERGE INTO non_dist_table_12345 AS target_0
USING pg_catalog.pg_class AS ref_0
ON target_0.id = ref_0.relpages
WHEN NOT MATCHED THEN DO NOTHING;
SSL SYSCALL error: EOF detected
connection to server was lost
```
2025-01-21 17:48:06 +03:00
Colm c2bc7aca4a
Update tdigest_aggregate_support output for PG15+ (#7849)
Regress test tdigest_aggregate_support has been failing since at least
Citus 12.0, when tdigest extension is installed in Postgres. This
appears to be because of an omission by commit 03832f3 and a change in
the implementation of Postgres random() function (pg commit
[d4f109e4a](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d4f109e4a)).
To reproduce the test diff:
- Checkout [tdigest ](https://github.com/tvondra/tdigest)and run `make;
make install`
- In citus regress directory run `make check-multi` or
`./citus_tests/run_test.py tdigest_aggregate_support`

There are two parts to this commit:

1. Revert `Output: xxxxx` in EXPLAIN VERBOSE. Citus commit fe4ac51
normalized EXPLAIN VERBOSE output because of a change between pg12 and
pg13. When pg12 support was no longer required, the rule was removed
from normalize.sed and `Output: xxxx` was reverted in the impacted
regress output files (03832f3), but `tdigest_aggregate_support` was
omitted.

2. Adjust the query results; the tdigest_aggregate_support test file has
a comment _verifying results - should be stable due to seed while
inserting the data, if failure due to data these queries could be
removed or check for certain ranges_ but the result values in this
commit are consistent across citus 12.0 (pg 15), citus 12.1 (pg 16) and
citus 13.0 (pg 17), or since the Postgres changed their [implementation
of
random](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d4f109e4a),
so proposing to go with these results.
2025-01-20 22:00:33 +00:00
Naisila Puka fa8e867662
Bump to latest PG minors 17.2, 16.6, 15.10, 14.15 (#7843)
Similar to
5ef2cd67ed,
we use the commit sha of a local build of the images, pushed.
2025-01-13 22:35:11 +03:00
Emel Şimşek c55bc8c669 Propagates SECURITY LABEL ON ROLE stmt (#7304) (#7735)
Propagates SECURITY LABEL ON ROLE stmt (https://github.com/citusdata/citus/pull/7304)
We propagate `SECURITY LABEL [for provider] ON ROLE rolename IS
labelname` to the worker nodes.
We also make sure to run the relevant `SecLabelStmt` commands on a
newly added node by looking at roles found in `pg_shseclabel`.

See official docs for explanation on how this command works:
https://www.postgresql.org/docs/current/sql-security-label.html
This command stores the role label in the `pg_shseclabel` catalog table.

This commit also fixes the regex string in
`check_gucs_are_alphabetically_sorted.sh` script such that it escapes
the dot. Previously it was looking for all strings starting with "citus"
instead of "citus." as it should.

To test this feature, I currently make use of a special GUC to control
label provider registration in PG_init when creating the Citus extension.

(cherry picked from commit 0d1f18862b)

Co-authored-by: Naisila Puka <37271756+naisila@users.noreply.github.com>
(cherry picked from commit 686d2b46ca)
2025-01-13 19:56:01 +03:00
Nils Dijk 7e316c90c4 Shard moves/isolate report LSN's in lsn format (#7227)
DESCRIPTION: Shard moves/isolate report LSN's in lsn format

While investigating an issue with our catchup mechanism on certain
postgres versions we noticed we print LSN's in the format of the native
long type. This is an uncommon representation for LSN's in postgres
logs.

This patch changes the output of our log message to go from the long
type representation to the native LSN type representation. Making it
easier for postgres users to recognize and compare LSN's with other
related reports.

example of new output:
```
2023-09-25 17:28:47.544 CEST [11345] LOG:  The LSN of the target subscriptions on node localhost:9701 have increased from 0/0 to 0/E1ED20F8 at 2023-09-25 17:28:47.544165+02 where the source LSN is 1/415DCAD0
```

(cherry picked from commit b87fbcbf79)
2025-01-13 17:47:47 +03:00
Teja Mupparti d2ca63fb8c For scenarios, such as, Bug 3697586: Server crashes when assigning distributed transaction: Raise an ERROR instead of a crash
(cherry picked from commit ab7c13beb5)
2025-01-13 17:47:18 +03:00
Onur Tirtir a19e180212 Avoid re-assigning the global pid for client backends and bg workers when the application_name changes (#7791)
DESCRIPTION: Fixes a crash that happens because of unsafe catalog access
when re-assigning the global pid after application_name changes.

When application_name changes, we don't actually need to
try re-assigning the global pid for external client backends because
application_name doesn't affect the global pid for such backends. Plus,
trying to re-assign the global pid for external client backends would
unnecessarily cause performing a catalog access when the cached local
node id is invalidated. However, accessing to the catalog tables is
dangerous in certain situations like when we're not in a transaction
block. And for the other types of backends, i.e., the Citus internal
backends, we need to re-assign the global pid when the application_name
changes because for such backends we simply extract the global pid
inherited from the originating backend from the application_name -that's
specified by originating backend when openning that connection- and this
doesn't require catalog access.

(cherry picked from commit 73411915a4)
2025-01-13 17:47:11 +03:00
Pavel Seleznev cdded256ef Remove warnings on some builds (#7680)
Co-authored-by: Pavel Seleznev <PNSeleznev@sberbank.ru>
(cherry picked from commit fe6d198ab2)
2025-01-13 15:22:13 +03:00
Colm McHugh 353033d3f0 [Bug Fix] [SEGFAULT] Querying distributed tables with window partition may cause segfault #7705
In function MasterAggregateMutator(), when the original Node is a Var node use makeVar() instead
of copyObject() when constructing the Var node for the target list of the combine query.
The varnullingrels field of the original Var node is ignored because it is not relevant for the
combine query; copying this cause the problem in issue 7705, where a coordinator query had
a Var with a reference to a non-existent join relation.

(cherry picked from commit c52f36019f)
2025-01-13 15:21:50 +03:00
Parag Jain 1893d9a900 [Bug Fix] : writing incorrect data to target Merge repartition Command (#7659)
We were writing incorrect data to target collection in some cases of merge command. In case of repartition when source query is RELATION. We were referring to incorrect attribute number that was resulting into
this incorrect behavior.

Example :

![image](https://github.com/user-attachments/assets/a101cb36-7976-459c-befb-96a55a5b3dc1)

![image](https://github.com/user-attachments/assets/e5c83b7b-5b8e-4d79-a927-95684dc9ba49)

I have added fixed tests as part of this PR , Thanks.

(cherry picked from commit 5bad6c6a1d)
2025-01-13 15:21:37 +03:00
Mehmet YILMAZ 063cff908e Fix race condition in citus_set_coordinator_host when adding multiple coordinator nodes concurrently (#7682)
When multiple sessions concurrently attempt to add the same coordinator
node using `citus_set_coordinator_host`, there is a potential race
condition. Both sessions may pass the initial metadata check
(`isCoordinatorInMetadata`), but only one will succeed in adding the
node. The other session will fail with an assertion error
(`Assert(!nodeAlreadyExists)`), causing the server to crash. Even though
the `AddNodeMetadata` function takes an exclusive lock, it appears that
the lock is not preventing the race condition before the initial
metadata check.

- **Issue**: The current logic allows concurrent sessions to pass the
check for existing coordinators, leading to an attempt to insert
duplicate nodes, which triggers the assertion failure.

- **Impact**: This race condition leads to crashes during operations
that involve concurrent coordinator additions, as seen in
https://github.com/citusdata/citus/issues/7646.

**Test Plan:**

- Isolation Test Limitation: An isolation test was added to simulate
concurrent additions of the same coordinator node, but due to the
behavior of PostgreSQL locking mechanisms, the test does not trigger the
edge case. The lock applied within the function serializes the
operations, preventing the race condition from occurring in the
isolation test environment.
While the edge case is difficult to reproduce in an isolation test, the
fix addresses the core issue by ensuring concurrency control through
proper locking.

- Existing Tests: All existing tests related to node metadata and
coordinator management have been run to ensure that no regressions were
introduced.

**After the Fix:**

- Concurrent attempts to add the same coordinator node will be
serialized. One session will succeed in adding the node, while the
others will skip the operation without crashing the server.

Co-authored-by: Mehmet YILMAZ <mehmet.yilmaz@microsoft.com>
(cherry picked from commit 4775715691)
2025-01-13 15:20:03 +03:00
eaydingol a115b3043a Check if the limit is null (#7665)
DESCRIPTION: Add a check to see if the given limit is null. 

Fixes a bug by checking if the limit given in the query is null when the
actual limit is computed with respect to the given offset.
Prior to this change, null is interpreted as 0 during the limit
calculation when both limit and offset are given.

Fixes #7663
2025-01-12 23:59:17 +03:00
Evgeny Nechayev 5eb037ac50 Use macro wrapper to access PGPROC data, which allow to improve compa… (#7607)
DESCRIPTION: Use macro wrapper to access PGPROC data, to improve compatibility with PostgreSQL forks.
2025-01-12 23:49:21 +03:00
Onur Tirtir ef946e44af Fix incorrect "VALID UNTIL" assumption made for roles in node activation (#7534)
Fixes https://github.com/citusdata/citus/issues/7533.

DESCRIPTION: Fixes incorrect `VALID UNTIL` setting assumption made for
roles when syncing them to new nodes
2025-01-12 23:46:20 +03:00
Karina 89ccafff50 Fix server crash when trying to execute activate_node_snapshot() on a single-node cluster (#7552)
This fixes #7551 reported by Egor Chindyaskin

Function activate_node_snapshot() is not meant to be called on a cluster
without worker nodes. This commit adds ERROR report for such case to
prevent server crash.
2025-01-12 23:44:13 +03:00
eaydingol e2a03ce6a0 Fix: store the previous shard cost for order verification (#7550)
Store the previous shard cost so that the invariant checking performs as
expected.
2025-01-12 23:43:53 +03:00
Filip Sedlák 657575114c Fail early when shard can't be safely moved to a new node (#7467)
DESCRIPTION: citus_move_shard_placement now fails early when shard
cannot be safely moved

The implementation is quite simplistic -
`citus_move_shard_placement(...)` will fail with an error if there's any
new node in the cluster that doesn't have reference tables yet.

It could have been finer-grained, i.e. erroring only when trying to move
a shard to an unitialized node. Looking at the related functions -
`replicate_reference_tables()` or `citus_rebalance_start()`, I think
it's acceptable behaviour. These other functions also treat "any"
unitialized node as a temporary anomaly.

Fixes #7426

---------

Co-authored-by: Jelte Fennema-Nio <jelte.fennema@microsoft.com>
2025-01-12 22:54:09 +03:00
eaydingol ddef97212a Generate qualified relation name (#7427)
This change refactors the code by using generate_qualified_relation_name
from id instead of using a sequence of functions to generate the
relation name.

Fixes #6602

(cherry picked from commit ee11492a0e)
2025-01-12 22:08:41 +03:00
zhjwpku f42e8556ec [performance improvement] remove duplicate LoadShardList call (#7380)
LoadShardList is called twice, which is not neccessary, and there is no
need to sort the shard placement list since we only want to know the list
length.

(cherry picked from commit 8e979f7ac6)
2025-01-12 22:08:03 +03:00
Karina 01945bd3a3 Fix getting heap tuple size (#7387)
This fixes #7230.

First of all, using HeapTupleHeaderGetDatumLength(heapTuple) is
definetly wrong, it gives a number that's 4 times less than the correct
tuple size (heapTuple.t_len). See

https://github.com/postgres/postgres/blob/REL_16_0/src/include/access/htup_details.h#L455-L456

https://github.com/postgres/postgres/blob/REL_16_0/src/include/varatt.h#L279

https://github.com/postgres/postgres/blob/REL_16_0/src/include/varatt.h#L225-L226

When I fixed it, the limit_intermediate_size test failed, so I tried to
understand what's going on there. In original commit fd546cf these
queries were supposed to fail. Then in b3af63c three of the queries that
were supposed to fail suddenly worked and tests were changed to pass
without understanding why the output had changed or how to keep test
testing what it had to test. Even comments saying that these queries
should fail were left untouched. Commit message gives no clue about why
exactly test has changed:

> It seems that when we use adaptive executor instead of task tracker,
we
> exceed the intermediate result size less in the test. Therefore
updated
> the tests accordingly.

Then 3fda2c3 also blindly raised the limit for one of the queries to
keep it working:

3fda2c3254 (diff-a9b7b617f9dfd345318cb8987d5897143ca1b723c87b81049bbadd94dcc86570R19)

When in fe3caf3 that HeapTupleHeaderGetDatumLength(heapTuple) call was
finally added, one of those test queries became failing again.

The other two of them now also failing after the fix. I don't understand
how exactly the calculation of "intermediate result size" that is
limited by citus.max_intermediate_result_size had changed through
b3af63c and fe3caf3, but these numbers are now closer to what
they originally were when this limitation was added in
fd546cf. So these queries should fail, like in the original
version of the limit_intermediate_size test.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 20dc58cf5d)
2025-01-12 22:06:46 +03:00
Cédric Villemain 1a8349aeaf Fix #7242, CALL(@0) crash backend (#7288)
When executing a prepared CALL, which is not pure SQL but available with
some drivers like npgsql and jpgdbc, Citus entered a code path where a
plan is not defined, while trying to increase its cost. Thus SIG11 when
plan is a NULL pointer.

Fix by only increasing plan cost when plan is not null.

However, it is a bit suspicious to get here with a NULL plan and maybe a
better change will be to not call
ShardPlacementForFunctionColocatedWithDistTable() with a NULL plan at
all (in call.c:134)

bug hit with for example:
```
CallableStatement proc = con.prepareCall("{CALL p(?)}");
proc.registerOutParameter(1, java.sql.Types.BIGINT);
proc.setInt(1, -100);
proc.execute();
```

where `p(bigint)` is a distributed "function" and the param the
distribution key (also in a distributed table), see #7242 for details

Fixes #7242

(cherry picked from commit 0678a2fd89)
2025-01-12 22:06:08 +03:00
Emel Şimşek f18c21c2b4 Send keepalive messages in split decoder periodically to avoid wal receiver timeouts during large shard splits. (#7229)
DESCRIPTION: Send keepalive messages during the logical replication
phase of large shard splits to avoid timeouts.

During the logical replication part of the shard split process, split
decoder filters out the wal records produced by the initial copy. If the
number of wal records is big, then split decoder ends up processing for
a long time before sending out any wal records through pgoutput. Hence
the wal receiver may time out and restarts repeatedly causing our split
driver code catch up logic to fail.

Notes:

1. If the wal_receiver_timeout is set to a very small number e.g. 600ms,
it may time out before receiving the keepalives. My tests show that this
code works best when the` wal_receiver_timeout `is set to 1minute, which
is the default value.

2. Once a logical replication worker time outs, a new one gets launched.
The new logical replication worker sets the pg_stat_subscription columns
to initial values. E.g. the latest_end_lsn is set to 0. Our driver logic
in `WaitForGroupedLogicalRepTargetsToCatchUp` can not handle LSN value
to go back. This is the main reason for it to get stuck in the infinite
loop.

(cherry picked from commit e9035f6d32)
2025-01-12 22:05:33 +03:00
Naisila Puka 3e924db90a
Propagate MERGE ... WHEN NOT MATCHED BY SOURCE (#7807)
DESCRIPTION: Propagates MERGE ... WHEN NOT MATCHED BY SOURCE

It seems like there is not much needed to be done here.
`get_merge_query_def` from `ruleutils_17` is updated with "WHEN NOT
MATCHED BY SOURCE" therefore `deparse_shard_query` parses the merge
query for execution on the shard correctly.

Relevant PG commit:
https://github.com/postgres/postgres/commit/0294df2f1
2025-01-09 00:03:06 +03:00
Naisila Puka 32c058e448
Remove Debian Buster support from packaging pipelines (#7828) (#7837)
Remove Debian Buster support from packaging-test-pipelines

Co-authored-by: Gürkan İndibay <gindibay@microsoft.com>
(cherry picked from commit 70f84e4aee)

Co-authored-by: Seda Gündoğdu <69769369+sedagundogdu@users.noreply.github.com>
2025-01-08 23:37:47 +03:00
Naisila Puka 9bad33f89d
PG17 - Propagate EXPLAIN options: MEMORY and SERIALIZE (#7802)
DESCRIPTION: Propagates MEMORY and SERIALIZE options of EXPLAIN

The options for `MEMORY` can be true or false. Default is false.
The options for `SERIALIZE` can be none, text or binary. Default is
none.

I referred to how we added support for WAL option in this PR [Support
EXPLAIN(ANALYZE, WAL)](https://github.com/citusdata/citus/pull/4196).
For the tests however, I used the same tests as Postgres, not like the
tests in the WAL PR. I used exactly the same tests as Postgres does, I
simply distributed the table beforehand. See below the relevant Postgres
commits from where you can see the tests added as well:
- [Add EXPLAIN
(MEMORY)](https://github.com/postgres/postgres/commit/5de890e36)
- [Invent SERIALIZE option for
EXPLAIN.](https://github.com/postgres/postgres/commit/06286709e)

This PR required a lot of copying of Postgres static functions regarding
how `EXPLAIN` works for `MEMORY` and `SERIALIZE` options. Specifically,
these copy-pastes were required for updating `ExplainWorkerPlan()`
function, which is in fact based on postgres' `ExplainOnePlan()`:
```C
/* copied from explain.c to update ExplainWorkerPlan() in citus according to ExplainOnePlan() in postgres */
#define BYTES_TO_KILOBYTES(b)
typedef struct SerializeMetrics
static bool peek_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_memory_counters(ExplainState *es, const MemoryContextCounters *mem_counters);
static void ExplainIndentText(ExplainState *es);
static void ExplainPrintSerialize(ExplainState *es, SerializeMetrics *metrics);
static SerializeMetrics GetSerializationMetrics(DestReceiver *dest);
```

_Note_: it looks like we were missing some `buffers` option details as
well. I put them together with the memory option, like the code in
Postgres explain.c, as I didn't want to change the copied code. However,
I tested locally and there is no big deal in previous Citus versions,
and you can also see that existing Citus tests with `buffers true`
didn't change. Therefore, I prefer not to backport "buffers" changes to
previous versions.
2025-01-02 12:32:36 +03:00
Mehmet YILMAZ 48849ff3c2
PG17 - Add Regression Test for REINDEX support in event triggers (#7819)
This PR adds regression tests to verify REINDEX support with event
triggers. Tests validates trigger execution, shard placement
consistency, and distributed index rebuilding without disruption.
2025-01-02 11:44:32 +03:00
Naisila Puka 6d2a329da6 EXPLAIN generic_plan NOT supported in Citus (#7825)
We thought we provided support for this in

b8c493f2c4

However the use of parameters in SQL is not supported in Citus. Since
generic plan queries use parameters, we can't support for now.

Relevant PG16 commit https://github.com/postgres/postgres/commit/3c05284

Fixes #7813 with proper error message

(cherry picked from commit 0a6adf4ccc)
2025-01-02 10:53:36 +03:00
Mehmet YILMAZ a5780c519f
PG17 - Add Regression Test for Access Method Behavior on Partitioned Tables (#7818)
This PR adds a regression test to verify the behavior of access methods
for partitioned and distributed tables, including:

- Creating partitioned tables with heap.
- Distributing tables using create_distributed_table.
- Switching access methods to columnar with ALTER TABLE.
- Validating access method inheritance for new partitions.

Relecant PG17 commit: https://github.com/postgres/postgres/commit/374c7a229
2024-12-31 22:51:43 +03:00
Naisila Puka 103125fa08
Add tests with xmltext() and random(min, max) (#7824)
xmltext() converts text into xml text nodes.
Test with columnar and citus tables.
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/526fe0d79

random(min, max) generates random numbers in a specified range Add tests
like the ones for random() in aggregate_support.sql References:

https://github.com/citusdata/citus/blob/main/src/test/regress/sql/aggregate_support.sql#L493-L532
https://github.com/citusdata/citus/pull/7183
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/e6341323a
2024-12-31 17:36:53 +03:00
Naisila Puka bee8e9cbb6
Fix foreign_key_to_reference_shard_rebalance test (#7826)
foreign_key_to_reference_shard_rebalance failed because partition of
2024 year does not exist, fixed by add default partition.

Replaces https://github.com/citusdata/citus/pull/7396 by adding a rule
that allows properly testing foreign_key_to_reference_shard_rebalance
via run_test.py.

Closes #7396

Co-authored-by: chuhx <148182736+cstarc1@users.noreply.github.com>
(cherry picked from commit 968ac74cde)

Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
2024-12-31 16:49:37 +03:00
Naisila Puka 80678bb07e
Allow configuring sslnegotiation using citus.node_conn_info (#7821)
Relevant PG commit:
https://github.com/postgres/postgres/commit/d39a49c1e

PR similar to https://github.com/citusdata/citus/pull/5203
2024-12-30 21:25:50 +03:00
Naisila Puka b3448a1661
Add pg17 jsonpath methods tests (#7820)
various jsonpath methods were added in PG17
Relevant PG commit:
https://github.com/postgres/postgres/commit/66ea94e8e
Here we add the same test as in pg15_jsonpath.sql
for the new additions
2024-12-30 21:06:30 +03:00
Naisila Puka 7dc97d3dc9
Disallow infinite values for partition interval in create_time_partitions udf (#7822)
PG17 added +/- infinity values for the interval data type
Relevant PG commit:
https://github.com/postgres/postgres/commit/519fc1bd9
2024-12-30 20:27:28 +03:00
Naisila Puka 9e3c3297fc
Adds JSON_TABLE() support, and SQL/JSON constructor/query functions tests (#7816)
DESCRIPTION: Adds JSON_TABLE() support

PG17 has added basic `JSON_TABLE()` functionality
`JSON_TABLE()` allows `JSON` data to be converted into a relational view
and thus used, for example, in a `FROM` clause, like other tabular data.

We treat `JSON_TABLE` the same as correlated functions (e.g., recurring
tuples). In the end, for multi-shard `JSON_TABLE` commands, we apply the
same restrictions as reference tables (e.g., cannot perform a lateral
outer join when a distributed subquery references a (reference
table)/(json table) etc.)

Relevant PG17 commits:
[basic JSON
table](https://github.com/postgres/postgres/commit/de3600452), [nested
paths in json
table](https://github.com/postgres/postgres/commit/bb766cde6)

Onder had previously added json table support for PG15BETA1, but we
reverted that commit because json table was reverted in PG15.
ce7f1a530f
Previous relevant PG15Beta1 commit:
https://github.com/postgres/postgres/commit/4e34747c8
Therefore, I referred to Onder's commit for this commit as well, with a
few changes due to some differences between PG15/PG17:

1) In PG15Beta1, we had also `PLAN` clauses for `JSON_TABLE`
https://github.com/postgres/postgres/commit/fadb48b00, and Onder's
commit includes tests for those as well. However, `PLAN` nodes are _not_
added in PG17. Therefore, I didn't include the `json_table_select_only`
test, which had mostly queries involving `PLAN`. I only included the
last query from that test.

2) In PG15 timeline (Citus 11.1), we didn't support outer joins where
the outer rel is a recurring one and the inner one is a non-recurring
one. However, [Onur added support for that one in Citus
11.2](https://github.com/citusdata/citus/pull/6512), therefore I updated
the tests from Onder's commit accordingly.

3) PG17 json table has nested paths and columns, therefore I added a
test
with a distributed table, which is exactly the same as the one in
sqljson_jsontable in PG17.
https://github.com/postgres/postgres/commit/bb766cde6

This pull request also adds some basic tests on validation of SQL/JSON
constructor functions JSON(), JSON_SCALAR(), and JSON_SERIALIZE(),
and also SQL/JSON query functions JSON_EXISTS(), JSON_QUERY(), and
JSON_VALUE(). The relevant PG commits are the following:
[JSON(), JSON_SCALAR(),
JSON_SERIALIZE()](https://github.com/postgres/postgres/commit/03734a7fe)
[JSON_EXISTS(), JSON_VALUE(),
JSON_QUERY()](https://github.com/postgres/postgres/commit/6185c9737)
2024-12-30 19:19:07 +03:00
Naisila Puka 8a8b2f9a6d
Add tests for inserting with AT LOCAL operator (#7815)
PG17 has added support for AT LOCAL operator
it converts the given time type to
time stamp with the session's TimeZone value as time zone. Here we add
tests that validate that we can use AT LOCAL at INSERT commands

Relevant PG commit:
https://github.com/postgres/postgres/commit/97957fdba

With the tests, we verify that we evaluate AT LOCAL at the coordinator
and then perform the insert remotely.
2024-12-30 12:54:21 +03:00
Mehmet YILMAZ 1a3316281c
Error out for ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION (#7814)
PG17 added support for
ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION.
Relevant PG commit: https://github.com/postgres/postgres/commit/5d06e99a3

We currently don't support propagating this command for Citus tables.
It is added to future work.

This PR disallows `ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION` on
all Citus table types (local, distributed, and partitioned distributed)
by adding an error check in `ErrorIfUnsupportedAlterTableStmt`. A new
regression test verifies that each table type fails with a consistent
error message when attempting to set an expression.
2024-12-27 16:02:12 +03:00
Mehmet YILMAZ 62a00b1b34
Error out for ALTER TABLE ... SET ACCESS METHOD DEFAULT (#7803)
PG17 introduced ALTER TABLE ... SET ACCESS METHOD DEFAULT

This PR introduces and enforces an error check preventing ALTER TABLE
... SET ACCESS METHOD DEFAULT on both Citus local tables (added via
citus_add_local_table_to_metadata) and distributed/partitioned
distributed tables. The regression tests now demonstrate that each table
type raises an error advising users to explicitly specify an access
method, rather than relying on DEFAULT. This ensures consistent behavior
across local and distributed environments in Citus.

The reason why we currently don't support this is that we can't simply
propagate the command as it is, because the default table access method
may be different across Citus cluster nodes.

Relevant PG commit:
https://github.com/postgres/postgres/commit/d61a6cad6
2024-12-27 15:07:38 +03:00
Naisila Puka b4cc7219ae
Add tests for FORCE_NULL * and FORCE_NOT_NULL * options for COPY FROM (#7812)
These options already existed in PG17, and we support them and have
tests for them in `multi_copy.sql`.

In PG17, their capability was extended to specify ALL columns at once
using *.
Citus performs the COPY correctly, as is validated by the added tests in
this PR.

Relevant PG commit:
https://github.com/postgres/postgres/commit/f6d4c9cf1

Copy-pasting from Postgres documentation what these options do, such
that the reviewer may better understand the tests added:

`FORCE_NOT_NULL`: Do not match the specified columns' values against the
null string. In the default case where the null string is empty, this
means that empty values will be read as zero-length strings rather than
nulls, even when they are not quoted. If * is specified, the option will
be applied to all columns. This option is allowed only in `COPY FROM`,
and only when using `CSV` format.

`FORCE_NULL`: Match the specified columns' values against the null
string, even if it has been quoted, and if a match is found set the
value to `NULL`. In the default case where the null string is empty,
this converts a quoted empty string into `NULL`. If * is specified, the
option will be applied to all columns. This option is allowed only in
`COPY FROM`, and only when using `CSV` format.

`FORCE_NULL` and `FORCE_NOT_NULL` can be used simultaneously on the same
column. This results in converting quoted null strings to null values
and unquoted null strings to empty strings.

Explain it to me like I'm a 5-year-old, for a text column:
`FORCE_NULL` looks for empty strings and registers them as `NULL`
`FORCE_NOT_NULL` looks for null values and registers them as empty
strings.
2024-12-26 16:52:42 +03:00
Naisila Puka d682bf06f7
Error for COPY FROM ... on_error, log_verbosity with Citus tables (#7811)
PG17 added the new ON_ERROR option for COPY FROM. When this option is
specified, COPY skips soft errors and
continues copying.
Relevant PG commits:
-- https://github.com/postgres/postgres/commit/9e2d87011
-- https://github.com/postgres/postgres/commit/b725b7eec

I tried it locally with Citus tables.
Without further implementation, it doesn't work correctly.
Therefore, we error out for now, and add it to future work.

PG17 also added log_verbosity option, which controls the
 amount of messages emitted during processing. This is
 currently used in COPY FROM when ON_ERROR option is set to
 ignore. Therefore, we error out for this option as well.
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/f5a227895
2024-12-26 15:29:44 +03:00
Naisila Puka af88c37b56
PG17: ALTER INDEX ALTER COLUMN SET STATISTICS DEFAULT (#7808)
DESCRIPTION: Propagates ALTER INDEX ALTER COLUMN SET STATISTICS DEFAULT

We automatically support this. Adding tests only.

We currently don't support ALTER TABLE ALTER COLUMN SET STATISTICS

Relevant PG commit:
https://github.com/postgres/postgres/commit/4f622503d
2024-12-25 16:58:51 +03:00
Naisila Puka 6fed917a53
Adds PG17.1 support - Regression tests sanity (#7661)
This is the final commit that adds
PG17 compatibility with Citus's current capabilities.

You can use Citus community, release-13.0 branch, with PG17.1.

---------

Specifically, this commit:

- Enables PG17 in the configure script.

- Adds PG17 tests to CI using test images that have 17.1

- Fixes an upgrade test: see below for details
In `citus_prepare_upgrade()`, don't drop any_value when upgrading from
PG16+, because PG16+ has its own any_value function. Attempting to do so
results in the error seen in [pg16-pg17
upgrade](https://github.com/citusdata/citus/actions/runs/11768444117/job/32778340003?pr=7661):
```
ERROR:  cannot drop function any_value(anyelement) because it is required by the database system
CONTEXT:  SQL statement "DROP AGGREGATE IF EXISTS pg_catalog.any_value(anyelement)"
```
When 16 becomes the minimum supported Postgres version, the drop
statements can be removed.

---------

Several PG17 Compatibility commits have been merged before this final one.
All these subtasks are done https://github.com/citusdata/citus/issues/7653

See the list below:

Compilation PR: https://github.com/citusdata/citus/pull/7699
Ruleutils PR: https://github.com/citusdata/citus/pull/7725
Sister PR for tests: https://github.com/citusdata/the-process/pull/159

Helpful smaller PRs:
- https://github.com/citusdata/citus/pull/7714
- https://github.com/citusdata/citus/pull/7726
- https://github.com/citusdata/citus/pull/7731
- https://github.com/citusdata/citus/pull/7732
- https://github.com/citusdata/citus/pull/7733
- https://github.com/citusdata/citus/pull/7738
- https://github.com/citusdata/citus/pull/7745
- https://github.com/citusdata/citus/pull/7747
- https://github.com/citusdata/citus/pull/7748
- https://github.com/citusdata/citus/pull/7749
- https://github.com/citusdata/citus/pull/7752
- https://github.com/citusdata/citus/pull/7755
- https://github.com/citusdata/citus/pull/7757
- https://github.com/citusdata/citus/pull/7759
- https://github.com/citusdata/citus/pull/7760
- https://github.com/citusdata/citus/pull/7761
- https://github.com/citusdata/citus/pull/7762
- https://github.com/citusdata/citus/pull/7765
- https://github.com/citusdata/citus/pull/7766
- https://github.com/citusdata/citus/pull/7768
- https://github.com/citusdata/citus/pull/7769
- https://github.com/citusdata/citus/pull/7771
- https://github.com/citusdata/citus/pull/7774
- https://github.com/citusdata/citus/pull/7776
- https://github.com/citusdata/citus/pull/7780
- https://github.com/citusdata/citus/pull/7781
- https://github.com/citusdata/citus/pull/7785
- https://github.com/citusdata/citus/pull/7788
- https://github.com/citusdata/citus/pull/7793
- https://github.com/citusdata/citus/pull/7796

---------

Co-authored-by: Colm <colmmchugh@microsoft.com>
2024-12-24 17:56:51 +03:00
Naisila Puka 743f0ebb75
Bump Citus version into 13.0.0 (#7792)
We are using `release-13.0` branch for both development and release, to
deliver PG17 support in Citus.

Afterwards, we will (probably) merge this branch into main.

Some potential changes for main branch, after we are done working on
release-13.0:
- Merge changes from `release-13.0` to `main`
- Figure out what changes were there on 12.2, move them to 13.1 version.
In a nutshell: rename `12.1--12.2` to `13.0--13.1` and fix issues.
- Set version to 13.1devel
2024-12-24 11:40:59 +03:00
Mehmet YILMAZ 351b1ca63c
PG17 compatibility: Fix Test Failure in multi_alter_table_add_const (#7733)
In earlier versions of PostgreSQL, exclusion constraints were not
allowed on partitioned tables. This is why the error in your regression
test (ERROR: exclusion constraints are not supported on partitioned
tables) was raised in PostgreSQL 16. In PostgreSQL 17, exclusion
constraints are now allowed on partitioned tables, which is why the
error no longer appears when you attempt to add an exclusion constraint.

The constraint exclusion mechanism, described in the documentation,
relies on CHECK constraints to decide which partitions or child tables
need to be queried.

[CHECK
constraints](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-CONSTRAINT-EXCLUSION)

```diff
 -- Check "ADD EXCLUDE" errors out for partitioned table since the postgres does not allow it
 ALTER TABLE AT_AddConstNoName.citus_local_partitioned_table ADD EXCLUDE(partition_col WITH =);
-ERROR:  exclusion constraints are not supported on partitioned tables
 -- Check "ADD CHECK"
 SET client_min_messages TO DEBUG1;
 ALTER TABLE AT_AddConstNoName.citus_local_partitioned_table ADD CHECK (dist_col > 0);
 DEBUG:  the constraint name on the shards of the partition is too long, switching to sequential and local execution mode to prevent self deadlocks: longlonglonglonglonglonglonglonglonglonglonglo_537570f5_5_check
 DEBUG:  verifying table "longlonglonglonglonglonglonglonglonglonglonglonglonglonglongabc"
 DEBUG:  verifying table "p1"
 RESET client_min_messages;
 SELECT con.conname
     FROM pg_catalog.pg_constraint con
       INNER JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid
       INNER JOIN pg_catalog.pg_namespace nsp ON nsp.oid = connamespace
           WHERE rel.relname = 'citus_local_partitioned_table';
                      conname                      
 --------------------------------------------------
+ citus_local_partitioned_table_partition_col_excl
  citus_local_partitioned_table_check
-(1 row)
+(2 rows)
```
2024-12-23 15:24:46 +03:00
Naisila Puka aa8ed4c60b Bump postgres versions in CI and dev: 14.14, 15.9, 16.5 (#7779)
Upgrade postgres versions to:
 - 14.14
 - 15.9
 - 16.5

Depends on https://github.com/citusdata/the-process/pull/163

We had some errors with the latest minors, so this is a 2-level bump for now.
2024-12-23 15:18:53 +03:00
Mehmet YILMAZ 2d843cb382
PG17 compatibility: Fix Isolation Test Failure in isolation_multiuser_locking (#7714)
This PR enhances `isolation_multiuser_locking.spec` test compatibility
across multiple PostgreSQL versions by handling differences in error
messages and behavior. Key updates include:

- **Error Message Handling:** Adjustments to manage version-specific
error messages, ensuring consistent test results.
  
- Modified to address variations in locking behavior across PostgreSQL
versions, ensuring test stability in multiuser scenarios.

- **REINDEX Behavior Adjustment**: This PR accounts for a behavioral
change introduced in PostgreSQL by commit ecb0fd337, which alters how
REINDEX interacts with system catalogs.


https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ecb0fd337

---------

Co-authored-by: Mehmet YILMAZ <mehmet.yilmaz@microsoft.com>
2024-12-23 13:30:54 +03:00
Naisila Puka f123632f85
Fix pg17 test (#7797)
Broken from this commit
e3db375149

https://github.com/citusdata/citus/actions/runs/12429202397/attempts/1#summary-34702334056
2024-12-20 20:13:48 +03:00
Naisila Puka 61d11ec2d4
PG17 Compatibility - Fix HideCitusDependentObjects function (#7796)
There is a crash when running vanilla tests because of the
`citus.hide_citus_dependent_objects` GUC. We turn on this GUC only for
the pg vanilla tests. This GUC runs the following function
`HideCitusDependentObjectsOnQueriesOfPgMetaTables`. This function
doesn't take into account the new `mergeJoinCondition`. I rewrote the
function such that it checks for merge join conditions as well.

Relevant PG commit:
https://github.com/postgres/postgres/commit/0294df2f1

The crash could be reproduced locally like the following:
```SQL
SET citus.hide_citus_dependent_objects TO on;

CREATE OR REPLACE FUNCTION
    pg_catalog.is_citus_depended_object(oid,oid)
    RETURNS bool
    LANGUAGE C
    AS 'citus', $$is_citus_depended_object$$;

-- try a system catalog
MERGE INTO pg_class c
USING (SELECT 'pg_depend'::regclass AS oid) AS j
ON j.oid = c.oid
WHEN MATCHED THEN
UPDATE SET reltuples = reltuples + 1
RETURNING j.oid;

CREATE VIEW classv AS SELECT * FROM pg_class;

MERGE INTO classv c
USING pg_namespace n
ON n.oid = c.relnamespace
WHEN MATCHED AND c.oid = 'pg_depend'::regclass THEN
UPDATE SET reltuples = reltuples - 1
RETURNING c.oid;
-- crash happens here
```
2024-12-20 17:59:09 +03:00
Naisila Puka e4d48dc5da
Remove redundant normalize (#7794)
Redundant from this commit
acd7b1e690
2024-12-20 12:47:36 +03:00
Mehmet YILMAZ e3db375149
PG17 compatibility: Fix Test Failure in local_table_join (#7732)
PostgreSQL 17 seems to have introduced improvements in how correlated
subqueries are handled during plan generation. Instead of generating a
trivial subplan with WHERE true, it now applies more specific filtering
(WHERE (key = 5)), which makes the execution plan more efficient.

https://github.com/postgres/postgres/commit/b262ad44


```
diff -dU10 -w /__w/citus/citus/src/test/regress/expected/local_table_join.out /__w/citus/citus/src/test/regress/results/local_table_join.out
--- /__w/citus/citus/src/test/regress/expected/local_table_join.out.modified	2024-11-05 09:53:50.423970699 +0000
+++ /__w/citus/citus/src/test/regress/results/local_table_join.out.modified	2024-11-05 09:53:50.463971296 +0000
@@ -1420,32 +1420,32 @@
   ) as subq_1
 ) as subq_2;
 DEBUG:  Wrapping relation "custom_pg_type" to a subquery
 DEBUG:  generating subplan 204_1 for subquery SELECT typdefault FROM local_table_join.custom_pg_type WHERE true
 ERROR:  direct joins between distributed and local tables are not supported
 HINT:  Use CTE's or subqueries to select from local tables and use them in joins
 -- correlated sublinks are not yet supported because of #4470, unless we convert not-correlated table
 SELECT COUNT(*) FROM distributed_table d1 JOIN postgres_table using(key)
 WHERE d1.key IN (SELECT key FROM distributed_table WHERE d1.key = key and key = 5);
 DEBUG:  Wrapping relation "postgres_table" to a subquery
-DEBUG:  generating subplan XXX_1 for subquery SELECT key FROM local_table_join.postgres_table WHERE true
+DEBUG:  generating subplan 206_1 for subquery SELECT key FROM local_table_join.postgres_table WHERE (key OPERATOR(pg_catalog.=) 5)
```

Co-authored-by: Naisila Puka <37271756+naisila@users.noreply.github.com>
2024-12-19 22:21:51 +03:00
Mehmet YILMAZ acd7b1e690
PG17 compatibility: Fix Test Failure in local_dist_join_mixed (#7731)
PostgreSQL 16 adds an extra condition (id IS NOT NULL) to the subquery.
This condition is likely used to ensure that no null values are
processed in the subquery. Instead of using the condition id IS NOT
NULL, PostgreSQL 17 generates the subplan with a trivial condition
(WHERE true), indicating that it does not need to explicitly check for
non-null values.

PostgreSQL 17 likely includes optimizations to handle null checks more
efficiently. The WHERE (id IS NOT NULL) condition that was present in
PostgreSQL 16 may now be considered redundant by the planner, as it is
implicitly handled by the query execution engine.

https://github.com/postgres/postgres/commit/b262ad44

```diff
 SELECT
        foo1.id
    FROM
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo9,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo8,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo7,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo6,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo5,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo4,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo3,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo2,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo10,
 (SELECT local.id, local.title FROM local, distributed WHERE local.id = distributed.id ) as foo1
 WHERE
  foo1.id =  foo9.id AND
  foo1.id =  foo8.id AND
  foo1.id =  foo7.id AND
  foo1.id =  foo6.id AND
  foo1.id =  foo5.id AND
  foo1.id =  foo4.id AND
  foo1.id =  foo3.id AND
  foo1.id =  foo2.id AND
  foo1.id =  foo10.id AND
  foo1.id =  foo1.id
ORDER BY 1;
...
-DEBUG:  generating subplan XXX_10 for subquery SELECT id FROM local_dist_join_mixed.local WHERE (id IS NOT NULL)
+DEBUG:  generating subplan XXX_10 for subquery SELECT id FROM local_dist_join_mixed.local WHERE true
...
```
2024-12-19 22:21:23 +03:00
Colm ad9a3bfc1a
PG17 compatibility: ensure get_progress() output is consistent (#7793)
in regress test isolation_progress_monitoring, with an ORDER BY. The
implementation of get_progress() uses a tuplestore to hold the step and
progress values, and tuplestore does not provide any guarantee on the
ordering of the tuples so ORDER BY ensures stable test output. Also make
the output more user friendly by including the column names. Fixing
occasional failures seen in isolation_progress_monitoring.

![Screenshot
(86)](https://github.com/user-attachments/assets/a019639f-559f-408d-b8a8-8b7a44d8095d)
2024-12-19 18:17:57 +00:00
Teja Mupparti a0cd8bd37b
PG17 Compatibility: Support MERGE features in Citus with clean exceptions (#7781)
- Adapted `pgmerge.sql` tests from PostgreSQL community's `merge.sql` to
Citus by converting tables into Citus local tables.
- Identified two new PostgreSQL 17 MERGE features (`RETURNING` support
and MERGE on updatable views) not yet supported by Citus.
- Implemented changes to detect unsupported features and raise clean
exceptions, ensuring pgmerge tests pass without diffs.
- Addressed breaking changes caused by `MERGE ... WHEN NOT MATCHED BY
SOURCE` restructuring, reducing diffs in pgmerge tests.
- Segregated unsupported test cases into `merge_unsupported.sql` to
maintain clarity and avoid large diffs in test files.
- Prepared the Citus MERGE planner to handle new PostgreSQL changes,
reducing remaining test discrepancies.

All merge tests now pass cleanly, with unsupported cases clearly
isolated.

Relevant PG commits:
c649fa24a
https://github.com/postgres/postgres/commit/c649fa24a
0294df2f1
https://github.com/postgres/postgres/commit/0294df2f1
---------

Co-authored-by: naisila <nicypp@gmail.com>
2024-12-19 14:02:24 +03:00
Colm e91ee245ac
PG17 compatibility: account for identity columns in partitioned tables. (#7785)
PG17 added support for identity columns in partitioned tables:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=699586315
A consequence is that a table with an identity column cannot be attached
as a partition. But Citus on Postgres 17 will generate identity column
for the partitions if the parent table has one (or more) identity
columns when propagating distributed table DDL to worker nodes, as
happens in the `generated_identity` regress test in #7768:
```
 CREATE TABLE partitioned_table (
     a bigint CONSTRAINT myconname GENERATED BY DEFAULT AS IDENTITY (START WITH 10 INCREMENT BY 10),
     b bigint GENERATED ALWAYS AS IDENTITY (START WITH 10 INCREMENT BY 10),
     c int
 )
 PARTITION BY RANGE (c);
 CREATE TABLE partitioned_table_1_50 PARTITION OF partitioned_table FOR VALUES FROM (1) TO (50);
 CREATE TABLE partitioned_table_50_500 PARTITION OF partitioned_table FOR VALUES FROM (50) TO (1000);
 SELECT create_distributed_table('partitioned_table', 'a');
- create_distributed_table
----------------------------------------------------------------------
-
-(1 row)
-
+ERROR:  table "partitioned_table_1_50" being attached contains an identity column "a"
+DETAIL:  The new partition may not contain an identity column.
```
It is the Citus-generated ATTACH PARTITION statement that errors out,
because the Citus-generated CREATE TABLE for the partitions included
identity column definitions. The fix is straightforward - when
propagating the CREATE TABLE ddl for a partition of a table with an
identity column, don't include the identity column(s), they will be
inherited on attaching the partition. In Citus on Postgres 16 (or less)
partitions do not inherit identity; the partitions in the example would
not have any identity columns so it was not an issue previously.
2024-12-18 13:18:53 +00:00
Colm 89e0b53644
PG17 compatibility: fix plan diffs in multi_explain (#7780)
Regress test `multi_explain` has two queries that have a different query
plan with PG17. Here is part of the plan diff for the query labelled
_Union and left join subquery pushdown_ in `multi_explain.sql` (for the
complete diff, search for `multi_explain`
[here](https://github.com/citusdata/citus/actions/runs/12158205599/attempts/1)):
```
                                       ->  Sort
                                             Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), subquery_2.hasdone, events.event_time
-                                            ->  Hash Left Join
-                                                  Hash Cond: (users.composite_id = subquery_2.composite_id)
-                                                  ->  HashAggregate
-                                                        Group Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
+                                            ->  Nested Loop Left Join
+                                                  Join Filter: (users.composite_id = subquery_2.composite_id)
+                                                  ->  Unique
+                                                        ->  Sort
+                                                              Sort Key: ((users.composite_id).tenant_id), ((users.composite_id).user_id), users.composite_id, ('action=>1'::text), events.event_time
                                                               ->  Append
```
The change is the same in both queries; a hash left join with subquery_1
on the outer and subquery_2 on the inner side of the join is now a
nested loop left join with subquery_1 on the outer and subquery_2 on the
inner; additionally, the chosen method of uniquifying the UNION in
subquery_1 has changed from hashed grouping to sort followed by unique,
as shown in the diff above.

The PG17 commit that caused this plan change is likely _[Fix MergeAppend
to more accurately compute the number of rows that need to be
sorted](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9d1a5354f)_
because it impacts the estimated rows counts of UNION paths. Comparing a
costed plan of the query between PG16 and PG17 I noticed that with PG16
the rows estimate for the UNION in subquery_1 is 4, whereas with PG17
the rows estimate is 2. A lower rows estimate in the outer side of the
join may result in nested loop looking cheaper than hash join for the
left outer join, hence the plan change in the two queries where there is
a UNION on the outer side of a left outer join.

The proposed fix achieves a consistent plan across all supported
postgres versions by temporarily disabling nested loop join and sort for
the two impacted queries; the postgres optimizer selects hash join for
the outer left join and hashed aggregation for the UNION operation. I
investigated tweaking the queries, but was not able to arrive at a
consistent plan, and I believe the SQL operator (e.g. join, group by,
union) implementations are orthogonal to the intent of the test, so this
should be a satisfactory solution, particularly as it avoids introducing
a second alternative output file for `multi_explain`.
2024-12-17 21:42:15 +00:00
Colm 1c2e78405b
PG17 compatibility: account for MAINTAIN privilege in regress tests (#7774)
This PR addresses regress tests impacted by the introduction of [the
MAINTAIN privilege in
PG17](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ecb0fd337).
The impacted tests include `generated_identity`,
`create_single_shard_table`, `grant_on_sequence_propagation`,
`grant_on_foreign_server_propagation`, `single_node_enterprise`,
`multi_multiuser_master_protocol`,
`multi_alter_table_row_level_security`, `shard_move_constraints` which
show the following error:
```
SELECT start_metadata_sync_to_node('localhost', :worker_2_port);
- start_metadata_sync_to_node
----------------------------------------------------------------------
-
-(1 row)
-
+ERROR:  unrecognized aclright: 16384
```

and `multi_multiuser_master_protocol`, where the `pg_class.relacl`
column has 'm' for MAINTAIN if applicable:
```
        relname       |   rolname   |                           relacl                           
 ---------------------+-------------+------------------------------------------------------------
  trivial_full_access | full_access | 
- trivial_postgres    | postgres    | {postgres=arwdDxt/postgres,full_access=arwdDxt/postgres}
+ trivial_postgres    | postgres    | {postgres=arwdDxtm/postgres,full_access=arwdDxtm/postgres}
```

The PR updates function `convert_aclright_to_string()` in
citus_ruleutils.c to include a case for `ACL_MAINTAIN`. Per the comment
on `convert_aclright_to_string()` in citus_ruleutils.c, it is a copy of
`convert_aclright_to_string()` in Postgres (where it is in
`src/backend/utils/adt/acl.c`), so requires updating to be consistent
with Postgres. With this change Citus can recognize the MAINTAIN
privilege, and will not emit the `unrecognized aclright` error. The PR
also adds an alternative goldfile for `multi_multiuser_master_protocol`.

Note that `convert_aclright_to_string()` in Postgres includes access
types SET and ALTER SYSTEM on system parameters (aka GUCs), added by
[this PG16
commit](https://github.com/postgres/postgres/commit/a0ffa885e). If Citus
were to have a requirement to support granting SET and ALTER SYSTEM we
would need to update `convert_aclright_to_string()` in citus_ruleutils.c
with SET and ALTER SYSTEM.
2024-12-06 13:03:51 +00:00
Colm 7c9280f6e3
PG17 compatibility: fix multi-1 diffs caused by PG17 optimizer enhancements (#7769)
This fix ensures that the expected DEBUG error messages from the router
planner in `multi_router_planner`, `multi_router_planner_fast_path` and
`query_single_shard_table` are present with PG17.

In `query_single_shard_table` the diff:
```
SELECT COUNT(*) FROM citus_local_table t1
 WHERE t1.b IN (
     SELECT b+1 FROM nullkey_c1_t1 t2 WHERE t2.b = t1.a
 );
-DEBUG:  router planner does not support queries that reference non-colocated distributed tables
+DEBUG:  Local tables cannot be used in distributed queries.
```
occurred because of[ this PG17
commit](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9f1337639)
which enables the optimizer to pull up a correlated ANY subquery to a
join. The fix inhibits subquery pull up by including a volatile function
in the predicate involving the ANY subquery, preserving the pre-PG17
optimizer treatment of the query.

In the case of `multi_router_planner` and
`multi_router_planner_fast_path` the diffs:
```
-- partition_column is null clause does not prune out any shards,
 -- all shards remain after shard pruning, not router plannable
 SELECT *
 	FROM articles_hash a
 	WHERE a.author_id is null;
-DEBUG:  Router planner cannot handle multi-shard select queries
+DEBUG:  Creating router plan
```
are because of [this PG17
commit](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=b262ad440),
which enables the optimizer to detect and remove redundant IS (NOT) NULL
expressions. The fix is to adjust the table definition so the column
used for distribution is not marked NOT NULL, thus preserving the
pre-PG17 query planning behavior.

Finallly, a rule is added to `normalize.sed` to ignore DEBUG logging in CREATE MATERIALIZED
VIEW AS statements introduced by [this PG17
commit](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=b4da732fd64);
_when creating materialized views, use REFRESH logic to load data_, a
consequence of which is that with `client_min_messages` at `DEBUG2`
Postgres emits extra detail for CREATE MATERIALIZED VIEW AS statements.
```
CREATE MATERIALIZED VIEW mv_articles_hash_empty AS
 	SELECT * FROM articles_hash WHERE author_id = 1;
 DEBUG:  Creating router plan
 DEBUG:  query has a single distribution column value: 1
+DEBUG:  drop auto-cascades to type multi_router_planner.pg_temp_61391
+DEBUG:  drop auto-cascades to type multi_router_planner.pg_temp_61391[]
```
The rule can be changed to a normalization, or possibly dropped, when 17 becomes the minimum supported version.
2024-12-06 11:55:12 +00:00
Naisila Puka 90d76e8097
Fix bug: alter database shouldn't propagate for undistributed database (#7777)
Before this fix, the following would happen for a database that is not
distributed:

```sql
CREATE DATABASE db_to_test;
NOTICE:  Citus partially supports CREATE DATABASE for distributed databases
DETAIL:  Citus does not propagate CREATE DATABASE command to workers
HINT:  You can manually create a database and its extensions on workers.

ALTER DATABASE db_to_test with CONNECTION LIMIT 100;
NOTICE:  issuing ALTER DATABASE db_to_test WITH  CONNECTION LIMIT 100;
DETAIL:  on server postgres@localhost:57638 connectionId: 2
NOTICE:  issuing ALTER DATABASE db_to_test WITH  CONNECTION LIMIT 100;
DETAIL:  on server postgres@localhost:57637 connectionId: 1
ERROR:  database "db_to_test" does not exist
```

With this fix, we also take care of
https://github.com/citusdata/citus/issues/7763

Fixes #7763
2024-12-05 13:47:51 +03:00
Colm 30a75eadf1
PG17 compatibility: fix diffs in create_index, privileges vanilla tests (#7766)
PG17 regress sanity (#7653) fix; address diffs in vanilla tests
`create_index` and `privileges`. There is a change from `permission
denied` to `must be owner of`, seen in create_index:
```
@@ -2970,21 +2970,21 @@
 REINDEX TABLE pg_toast.pg_toast_1260;
 ERROR:  permission denied for table pg_toast_1260
 REINDEX INDEX pg_toast.pg_toast_1260_index;
-ERROR:  permission denied for index pg_toast_1260_index
+ERROR:  must be owner of index pg_toast_1260_index
```
and privileges:
```
@@ -2945,41 +2945,43 @@
ERROR:  permission denied for table maintain_test
 REINDEX INDEX maintain_test_a_idx;
-ERROR:  permission denied for index maintain_test_a_idx
+ERROR:  must be owner of index maintain_test_a_idx
 REINDEX SCHEMA reindex_test;

 REINDEX INDEX maintain_test_a_idx;
+ERROR:  must be owner of index maintain_test_a_idx
 REINDEX SCHEMA reindex_test;
```

The fix updates function `RangeVarCallbackForReindexIndex()` in
`index.c` with changes made by the introduction of the [MAINTAIN
privilege in
PG17](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ecb0fd337)
to the function `RangeVarCallbackForReindexIndex()` in `indexcmds.c`.
The code is under a Postgres 17 version directive, which can be removed
when 17 becomes the oldest supported Postgres version.
2024-12-05 10:03:28 +00:00
Colm 063be46444
PG17 compatibility: Fix check-style, broken by PG17 columnar test fix… (#7776)
… (698699d89e)

---------

Co-authored-by: naisila <nicypp@gmail.com>
2024-12-04 15:29:39 +00:00
Colm 698699d89e
PG17 compatibility (#7653): Fix test diffs in columnar schedule (#7768)
This PR fixes diffs in `columnnar_chunk_filtering` and `columnar_paths`
tests.

In `columnnar_chunk_filtering` an expression `(NOT (SubPlan 1))` changed
to `(NOT (ANY (a = (SubPlan 1).col1)))`. This is due to [aPG17
commit](https://github.com/postgres/postgres/commit/fd0398fc) that
improved how scalar subqueries (InitPlans) and ANY subqueries (SubPlans)
are EXPLAINed in expressions. The fix uses a helper function which
converts the PG17 format to the pre-PG17 format. It is done this way
because pre-PG17 EXPLAIN does not provide enough context to convert to
the PG17 format. The helper function can (and should) be retired when 17
becomes the minimum supported PG.

In `columnar_paths`, a merge join changed to a hash join. This is due to
[this PG17
commit](f7816aec23),
which improved the PG optimizer's ability to estimate the size of a CTE
scan. The impacted query involves a CTE scan with a point predicate
`(a=123)` and before the change the CTE size was estimated to be 5000,
but with the change it is correctly (given the data in the table)
estimated to be 1, making hash join a more attractive join method. The
fix is to have an alternative goldfile for pre-PG17. I tried, but was
unable, to force a specific kind of join method using the GUCs
(`enable_nestloop`, `enable_hashjoin`, `enable_mergejoin`), but it was
not possible to obtain a consistent plan across all supported PG
versions (in some cases the join inputs switched sides).
2024-12-03 09:14:47 +00:00
Colm 5f479d5f47
PG17 compatibility: revert #7764 (#7775)
Revert PG17 compatibility fix #7764
2024-12-03 08:44:56 +00:00
Naisila Puka 26c09f340f
PG17 compatibility: fix some tests outputs (#7765)
There are two commits in this PR:

1) Remove domain_default column since it has been removed from PG17
Relevant PG commit:
78806a9509
78806a95095c4fb9230a441925244690d9c07d23

2) pg_stat_statements reset output diff fix
pg_stat_statements reset output changed in PG17, fix idea from
Relevant PG commits:
6ab1dbd26b
6ab1dbd26bbf307055d805feaaca16dc3e750d36
2024-12-02 18:03:38 +03:00
Colm 089555c197
PG17 compatibility: fix diff in tableam (#7771)
Test `tableam` expects that this CREATE TABLE statement: `CREATE TABLE
test_partitioned(id int, p int, val int) PARTITION BY RANGE (p) USING
fake_am;`
will produce this error:
`specifying a table access method is not supported on a partitioned
table`

but as of [this PG
commit](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=374c7a229)
it is possible to specify an access method on a partitioned table. This
fix moves the CREATE TABLE statement to pg17, and adds an additional
test to show parent access method is inherited.
2024-12-02 13:47:19 +00:00
Mehmet YILMAZ e9110de7e1
PG17 compatibility: Fix Test Failure in multi_name_lengths multi_create_table_constraints (#7726)
PG 17 Removes outer parentheses from CHECK constraints
we add them back for pg15,pg16 compatibility
e.g. change CHECK other_col >= 100 to CHECK (other_col >= 100)

Relevant PG commit:
e59fcbd712c777eb2987d7c9ad542a7e817954ec
e59fcbd712

CI link https://github.com/citusdata/citus/actions/runs/11844794788

```difft
 SELECT "Constraint", "Definition" FROM table_checks WHERE relid='public.check_example_365068'::regclass;
              Constraint              |            Definition             
 -------------------------------------+-----------------------------------
- check_example_other_col_check       | CHECK (other_col >= 100)
- check_example_other_other_col_check | CHECK (abs(other_other_col) >= 100)
+ check_example_other_col_check       | CHECK other_col >= 100
+ check_example_other_other_col_check | CHECK abs(other_other_col) >= 100
 
```

Co-authored-by: Mehmet YILMAZ <mehmet.yilmaz@microsoft.com>
2024-12-02 13:08:21 +03:00
Colm d5f067a03f
PG17 regress sanity: fix error unrecognized alter database option tablespace seen in database vanilla test (#7764)
Disable DDL propagation for the vanilla test suite. This enables the
vanilla `database ` test to pass, where previously it was correctly
returning `ERROR: unrecognized ALTER DATABASE option: tablespace`
because release-13.0 does not propagate this ALTER DATABASE variant.

We (Citus team) discussed cherry picking
[#7253](https://github.com/citusdata/citus/pull/7253) from main to
release-13.0 because it does propagate ALTER DATABASE tablespace option
(as well as a couple of others) but decided fixing the regress test was
not the proper context for that. The fix disables
`citus.enable_metadata_sync` when running vanilla, we discussed
disabling `citus.enable_create_database_propagation` but this is not in
release-13.0.
2024-11-25 21:11:34 +00:00
Colm 1d0111aed6
PG17 regress test sanity: fix diffs in union_pushdown. (#7762)
Preserve the test error message by adjusting the query so that PG17
cannot pull it up to a join. Another instance of a subquery that can be
pulled up to a join with PG17 (#7745)

This should have been fixed in, but slipped by, #7745
2024-11-22 16:25:22 +00:00
Naisila Puka 12dd9c1f6b
PG17 compatibility: Adjust print_extension_changes function for extra type outputs in PG17 (#7761)
In PG17, Auto-generated array types, multirange types, and relation
rowtypes
are treated as dependent objects, hence changing the output of the
print_extension_changes function.

Relevant PG commit:
e5bc9454e527b1cba97553531d8d4992892fdeef

e5bc9454e5

Here we create a table with only the basic extension types
in order to avoid printing extra ones for now.
This can be removed when we drop PG16 support.


https://github.com/citusdata/citus/actions/runs/11960253650/attempts/1#summary-33343972656
```diff

                  | table pg_dist_rebalance_strategy 
+                 | type citus.distribution_type[] 
+                 | type citus.pg_dist_object 
+                 | type pg_dist_shard 
+                 | type pg_dist_shard[] 
+                 | type pg_dist_shard_placement 
+                 | type pg_dist_shard_placement[] 
+                 | type pg_dist_transaction 
+                 | type pg_dist_transaction[] 
                  | view citus_dist_stat_activity 
                  | view pg_dist_shard_placement 
```
2024-11-22 16:10:02 +03:00
Naisila Puka 65dcc5904d
PG17 compatibility: fix backend type orders in test (#7760)
This work was already done by @m3hm3t and approved as part of
https://github.com/citusdata/citus/pull/7722
I separated it in this PR since the previous one contained other changes
which we don't currently want to merge.

Relevant PG commit:

---------

Co-authored-by: Mehmet YILMAZ <mehmety87@gmail.com>
2024-11-22 01:08:15 +03:00
Colm d7f04aa187
PG17 compatibility: Normalize COPY error messages (#7759)
A recent Postgres commit (*) that refactored error messages is the cause
of the diffs in pg16 regress test when running Citus on Postgres 17. The
fix changes the pg16 goldfile and includes a normalization rule for the
error messages so pg16 will pass when running with version 16 of
Postgres.
    
(*)
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=498ee9ee2f
2024-11-22 00:45:04 +03:00
Colm 7e701befde
PG17 compatibility: add helper function for EXPLAIN diffs in scalar subquery output (#7757)
PG17 changed how scalar subquery outputs appear in EXPLAIN output (*).
This commit changes impacted regress goldfiles to the PG17 format, and
adds a helper function to covert pre-PG17 plans to the PG17 format. The
conversion is required when testing Citus on pgversions prior to 17. The
helper function can and should be removed when 17 becomes the minimum
supported version.

(*)
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fd0398fcb
2024-11-21 22:22:30 +03:00
Colm 680c23ffcf
PG17 compatibility: add/fix tests with correlated subqueries that can be pulled to a join (#7745)
Fix Test Failure in subquery_in_where, set_operations, dml_recursive in
PG17 #7741

The test failures are caused by[ this commit in
PG17](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9f1337639),
which enables correlated subqueries to be pulled up to a join. Prior to
this, the correlated subquery was implemented as a subplan. In citus, it
is not possible to pushdown a correlated subplan, but with a different
plan in PG17 the query can be executed, per the test diff from
`subquery_in_where`:

```
37,39c37,41
< DEBUG:  generating subplan XXX_1 for CTE event_id: SELECT user_id AS events_user_id, "time" AS events_time, event_type FROM public.events_table
< DEBUG:  Plan XXX query after replacing subqueries and CTEs: SELECT count(*) AS count FROM ...
< ERROR:  correlated subqueries are not supported when the FROM clause contains a CTE or subquery
---
>  count
> ---------------------------------------------------------------------
>      0
> (1 row)
> 
```

This is because with pg17 `= ANY subquery` in the queries can be
implemented as a join, instead of as a subplan filter on a table scan.
For example, `SELECT * FROM test a WHERE x IN (SELECT x FROM test b
UNION SELECT y FROM test c WHERE a.x = c.x) ORDER BY 1,2` (from
set_operations) has this plan in pg17; note that the subquery is the
inner side of a nested loop join:
```
┌───────────────────────────────────────────────────┐
│                    QUERY PLAN                     │
├───────────────────────────────────────────────────┤
│ Sort                                              │
│   Sort Key: a.x, a.y                              │
│   ->  Nested Loop                                 │
│         ->  Seq Scan on test a                    │
│         ->  Subquery Scan on "ANY_subquery"       │
│               Filter: (a.x = "ANY_subquery".x)    │
│               ->  HashAggregate                   │
│                     Group Key: b.x                │
│                     ->  Append                    │
│                           ->  Seq Scan on test b  │
│                           ->  Seq Scan on test c  │
│                                 Filter: (a.x = x) │
└───────────────────────────────────────────────────┘
```
and this plan in pg16 (and previous pg versions); the subquery is a
correlated subplan filter on a table scan:
```
┌───────────────────────────────────────────────┐
│                  QUERY PLAN                   │
├───────────────────────────────────────────────┤
│ Sort                                          │
│   Sort Key: a.x, a.y                          │
│   ->  Seq Scan on test a                      │
│         Filter: (SubPlan 1)                   │
│         SubPlan 1                             │
│           ->  HashAggregate                   │
│                 Group Key: b.x                │
│                 ->  Append                    │
│                       ->  Seq Scan on test b  │
│                       ->  Seq Scan on test c  │
│                             Filter: (a.x = x) │
└───────────────────────────────────────────────┘
```

The fix Modifies the queries causing the test failures so that an ANY
subquery is not folded to a join, preserving the expected output of the
tests. A similar approach was taken for existing regress tests in the[
postgres
commit](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9f1337639).
See the `join `regress test, for example.

We also add pg17 specific tests that leverage this improvement in Postgres
with Citus distributed planning as well.
2024-11-20 14:51:16 +03:00
Colm 0fed87ada9
PG17 compatibility: Preserve DEBUG output in cte_inline (#7755)
Regression test cte_inline has the following diff;
```
DEBUG: CTE cte_1 is going to be inlined via distributed planning 
DEBUG: CTE cte_1 is going to be inlined via distributed planning 
DEBUG: Creating router plan 
-DEBUG: query has a single distribution column value: 1 
```
DEBUG message `query has a single distribution column value` does not
appear with PG17. This is because PG17 can recognize when a Result node
does not need to have an input node, so the predicate on the
distribution column is not present in the query plan. Comparing the
query plan obtained before PG17:
```
│ Result                                                                         │
│   One-Time Filter: false                                                       │
│   ->  GroupAggregate                                                           │
│         ->  Seq Scan on public.test_table                                      │
│               Filter: (test_table.key = 1)                                     │

```
with the PG17 query plan:
```
┌──────────────────────────────────┐
│            QUERY PLAN            │
├──────────────────────────────────┤
│ Result                           │
│   One-Time Filter: false         │
└──────────────────────────────────┘
```
we see that the Result node in the PG16 plan has an Aggregate node, but
the Result node in the PG17 plan does not have any input node; PG17
recognizes it is not needed given a Filter that evaluates to False at
compile-time. The Result node is present in both plans because PG in
both versions can recognize when a combination of predicates equate to
false at compile time; this is the because the successive predicates in
the test query (key=6, key=5, key=4, etc) become contradictory when the
CTEs are inlined. Here is an example query showing the effect of the CTE
inlining:
```
select count(*), key FROM test_table WHERE key = 1 AND key = 2 GROUP BY key;
```
In this case, the WHERE clause obviously evaluates to False. The PG16
query plan for this query is:
```
┌────────────────────────────────────┐
│             QUERY PLAN             │
├────────────────────────────────────┤
│ GroupAggregate                     │
│   ->  Result                       │
│         One-Time Filter: false     │
│         ->  Seq Scan on test_table │
│               Filter: (key = 1)    │
└────────────────────────────────────┘
```
The PG17 query plan is:
```
┌────────────────────────────────┐
│           QUERY PLAN           │
├────────────────────────────────┤
│ GroupAggregate                 │
│   ->  Result                   │
│         One-Time Filter: false │
└────────────────────────────────┘
```
In both plans the PG optimizer is able to derive the predicate 1=2 from
the equivalence class { key, 1, 2 } and then constant fold this to
False. But, in the PG16 plan the Result node has an input node (a
sequential scan on test_table), while in the PG17 plan the Result node
does not have any input. This is because PG17 recognizes that when the
Result filter resolves to False at compile time it is not necessary to
set an input on the Result. I think this is a consequence of this PG17
commit:

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=b262ad440
which handles redundant IS [NOT] NULL predicates, but also refactored
evaluating of predicates to true/false at compile-time, enabling
optimizations such as those seen here.

Given the reason for the diff, the fix preserves the test output by
modifying the query so the predicates are not contradictory when the
CTEs are inlined.
2024-11-20 00:14:57 +03:00
Naisila Puka b29ecd1b12
citus_indent fix (#7746) 2024-11-19 13:02:04 +03:00
Naisila Puka ed137001a5
PG17 compatibility: add COLLPROVIDER_BUILTIN option and fix tests (#7752)
In PG17 adds builtin C.UTF-8 locale option, we add it in the code to
avoid "unknown collation provider" in vanilla tests.

Relevant PG commit:

f69319f2f1
f69319f2f1fb16eda4b535bcccec90dff3a6795e

Also in PG17, colliculocale, daticulocale renamed to colllocale,
datlocale
Here we fix the following tests to avoid alternative output
pg15 pg16 multi_mx_create_table multi_schema_support

Relevant PG commit:

f696c0cd5f
f696c0cd5f299f1b51e214efc55a22a782cc175d
2024-11-19 12:26:45 +03:00
Naisila Puka a93887b3de
PG17 compatibility: Check whether table AM is default (#7747)
PG 17 added support for DEFAULT in ALTER TABLE .. SET ACCESS METHOD

Relevant PG commit:
d61a6cad6418f643a5773352038d0dfe5d3535b8
d61a6cad64

In that case, name in `AlterTableCmd->name` would be null.
Add a null check here to avoid crash.
2024-11-18 18:09:43 +03:00
Naisila Puka 84b52fc908
PG17 compatibility - Check if there are blocks left in columnar_scan_analyze_next_block (#7738)
In PG17, the outer loop in `acquire_sample_rows()` changed
from
`while (BlockSampler_HasMore(&bs))`
to
`while (table_scan_analyze_next_block(scan, stream))`

Relevant PG commit:
041b96802efa33d2bc9456f2ad946976b92b5ae1

041b96802e

It is expected that the `scan_analyze_next_block` function will
check if there are any blocks left. So we add that check in
`columnar_scan_analyze_next_block`

Without this fix, we will have an indefinite loop causing timeout.
Specifically, in our test schedules,
`multi schedule` stuck at `drop_column_partitioned_table` test
`multi-mx` schedule stuck at `start_stop_metadata_sync` test
`columnar schedule` stuck at `columnar_create` test
2024-11-18 17:27:49 +03:00
Mehmet YILMAZ 32a2a31b13
PG17 compatibility: Fix -1/Null diff in attstattarget test output (#7749)
Changed `attstattarget` in `pg_attribute` to use `NullableDatum`,
allowing null representation for default statistics target in PostgreSQL
17.

Relevant PG commit:
6a004f1be87d34cfe51acf2fe2552d2b08a79273
6a004f1be8

```diff
 -- verify statistics is set
 SELECT c.relname, a.attstattarget
 FROM pg_attribute a
 JOIN pg_class c ON a.attrelid = c.oid AND c.relname LIKE 'test\_idx%'
 ORDER BY c.relname, a.attnum;
   relname  | attstattarget 
 -----------+---------------
  test_idx  |          4646
- test_idx2 |            -1
+ test_idx2 |              
  test_idx2 |         10000
  test_idx2 |          3737
 (4 rows)
```
2024-11-17 23:43:39 +03:00
Mehmet YILMAZ 8c0feee74d
PG17 compatibility: Fix -1/Null diff in stxstattarget test output (#7748)
Changed stxstattarget in pg_statistic_ext to use nullable
representation, removing explicit -1 for default statistics target in
PostgreSQL 17.

Relevant PG commit:
012460ee93c304fbc7220e5b55d9d0577fc766ab
012460ee93

```diff
SELECT stxstattarget, stxrelid::regclass
FROM pg_statistic_ext
WHERE stxnamespace IN (
	SELECT oid
	FROM pg_namespace
	WHERE nspname IN ('statistics''TestTarget')
)
AND stxname SIMILAR TO '%\_\d+'
ORDER BY stxstattarget, stxrelid::regclass ASC;
  stxstattarget |             stxrelid              
 ---------------+-----------------------------------
-            -1 | "statistics'TestTarget".t1_980000
-            -1 | "statistics'TestTarget".t1_980002
...
+               | "statistics'TestTarget".t1_980000
+               | "statistics'TestTarget".t1_980002
...
```
2024-11-17 22:41:53 +03:00
Mehmet YILMAZ 16ab11622b
Add devcontainer support to release 13.0 (#7739)
Add devcontainer support to release 13.0
2024-11-14 15:30:09 +03:00
Naisila Puka c0a5f5c78c
PG17 compatibility: ruleutils (#7725)
PG17 compatibility - Part 2

https://github.com/citusdata/citus/pull/7699 was the first PG17
compatibility PR merged to main branch, which provided ONLY successful
Citus compilation with PG17.0.

This PR, consider it as Part 2, provides ruleutils changes for PG17.
Ruleutils changes is the first thing we should merge, after successful
build. It's the core for deparsing logic in Citus.

# Question: How do we add ruleutils changes?
- We add a new ruleutils file specific to PG17.
- We keep track of the changes in Postgres's ruleutils file from here
https://github.com/postgres/postgres/commits/REL_17_0/src/backend/utils/adt/ruleutils.c
- Per each commit in that history that belongs only to 17.0, we add the
relevant changes to static functions to our ruleutils file for PG17.
It's like a manual commit copying.

# Check the PR's commits for detailed steps
https://github.com/citusdata/citus/pull/7725/commits
2024-11-11 11:55:10 +03:00
Naisila Puka da2624cee8
PG17 compatibility: Resolve compilation issues (#7699)
This PR provides successful compilation against PG17.0.

- Remove ExecFreeExprContext call
Relevant PG commit
d060e921ea5aa47b6265174c32e1128cebdbc3df
d060e921ea

- PG17 uses streaming IO in analyze, fix scan_analyze_next_block function
Relevant PG commit
041b96802efa33d2bc9456f2ad946976b92b5ae1
041b96802e

- Define ObjectClass for PG17+ only since it's removed
Relevant PG commit:
89e5ef7e21812916c9cf9fcf56e45f0f74034656
89e5ef7e21

- Remove ReorderBufferTupleBuf structure.
Relevant PG commit:
08e6344fd6423210b339e92c069bb979ba4e7cd6
08e6344fd6

- Define colliculocale and daticulocale since they have been renamed
Relevant PG commit:
f696c0cd5f299f1b51e214efc55a22a782cc175d
f696c0cd5f

- makeStringConst defined in PG17
Relevant PG commit:
de3600452b61d1bc3967e9e37e86db8956c8f577
de3600452b

- RangeVarCallbackOwnsTable was replaced by RangeVarCallbackMaintainsTable
Relevant PG commit:
ecb0fd33720fab91df1207e85704f382f55e1eb7
ecb0fd3372

- attstattarget is nullable, define pg compatible functions for it
Relevant PG commit:
4f622503d6de975ac87448aea5cea7de4bc140d5
4f622503d6

- stxstattarget is nullable in PG17, write compat functions for it
Relevant PG commit:
012460ee93c304fbc7220e5b55d9d0577fc766ab
012460ee93

- Use ResourceOwner to track WaitEventSet in PG17
Relevant PG commit:
50c67c2019ab9ade8aa8768bfe604cd802fe8591
50c67c2019

- getIdentitySequence now uses Relation instead of relation_id
Relevant PG commit:
509199587df73f06eda898ae13284292f4ae573a
509199587d

- Remove no-op tuplestore_donestoring function
Relevant PG commit:
75680c3d805e2323cd437ac567f0677fdfc7b680
75680c3d80

- MergeAction can have 3 merge kinds (now enum) in PG17, write compat
Relevant PG commit:
0294df2f1f842dfb0eed79007b21016f486a3c6c
0294df2f1f

- EXPLAIN (MEMORY) is added, make changes to ExplainOnePlan
Relevant PG commit:
5de890e3610d5a12cdaea36413d967cf5c544e20
5de890e361

- LIMIT_OPTION_DEFAULT has been removed as it's useless, use LIMIT_OPTION_COUNT
Relevant PG commit:
a6be0600ac3b71dda8277ab0fcbe59ee101ac1ce
a6be0600ac

- write compat for create_foreignscan_path bcs of more arguments in PG17
Relevant PG commit:
9e9931d2bf40e2fea447d779c2e133c2c1256ef3
9e9931d2bf

- pgprocno and lxid have been combined into a struct in PGPROC
Relevant PG commits:
28f3915b73f75bd1b50ba070f56b34241fe53fd1
28f3915b73

ab355e3a88de745607f6dd4c21f0119b5c68f2ad
ab355e3a88

024c521117579a6d356050ad3d78fdc95e44eefa
024c521117

- Simplify CitusNewNode (#7434)
postgres refactored newNode() in PG 17, the main point for doing this is
the original tricks is no longer neccessary for modern compilers[1].
This does the same for Citus.
This should have no backward compatibility issues since it just replaces
palloc0fast with palloc0.
This is good for forward compatibility since palloc0fast no longer
exists in PG 17.
[1]
https://www.postgresql.org/message-id/b51f1fa7-7e6a-4ecc-936d-90a8a1659e7c@iki.fi
(cherry picked from commit 4b295cc)
2024-10-17 15:37:13 +03:00
Naisila Puka 9d364332ac
Rename foreach_ macros to foreach_declared_ macros (#7700)
This is prep work for successful compilation with PG17

PG17added foreach_ptr, foreach_int and foreach_oid macros
Relevant PG commit
14dd0f27d7cd56ffae9ecdbe324965073d01a9ff

14dd0f27d7

We already have these macros, but they are different with the
PG17 ones because our macros take a DECLARED variable, whereas
the PG16 macros declare a locally-scoped loop variable themselves.

Hence I am renaming our macros to foreach_declared_

I am separating this into its own PR since it touches many files. The
main compilation PR is https://github.com/citusdata/citus/pull/7699
2024-10-16 17:01:39 +03:00
Hanefi Onaldi 15ecc37ecd
Bump Citus to 12.1.5 2024-07-17 15:11:38 +03:00
Hanefi Onaldi 5c2ef8e2d8
Add changelog entries for 12.1.5
(cherry picked from commit 5c097860aa)
2024-07-17 15:11:38 +03:00
Parag Jain 6349f2d52d
Support MERGE command for single_shard_distributed Target (#7643)
This PR has following changes :
1. Enable MERGE command for single_shard_distributed targets.

(cherry picked from commit 3c467e6e02)
2024-07-17 15:11:38 +03:00
Nils Dijk f60c4cbd19
bump postgres versions in CI and dev (#7655)
Upgrade postgres versions to:
 - 14.12
 -  15.7
 - 16.3

Depends on https://github.com/citusdata/the-process/pull/158

(cherry picked from commit accb7d09f7)
2024-07-17 15:11:38 +03:00
Gürkan İndibay f0ea07a813
Removes el/7 and ol/7 as runners (#7650)
Removes el/7 and ol/7 as runners and update checkout action to v4

We use EL/7 and OL/7 runners to test packaging for these distributions.
However, for the past two weeks, we've encountered errors during the
checkout step in the pipelines. The error message is as follows:
```
/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)
```
The GCC version within the EL/7 and OL/7 Docker images is 2.17, and we
cannot upgrade it. Therefore, we need to remove these images from the
packaging test pipelines. Consequently, we will no longer verify if the
code builds for EL/7 and OL/7.

However, we are not using these packaging images as runners within the
packaging infrastructure, so we can continue to use these images for
packaging.

Additional Info: I learned that Marlin team fully dropped the el/7
support so we will drop in further releases as well

(cherry picked from commit c603c3ed74)
2024-07-17 15:11:38 +03:00
Nils Dijk 9dcfcb92ff
CI: move to github container registry (#7652)
We move the CI images to the github container registry.

Given we mostly (if not solely) run these containers on github actions
infra it makes sense to have them hosted closer to where they are
needed.

Image changes: https://github.com/citusdata/the-process/pull/157

(cherry picked from commit e776a7ebbb)
2024-07-17 14:57:50 +03:00
paragjain caee20ad7c fixing expected file of multi_move_mx test 2024-06-18 16:49:39 +02:00
Onur Tirtir d9635609f4 Fix flaky multi_mx_node_metadata.sql test (#7317)
Fixes the flaky test that results in following diff:
```diff
--- /__w/citus/citus/src/test/regress/expected/multi_mx_node_metadata.out.modified	2023-11-01 14:22:12.890476575 +0000
+++ /__w/citus/citus/src/test/regress/results/multi_mx_node_metadata.out.modified	2023-11-01 14:22:12.914476657 +0000
@@ -840,24 +840,26 @@
 (1 row)

 \c :datname - - :master_port
 SELECT datname FROM pg_stat_activity WHERE application_name LIKE 'Citus Met%';
   datname
 ------------
  db_to_drop
 (1 row)

 DROP DATABASE db_to_drop;
+ERROR:  database "db_to_drop" is being accessed by other users
 SELECT datname FROM pg_stat_activity WHERE application_name LIKE 'Citus Met%';
   datname
 ------------
-(0 rows)
+ db_to_drop
+(1 row)

 -- cleanup
 DROP SEQUENCE sequence CASCADE;
 NOTICE:  drop cascades to default value for column a of table reference_table
```

(cherry picked from commit 9867c5b949)
2024-06-18 16:49:39 +02:00
Jelte Fennema-Nio 4f0053ed6d Redo #7620: Fix merge command when insert value does not have source distributed column (#7627)
Related to issue #7619, #7620
Merge command fails when source query is single sharded and source and
target are co-located and insert is not using distribution key of
source.

Example
```
CREATE TABLE source (id integer);
CREATE TABLE target (id integer );

-- let's distribute both table on id field
SELECT create_distributed_table('source', 'id');
SELECT create_distributed_table('target', 'id');

MERGE INTO target t
  USING ( SELECT 1 AS somekey
          FROM source
        WHERE source.id = 1) s
  ON t.id = s.somekey
  WHEN NOT MATCHED
  THEN INSERT (id)
    VALUES (s.somekey)

ERROR:  MERGE INSERT must use the source table distribution column value
HINT:  MERGE INSERT must use the source table distribution column value
```

Author's Opinion: If join is not between source and target distributed
column, we should not force user to use source distributed column while
inserting value of target distributed column.

Fix: If user is not using distributed key of source for insertion let's
not push down query to workers and don't force user to use source
distributed column if it is not part of join.

This reverts commit fa4fc0b372.

Co-authored-by: paragjain <paragjain@microsoft.com>
(cherry picked from commit aaaf637a6b)
2024-06-18 16:49:39 +02:00
Jelte Fennema-Nio 3594bd7ac0 Fix CI issues after Github Actions networking changes (#7624)
For some reason using localhost in our hba file doesn't have the
intended effect anymore in our Github Actions runners. Probably because
of some networking change (IPv6 maybe) or some change in the
`/etc/hosts` file.

Replacing localhost with the equivalent loopback IPv4 and IPv6 addresses
resolved this issue.

(cherry picked from commit 8c9de08b76)
2024-06-18 16:49:39 +02:00
Gürkan İndibay 7e0dc18b22
Bump Citus version to 12.1.4 (#7610) 2024-05-29 11:35:08 +03:00
Gürkan İndibay 4e838a471a
Adds null check for node in HasRangeTableRef (#7604)
DESCRIPTION: Adds null check for node in HasRangeTableRef to prevent
errors

When executing the query below, users encountered an error due to a null
Node object. This PR adds a null check to handle this error.

Query:
```sql
select
    ct.conname as constraint_name,
    a.attname as column_name,
    fc.relname as foreign_table_name,
    fns.nspname as foreign_table_schema,
    fa.attname as foreign_column_name
from
    (SELECT ct.conname, ct.conrelid, ct.confrelid, ct.conkey, ct.contype,
ct.confkey, generate_subscripts(ct.conkey, 1) AS s
       FROM pg_constraint ct
    ) AS ct
    inner join pg_class c on c.oid=ct.conrelid
    inner join pg_namespace ns on c.relnamespace=ns.oid
    inner join pg_attribute a on a.attrelid=ct.conrelid and a.attnum =
ct.conkey[ct.s]
    left join pg_class fc on fc.oid=ct.confrelid
    left join pg_namespace fns on fc.relnamespace=fns.oid
    left join pg_attribute fa on fa.attrelid=ct.confrelid and fa.attnum =
ct.confkey[ct.s]
where
    ct.contype='f'
    and c.relname='table1'
    and ns.nspname='schemauser'
order by
    fns.nspname, fc.relname, a.attnum
;
```

Error:
```
#0  HasRangeTableRef (node=0x0, varno=varno@entry=0x7ffe18cc3674) at worker/worker_shard_visibility.c:507
507             if (IsA(node, RangeTblRef))
#0  HasRangeTableRef (node=0x0, varno=varno@entry=0x7ffe18cc3674) at worker/worker_shard_visibility.c:507
#1  0x0000561b0aae390e in expression_tree_walker_impl (node=0x561b0d19cc78, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2091
#2  0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#3  0x0000561b0aae3e09 in expression_tree_walker_impl (node=0x561b0d19cd68, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=context@entry=0x7ffe18cc3674)
    at nodeFuncs.c:2405
#4  0x0000561b0aae3945 in expression_tree_walker_impl (node=0x561b0d19d0f8, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2111
#5  0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#6  0x0000561b0aae3e09 in expression_tree_walker_impl (node=0x561b0d19cb38, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=context@entry=0x7ffe18cc3674)
    at nodeFuncs.c:2405
#7  0x0000561b0aae396d in expression_tree_walker_impl (node=0x561b0d19d198, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2127
#8  0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#9  0x0000561b0aae3ef7 in expression_tree_walker_impl (node=0x561b0d183e88, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2464
#10 0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#11 0x0000561b0aae3ed3 in expression_tree_walker_impl (node=0x561b0d184278, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2460
#12 0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#13 0x0000561b0aae3ed3 in expression_tree_walker_impl (node=0x561b0d184668, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2460
#14 0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#15 0x0000561b0aae3ed3 in expression_tree_walker_impl (node=0x561b0d184f68, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=0x7ffe18cc3674)
    at nodeFuncs.c:2460
#16 0x00007f2a73249f26 in HasRangeTableRef (node=<optimized out>, varno=<optimized out>) at worker/worker_shard_visibility.c:513
#17 0x0000561b0aae3e09 in expression_tree_walker_impl (node=0x7f2a68010148, walker=walker@entry=0x7f2a73249f0a <HasRangeTableRef>, context=context@entry=0x7ffe18cc3674)
    at nodeFuncs.c:2405
#18 0x00007f2a7324a0eb in FilterShardsFromPgclass (node=node@entry=0x561b0d185de8, context=context@entry=0x0) at worker/worker_shard_visibility.c:464
#19 0x00007f2a7324a5ff in HideShardsFromSomeApplications (query=query@entry=0x561b0d185de8) at worker/worker_shard_visibility.c:294
#20 0x00007f2a731ed7ac in distributed_planner (parse=0x561b0d185de8, 
    query_string=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"..., cursorOptions=<optimized out>, boundParams=0x0) at planner/distributed_planner.c:237
#21 0x00007f2a7311a52a in pgss_planner (parse=0x561b0d185de8, 
    query_string=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"..., cursorOptions=2048, boundParams=0x0) at pg_stat_statements.c:953
#22 0x0000561b0ab65465 in planner (parse=parse@entry=0x561b0d185de8, 
    query_string=query_string@entry=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"..., cursorOptions=cursorOptions@entry=2048, boundParams=boundParams@entry=0x0)
    at planner.c:279
#23 0x0000561b0ac53aa3 in pg_plan_query (querytree=querytree@entry=0x561b0d185de8, 
    query_string=query_string@entry=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"..., cursorOptions=cursorOptions@entry=2048, boundParams=boundParams@entry=0x0)
    at postgres.c:904
#24 0x0000561b0ac53b71 in pg_plan_queries (querytrees=0x7f2a68012878, 
    query_string=query_string@entry=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"..., cursorOptions=cursorOptions@entry=2048, boundParams=boundParams@entry=0x0)
    at postgres.c:996
#25 0x0000561b0ac5408e in exec_simple_query (
    query_string=query_string@entry=0x561b0d009478 "select\n    ct.conname as constraint_name,\n    a.attname as column_name,\n    fc.relname as foreign_table_name,\n    fns.nspname as foreign_table_schema,\n    fa.attname as foreign_column_name\nfrom\n    (S"...) at postgres.c:1193
#26 0x0000561b0ac56116 in PostgresMain (dbname=<optimized out>, username=<optimized out>) at postgres.c:4637
#27 0x0000561b0abab7a7 in BackendRun (port=port@entry=0x561b0d0caf50) at postmaster.c:4464
#28 0x0000561b0abae969 in BackendStartup (port=port@entry=0x561b0d0caf50) at postmaster.c:4192
#29 0x0000561b0abaeaa6 in ServerLoop () at postmaster.c:1782
```


Fixes #7603
2024-05-28 08:54:40 +03:00
Gürkan İndibay 035aa6eada
Bump Citus version to 12.1.3 (#7588) 2024-04-24 11:15:04 +03:00
Gürkan İndibay 75ff237340 Removes unnecessary package installations in packaging pipelines (#7341)
With the recent changes in packaging images, linux package installations
to execute validate_output is unnecessary now.
In this PR, I removed them to make the pipeline more effective.

- [x] Remove the test warning before merge

(cherry picked from commit 32b0fc23f5)
2024-04-17 10:26:50 +02:00
Gürkan İndibay 40e9e2614d Removes centos 7 for PG 16 in packaging pipelines (#7205)
centos 7 and oracle 7 is not being supported for newer releases by
Postgres. Therefore, getting package download errors in packaging
pipelines.
This PR removes el/7 and ol/7 Postgres 16 pipelines

(cherry picked from commit b0e982d0b5)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio bac95cc523 Greatly speed up "\d tablename" on servers with many tables (#7577)
DESCRIPTION: Fix performance issue when using "\d tablename" on a server
with many tables

We introduce a filter to every query on pg_class to automatically remove
shards. This is useful to make sure \d and PgAdmin are not cluttered
with shards. However, the way we were introducing this filter was using
`securityQuals` which can have negative impact on query performance.

On clusters with 100k+ tables this could cause a simple "\d tablename"
command to take multiple seconds, because a skipped optimization by
Postgres causes a full table scan. This changes the code to introduce
this filter in the regular `quals` list instead of in `securityQuals`.
Which causes Postgres to use the intended optimization again.

For reference, this was initially reported as a Postgres issue by me:

https://www.postgresql.org/message-id/flat/4189982.1712785863%40sss.pgh.pa.us#b87421293b362d581ea8677e3bfea920
(cherry picked from commit a0151aa31d)
2024-04-17 10:26:50 +02:00
Xing Guo 38967491ef Add missing volatile qualifier. (#7570)
Variables being modified in the PG_TRY block and read in the PG_CATCH
block should be qualified with volatile.

The variable waitEventSet is modified in the PG_TRY block (line 1085)
and read in the PG_CATCH block (line 1095).

The variable relation is modified in the PG_TRY block (line 500) and
read in the PG_CATCH block (line 515).

Besides, the variable objectAddress doesn't need the volatile qualifier.

Ref: C99 7.13.2.1[^1],

> All accessible objects have values, and all other components of the
abstract machine have state, as of the time the longjmp function was
called, except that the values of objects of automatic storage duration
that are local to the function containing the invocation of the
corresponding setjmp macro that do not have volatile-qualified type and
have been changed between the setjmp invocation and longjmp call are
indeterminate.

[^1]: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf

DESCRIPTION: Correctly mark some variables as volatile

---------

Co-authored-by: Hong Yi <zouzou0208@gmail.com>
(cherry picked from commit ada3ba2507)
2024-04-17 10:26:50 +02:00
Filip Sedlák fc09e1cfdc Log username in the failed connection message (#7432)
This patch includes the username in the reported error message.
This makes debugging easier when certain commands open connections
as other users than the user that is executing the command.

```
monitora_snapshot=# SELECT citus_move_shard_placement(102030, 'monitora.db-dev-worker-a', 6005, 'monitora.db-dev-worker-a', 6017);
ERROR:  connection to the remote node monitora_user@monitora.db-dev-worker-a:6017 failed with the following error: fe_sendauth: no password supplied
Time: 40,198 ms
```

(cherry picked from commit 8b48d6ab02)
2024-04-17 10:26:50 +02:00
Karina 7513061057 Make isolation_update_node test system independent (#7423)
Test isolation_update_node fails on some systems with the following error:
```
-s2: WARNING:  connection to the remote node non-existent:57637 failed with the following error: could not translate host name "non-existent" to address: Name or service not known
+s2: WARNING:  connection to the remote node non-existent:57637 failed with the following error: could not translate host name "non-existent" to address: Temporary failure in name resolution
```

This slightly modifies an already existing [normalization
rule](739c6d26df/src/test/regress/bin/normalize.sed (L217-L218))
to fix it.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 21464adfec)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio f4af59ab4b Support running isolation_update_node in flaky test detection (#7425)
I noticed in #7423 that `isolation_update_node` could not be run using
flaky test detection. This fixes that.
2024-04-17 10:26:50 +02:00
sminux 5708fca1ea fix bad copy-paste rightComparisonLimit (#7547)
DESCRIPTION: change for #7543
(cherry picked from commit d59c93bc50)
2024-04-17 10:26:50 +02:00
LightDB Enterprise Postgres 2a6164d2d9 Fix timeout when underlying socket is changed in a MultiConnection (#7377)
When there are multiple localhost entries in /etc/hosts like following
/etc/hosts:
```
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1   localhost
```

multi_cluster_management check will failed:
```

@@ -857,20 +857,21 @@
 ERROR:  group 14 already has a primary node
 -- check that you can add secondaries and unavailable nodes to a group
 SELECT groupid AS worker_2_group FROM pg_dist_node WHERE nodeport = :worker_2_port \gset
 SELECT 1 FROM master_add_node('localhost', 9998, groupid => :worker_1_group, noderole => 'secondary');
  ?column?
 ----------
         1
 (1 row)

 SELECT 1 FROM master_add_node('localhost', 9997, groupid => :worker_1_group, noderole => 'unavailable');
+WARNING:  could not establish connection after 5000 ms
  ?column?
 ----------
         1
 (1 row)
```

This actually isn't just a problem in test environments, but could occur
as well during actual usage when a hostname in pg_dist_node
resolves to multiple IPs and one of those IPs is unreachable.
Postgres will then automatically continue with the next IP, but
Citus should listen for events on the new socket. Not on the
old one.

Co-authored-by: chuhx43211 <chuhx43211@hundsun.com>
(cherry picked from commit 9a91136a3d)
2024-04-17 10:26:50 +02:00
eaydingol db391c0bb7 Change the order in which the locks are acquired (#7542)
This PR changes the order in which the locks are acquired (for the
target and reference tables), when a modify request is initiated from a
worker node that is not the "FirstWorkerNode".

To prevent concurrent writes, locks are acquired on the first worker
node for the replicated tables. When the update statement originates
from the first worker node, it acquires the lock on the reference
table(s) first, followed by the target table(s). However, if the update
statement is initiated in another worker node, the lock requests are
sent to the first worker in a different order. This PR unifies the
modification order on the first worker node. With the third commit,
independent of the node that received the request, the locks are
acquired for the modified table and then the reference tables on the
first node.

The first commit shows a sample output for the test prior to the fix.

Fixes #7477

---------

Co-authored-by: Jelte Fennema-Nio <jelte.fennema@microsoft.com>
(cherry picked from commit 8afa2d0386)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio 146725fc9b Replace more spurious strdups with pstrdups (#7441)
DESCRIPTION: Remove a few small memory leaks

In #7440 one instance of a strdup was removed. But there were a few
more. This removes the ones that are left over, or adds a comment why
strdup is on purpose.

(cherry picked from commit 9683bef2ec)
2024-04-17 10:26:50 +02:00
Marco Slot 94ab1dc240 Replace spurious strdup with pstrdup (#7440)
Not sure why we never found this using valgrind, but using strdup will
cause memory leaks because the pointer is not tracked in a memory
context.

(cherry picked from commit 72fbea20c4)
2024-04-17 10:26:50 +02:00
Onur Tirtir 812a2b759f Improve error message for recursive CTEs (#7407)
Fixes #2870

(cherry picked from commit 5aedec4242)
2024-04-17 10:26:50 +02:00
Onur Tirtir 452564c19b Allow providing "host" parameter via citus.node_conninfo (#7541)
And when that is the case, directly use it as "host" parameter for the
connections between nodes and use the "hostname" provided in
pg_dist_node / pg_dist_poolinfo as "hostaddr" to avoid host name lookup.

This is to avoid allowing dns resolution (and / or setting up DNS names
for each host in the cluster). This already works currently when using
IPs in the hostname. The only use of setting host is that you can then
use sslmode=verify-full and it will validate that the hostname matches
the certificate provided by the node you're connecting too.

It would be more flexible to make this a per-node setting, but that
requires SQL changes. And we'd like to backport this change, and
backporting such a sql change would be quite hard while backporting this
change would be very easy. And in many setups, a different hostname for
TLS validation is actually not needed. The reason for that is
query-from-any node: With query-from-any-node all nodes usually have a
certificate that is valid for the same "cluster hostname", either using
a wildcard cert or a Subject Alternative Name (SAN). Because if you load
balance across nodes you don't know which node you're connecting to, but
you still want TLS validation to do it's job. So with this change you
can use this same "cluster hostname" for TLS validation within the
cluster. Obviously this means you don't validate that you're connecting
to a particular node, just that you're connecting to one of the nodes in
the cluster, but that should be fine from a security perspective (in
most cases).

Note to self: This change requires updating

https://docs.citusdata.com/en/latest/develop/api_guc.html#citus-node-conninfo-text.

DESCRIPTION: Allows overwriting host name for all inter-node connections
by supporting "host" parameter in citus.node_conninfo

(cherry picked from commit 3586aab17a)
2024-04-17 10:26:50 +02:00
Karina 9b06d02c43 Fix error in master_disable_node/citus_disable_node (#7492)
This fixes #7454: master_disable_node() has only two arguments, but
calls citus_disable_node() that tries to read three arguments

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 683e10ab69)
2024-04-17 10:26:50 +02:00
copetol 2ee43fd00c Fix segfault when using certain DO block in function (#7554)
When using a CASE WHEN expression in the body
of the function that is used in the DO block, a segmentation
fault occured. This fixes that.

Fixes #7381

---------

Co-authored-by: Konstantin Morozov <vzbdryn@yahoo.com>
(cherry picked from commit 12f56438fc)
2024-04-17 10:26:50 +02:00
Emel Şimşek f2d102d54b Fix crash caused by some form of ALTER TABLE ADD COLUMN statements. (#7522)
DESCRIPTION: Fixes a crash caused by some form of ALTER TABLE ADD COLUMN
statements. When adding multiple columns, if one of the ADD COLUMN
statements contains a FOREIGN constraint ommitting the referenced
columns in the statement, a SEGFAULT occurs.

For instance, the following statement results in a crash:

```
  ALTER TABLE lt ADD COLUMN new_col1 bool,
                          ADD COLUMN new_col2 int references rt;

```

Fixes #7520.

(cherry picked from commit fdd658acec)
2024-04-17 10:26:50 +02:00
Karina 82637f3e13 Use expecteddir option in _run_pg_regress() (#7582)
Fix check-arbitrary-configs tests failure with current REL_16_STABLE.
This is the same problem as described in #7573. I missed pg_regress call
in _run_pg_regress() in that PR.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 41e2af8ff5)
2024-04-17 10:26:50 +02:00
Karina 79616bc7db Use expecteddir option when running vanilla tests (#7573)
In PostgreSQL 16 a new option expecteddir was introduced to pg_regress.
Together with fix in
[196eeb6b](https://github.com/postgres/postgres/commit/196eeb6b) it
causes check-vanilla failure if expecteddir is not specified.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
(cherry picked from commit 41d99249d9)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio 364e8ece14 Speed up GetForeignKeyOids (#7578)
DESCRIPTION: Fix performance issue in GetForeignKeyOids on systems with
many constraints

GetForeignKeyOids was showing up in CPU profiles when distributing
schemas on systems with 100k+ constraints. The reason was that this
function was doing a sequence scan of pg_constraint to get the foreign
keys that referenced the requested table.

This fixes that by finding the constraints referencing the table through
pg_depend instead of pg_constraint. We're doing this indirection,
because pg_constraint doesn't have an index that we can use, but
pg_depend does.

(cherry picked from commit a263ac6f5f)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio 62c32067f1 Use an index to get FDWs that depend on extensions (#7574)
DESCRIPTION: Fix performance issue when distributing a table that
depends on an extension

When the database contains many objects this function would show up in
profiles because it was doing a sequence scan on pg_depend. And with
many objects pg_depend can get very large.

This starts using an index scan to only look for rows containing FDWs,
of which there are expected to be very few (often even zero).

(cherry picked from commit 16604a6601)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio d9069b1e01 Speed up SequenceUsedInDistributedTable (#7579)
DESCRIPTION: Fix performance issue when creating distributed tables if
many already exist

This builds on the work to speed up EnsureSequenceTypeSupported, and now
does something similar for SequenceUsedInDistributedTable.
SequenceUsedInDistributedTable had a similar O(number of citus tables)
operation. This fixes that and speeds up creation of distributed tables
significantly when many distributed tables already exist.

Fixes #7022

(cherry picked from commit cdf51da458)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio fa6743d436 Speed up EnsureSequenceTypeSupported (#7575)
DESCRIPTION: Fix performance issue when creating distributed tables and many already exist

EnsureSequenceTypeSupported was doing an O(number of distributed tables)
operation. This can become very slow with lots of Citus tables, which
now happens much more frequently in practice due to schema based sharding.

Partially addresses #7022

(cherry picked from commit 381f31756e)
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio 26178fb538 Add missing postgres.h includes
After sorting includes in the previous commit some files were now
invalid because they were not including postgres.h
2024-04-17 10:26:50 +02:00
Jelte Fennema-Nio 48c62095ff Actually sort includes after cherry-pick 2024-04-17 10:26:50 +02:00
Nils Dijk f1b1d7579c Sort includes (#7326)
CHERRY-PICK NOTES: This cherry-pick only includes the scripts, not the
actual changes. These are done in a follow up commit to ease further
backporting.

This change adds a script to programatically group all includes in a
specific order. The script was used as a one time invocation to group
and sort all includes throught our formatted code. The grouping is as
follows:

 - System includes (eg. `#include<...>`)
 - Postgres.h (eg. `#include "postgres.h"`)
- Toplevel imports from postgres, not contained in a directory (eg.
`#include "miscadmin.h"`)
 - General postgres includes (eg . `#include "nodes/..."`)
- Toplevel citus includes, not contained in a directory (eg. `#include
"citus_verion.h"`)
 - Columnar includes (eg. `#include "columnar/..."`)
 - Distributed includes (eg. `#include "distributed/..."`)

Because it is quite hard to understand the difference between toplevel
citus includes and toplevel postgres includes it hardcodes the list of
toplevel citus includes. In the same manner it assumes anything not
prefixed with `columnar/` or `distributed/` as a postgres include.

The sorting/grouping is enforced by CI. Since we do so with our own
script there are not changes required in our uncrustify configuration.

(cherry picked from commit 0620c8f9a6)
2024-04-17 10:26:50 +02:00
Nils Dijk 75df19b616 move pg_version_constants.h to toplevel include (#7335)
In preparation of sorting and grouping all includes we wanted to move
this file to the toplevel includes for good grouping/sorting.

(cherry picked from commit 0dac63afc0)
2024-04-17 10:26:50 +02:00
Gürkan İndibay e2d18c5472
Bump Citus version to 12.1.2 (#7504) 2024-02-14 08:41:15 +03:00
Gürkan İndibay c12a4f7626
Adds Changelog for v12.1.2 (#7499) 2024-02-13 16:44:57 +03:00
Teja Mupparti a945971f48 Fix the incorrect column count after ALTER TABLE, this fixes the bug #7378 (please read the analysis in the bug for more information)
(cherry picked from commit 00068e07c5)
2024-01-24 11:48:06 -08:00
Hanefi Onaldi 4c110faf1b
Fix wrong PR links in changelog (#7350)
When preparing changelog for 12.1.1 release, I accidentally swapped
the PR numbers for the two commits. This commit fixes the changelog
to point to the correct PRs.

(cherry picked from commit 5efd3f181a)
2023-11-16 14:13:20 +03:00
Hanefi Onaldi 2c630eca50
Bump Citus version to 12.1.1 2023-11-13 14:47:11 +03:00
Hanefi Onaldi b421479d46
Add changelog entries for 12.1.1 (#7332)
Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
(cherry picked from commit 92228b279a)
2023-11-13 14:47:11 +03:00
Gokhan Gulbiz 2502e7e754
Backport GHA Migration to release-12.1 (#7277)
Co-authored-by: Jelte Fennema-Nio <jelte.fennema@microsoft.com>
2023-11-13 11:46:31 +00:00
Onur Tirtir a4fe969947 Make sure to disallow creating a replicated distributed table concurrently (#7219)
See explanation in https://github.com/citusdata/citus/issues/7216.
Fixes https://github.com/citusdata/citus/issues/7216.

DESCRIPTION: Makes sure to disallow creating a replicated distributed
table concurrently

(cherry picked from commit 111b4c19bc)
2023-10-24 14:04:37 +03:00
Nils Dijk e59ffbf549
Fix leaking of memory and memory contexts in Foreign Constraint Graphs (#7236)
DESCRIPTION: Fix leaking of memory and memory contexts in Foreign
Constraint Graphs

Previously, every time we (re)created the Foreign Constraint
Relationship Graph, we created a new Memory Context while loosing a
reference to the previous context. This old context could still have
left over memory in there causing a memory leak.

With this patch we statically have one memory context that we lazily
initialize the first time we create our foreign constraint relationship
graph. On every subsequent creation, beside destroying our previous
hashmap we also reset our memory context to remove any left over
references.
2023-10-09 13:07:30 +02:00
aykut-bozkurt 3b908eec2a
Fix the changelog entry for citus_pause_node_within_txn() UDF (#7215)
(cherry picked from commit 2c190d0689)
2023-09-20 17:06:07 +03:00
Naisila Puka 9b6ffece5e
Adds PostgreSQL 16.0 Support (#7201)
This commit concludes PG16.0 Support in Citus.

The main PG16 support work has been done for 16beta3
https://github.com/citusdata/citus/pull/6952
There was some extra work needed for 16rc1
https://github.com/citusdata/citus/pull/7173
And this PR yet introduces some extra work needed to 16.0 :)

`pgstat_fetch_stat_local_beentry` has been renamed to
`pgstat_get_local_beentry_by_index` in PG16.0

Relevant PG commit:
8dfa37b797
8dfa37b797843a83a5756ea3309055e8953e1a86

Sister PR
https://github.com/citusdata/the-process/pull/150

(cherry picked from commit 4e46708789)
2023-09-15 12:27:09 +03:00
aykut-bozkurt 1b4d7a51f8
bump citus into 12.1.0 2023-09-13 14:20:21 +03:00
575 changed files with 31598 additions and 32337 deletions

View File

@ -6,12 +6,9 @@ RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# install build tools
RUN apt update && apt install -y \
bison \
bzip2 \
cpanminus \
curl \
docbook-xml \
docbook-xsl \
flex \
gcc \
git \
@ -23,7 +20,6 @@ RUN apt update && apt install -y \
libreadline-dev \
libselinux1-dev \
libssl-dev \
libxml2-utils \
libxslt-dev \
libzstd-dev \
locales \
@ -36,7 +32,6 @@ RUN apt update && apt install -y \
sudo \
uuid-dev \
valgrind \
xsltproc \
zlib1g-dev \
&& add-apt-repository ppa:deadsnakes/ppa -y \
&& apt install -y \
@ -72,8 +67,20 @@ ENV PATH="/home/citus/.pgenv/pgsql/bin:${PATH}"
USER citus
# build postgres versions separately for effective parrallelism and caching of already built versions when changing only certain versions
FROM base AS pg14
RUN MAKEFLAGS="-j $(nproc)" pgenv build 14.15
RUN rm .pgenv/src/*.tar*
RUN make -C .pgenv/src/postgresql-*/ clean
RUN make -C .pgenv/src/postgresql-*/src/include install
# create a staging directory with all files we want to copy from our pgenv build
# we will copy the contents of the staged folder into the final image at once
RUN mkdir .pgenv-staging/
RUN cp -r .pgenv/src .pgenv/pgsql-* .pgenv/config .pgenv-staging/
RUN rm .pgenv-staging/config/default.conf
FROM base AS pg15
RUN MAKEFLAGS="-j $(nproc)" pgenv build 15.13
RUN MAKEFLAGS="-j $(nproc)" pgenv build 15.10
RUN rm .pgenv/src/*.tar*
RUN make -C .pgenv/src/postgresql-*/ clean
RUN make -C .pgenv/src/postgresql-*/src/include install
@ -85,7 +92,7 @@ RUN cp -r .pgenv/src .pgenv/pgsql-* .pgenv/config .pgenv-staging/
RUN rm .pgenv-staging/config/default.conf
FROM base AS pg16
RUN MAKEFLAGS="-j $(nproc)" pgenv build 16.9
RUN MAKEFLAGS="-j $(nproc)" pgenv build 16.6
RUN rm .pgenv/src/*.tar*
RUN make -C .pgenv/src/postgresql-*/ clean
RUN make -C .pgenv/src/postgresql-*/src/include install
@ -97,7 +104,7 @@ RUN cp -r .pgenv/src .pgenv/pgsql-* .pgenv/config .pgenv-staging/
RUN rm .pgenv-staging/config/default.conf
FROM base AS pg17
RUN MAKEFLAGS="-j $(nproc)" pgenv build 17.5
RUN MAKEFLAGS="-j $(nproc)" pgenv build 17.2
RUN rm .pgenv/src/*.tar*
RUN make -C .pgenv/src/postgresql-*/ clean
RUN make -C .pgenv/src/postgresql-*/src/include install
@ -198,9 +205,9 @@ RUN git clone https://github.com/so-fancy/diff-so-fancy.git \
COPY --link --from=uncrustify-builder /uncrustify/usr/ /usr/
COPY --link --from=pg14 /home/citus/.pgenv-staging/ /home/citus/.pgenv/
COPY --link --from=pg15 /home/citus/.pgenv-staging/ /home/citus/.pgenv/
COPY --link --from=pg16 /home/citus/.pgenv-staging/ /home/citus/.pgenv/
COPY --link --from=pg17 /home/citus/.pgenv-staging/ /home/citus/.pgenv/
COPY --link --from=pipenv /home/citus/.local/share/virtualenvs/ /home/citus/.local/share/virtualenvs/
@ -216,7 +223,7 @@ COPY --chown=citus:citus .psqlrc .
RUN sudo chown --from=root:root citus:citus -R ~
# sets default pg version
RUN pgenv switch 17.5
RUN pgenv switch 17.2
# make connecting to the coordinator easy
ENV PGPORT=9700

View File

@ -329,12 +329,11 @@
},
"jinja2": {
"hashes": [
"sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d",
"sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67"
"sha256:7d6d50dd97d52cbc355597bd845fabfbac3f551e1f99619e39a35ce8c370b5fa",
"sha256:ac8bd6544d4bb2c9792bf3a159e80bba8fda7f07e81bc3aed565432d5925ba90"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==3.1.6"
"version": "==3.1.3"
},
"kaitaistruct": {
"hashes": [
@ -354,70 +353,69 @@
},
"markupsafe": {
"hashes": [
"sha256:0bff5e0ae4ef2e1ae4fdf2dfd5b76c75e5c2fa4132d05fc1b0dabcd20c7e28c4",
"sha256:0f4ca02bea9a23221c0182836703cbf8930c5e9454bacce27e767509fa286a30",
"sha256:1225beacc926f536dc82e45f8a4d68502949dc67eea90eab715dea3a21c1b5f0",
"sha256:131a3c7689c85f5ad20f9f6fb1b866f402c445b220c19fe4308c0b147ccd2ad9",
"sha256:15ab75ef81add55874e7ab7055e9c397312385bd9ced94920f2802310c930396",
"sha256:1a9d3f5f0901fdec14d8d2f66ef7d035f2157240a433441719ac9a3fba440b13",
"sha256:1c99d261bd2d5f6b59325c92c73df481e05e57f19837bdca8413b9eac4bd8028",
"sha256:1e084f686b92e5b83186b07e8a17fc09e38fff551f3602b249881fec658d3eca",
"sha256:2181e67807fc2fa785d0592dc2d6206c019b9502410671cc905d132a92866557",
"sha256:2cb8438c3cbb25e220c2ab33bb226559e7afb3baec11c4f218ffa7308603c832",
"sha256:3169b1eefae027567d1ce6ee7cae382c57fe26e82775f460f0b2778beaad66c0",
"sha256:3809ede931876f5b2ec92eef964286840ed3540dadf803dd570c3b7e13141a3b",
"sha256:38a9ef736c01fccdd6600705b09dc574584b89bea478200c5fbf112a6b0d5579",
"sha256:3d79d162e7be8f996986c064d1c7c817f6df3a77fe3d6859f6f9e7be4b8c213a",
"sha256:444dcda765c8a838eaae23112db52f1efaf750daddb2d9ca300bcae1039adc5c",
"sha256:48032821bbdf20f5799ff537c7ac3d1fba0ba032cfc06194faffa8cda8b560ff",
"sha256:4aa4e5faecf353ed117801a068ebab7b7e09ffb6e1d5e412dc852e0da018126c",
"sha256:52305740fe773d09cffb16f8ed0427942901f00adedac82ec8b67752f58a1b22",
"sha256:569511d3b58c8791ab4c2e1285575265991e6d8f8700c7be0e88f86cb0672094",
"sha256:57cb5a3cf367aeb1d316576250f65edec5bb3be939e9247ae594b4bcbc317dfb",
"sha256:5b02fb34468b6aaa40dfc198d813a641e3a63b98c2b05a16b9f80b7ec314185e",
"sha256:6381026f158fdb7c72a168278597a5e3a5222e83ea18f543112b2662a9b699c5",
"sha256:6af100e168aa82a50e186c82875a5893c5597a0c1ccdb0d8b40240b1f28b969a",
"sha256:6c89876f41da747c8d3677a2b540fb32ef5715f97b66eeb0c6b66f5e3ef6f59d",
"sha256:6e296a513ca3d94054c2c881cc913116e90fd030ad1c656b3869762b754f5f8a",
"sha256:70a87b411535ccad5ef2f1df5136506a10775d267e197e4cf531ced10537bd6b",
"sha256:7e94c425039cde14257288fd61dcfb01963e658efbc0ff54f5306b06054700f8",
"sha256:846ade7b71e3536c4e56b386c2a47adf5741d2d8b94ec9dc3e92e5e1ee1e2225",
"sha256:88416bd1e65dcea10bc7569faacb2c20ce071dd1f87539ca2ab364bf6231393c",
"sha256:88b49a3b9ff31e19998750c38e030fc7bb937398b1f78cfa599aaef92d693144",
"sha256:8c4e8c3ce11e1f92f6536ff07154f9d49677ebaaafc32db9db4620bc11ed480f",
"sha256:8e06879fc22a25ca47312fbe7c8264eb0b662f6db27cb2d3bbbc74b1df4b9b87",
"sha256:9025b4018f3a1314059769c7bf15441064b2207cb3f065e6ea1e7359cb46db9d",
"sha256:93335ca3812df2f366e80509ae119189886b0f3c2b81325d39efdb84a1e2ae93",
"sha256:9778bd8ab0a994ebf6f84c2b949e65736d5575320a17ae8984a77fab08db94cf",
"sha256:9e2d922824181480953426608b81967de705c3cef4d1af983af849d7bd619158",
"sha256:a123e330ef0853c6e822384873bef7507557d8e4a082961e1defa947aa59ba84",
"sha256:a904af0a6162c73e3edcb969eeeb53a63ceeb5d8cf642fade7d39e7963a22ddb",
"sha256:ad10d3ded218f1039f11a75f8091880239651b52e9bb592ca27de44eed242a48",
"sha256:b424c77b206d63d500bcb69fa55ed8d0e6a3774056bdc4839fc9298a7edca171",
"sha256:b5a6b3ada725cea8a5e634536b1b01c30bcdcd7f9c6fff4151548d5bf6b3a36c",
"sha256:ba8062ed2cf21c07a9e295d5b8a2a5ce678b913b45fdf68c32d95d6c1291e0b6",
"sha256:ba9527cdd4c926ed0760bc301f6728ef34d841f405abf9d4f959c478421e4efd",
"sha256:bbcb445fa71794da8f178f0f6d66789a28d7319071af7a496d4d507ed566270d",
"sha256:bcf3e58998965654fdaff38e58584d8937aa3096ab5354d493c77d1fdd66d7a1",
"sha256:c0ef13eaeee5b615fb07c9a7dadb38eac06a0608b41570d8ade51c56539e509d",
"sha256:cabc348d87e913db6ab4aa100f01b08f481097838bdddf7c7a84b7575b7309ca",
"sha256:cdb82a876c47801bb54a690c5ae105a46b392ac6099881cdfb9f6e95e4014c6a",
"sha256:cfad01eed2c2e0c01fd0ecd2ef42c492f7f93902e39a42fc9ee1692961443a29",
"sha256:d16a81a06776313e817c951135cf7340a3e91e8c1ff2fac444cfd75fffa04afe",
"sha256:d8213e09c917a951de9d09ecee036d5c7d36cb6cb7dbaece4c71a60d79fb9798",
"sha256:e07c3764494e3776c602c1e78e298937c3315ccc9043ead7e685b7f2b8d47b3c",
"sha256:e17c96c14e19278594aa4841ec148115f9c7615a47382ecb6b82bd8fea3ab0c8",
"sha256:e444a31f8db13eb18ada366ab3cf45fd4b31e4db1236a4448f68778c1d1a5a2f",
"sha256:e6a2a455bd412959b57a172ce6328d2dd1f01cb2135efda2e4576e8a23fa3b0f",
"sha256:eaa0a10b7f72326f1372a713e73c3f739b524b3af41feb43e4921cb529f5929a",
"sha256:eb7972a85c54febfb25b5c4b4f3af4dcc731994c7da0d8a0b4a6eb0640e1d178",
"sha256:ee55d3edf80167e48ea11a923c7386f4669df67d7994554387f84e7d8b0a2bf0",
"sha256:f3818cb119498c0678015754eba762e0d61e5b52d34c8b13d770f0719f7b1d79",
"sha256:f8b3d067f2e40fe93e1ccdd6b2e1d16c43140e76f02fb1319a05cf2b79d99430",
"sha256:fcabf5ff6eea076f859677f5f0b6b5c1a51e70a376b0579e0eadef8db48c6b50"
"sha256:00e046b6dd71aa03a41079792f8473dc494d564611a8f89bbbd7cb93295ebdcf",
"sha256:075202fa5b72c86ad32dc7d0b56024ebdbcf2048c0ba09f1cde31bfdd57bcfff",
"sha256:0e397ac966fdf721b2c528cf028494e86172b4feba51d65f81ffd65c63798f3f",
"sha256:17b950fccb810b3293638215058e432159d2b71005c74371d784862b7e4683f3",
"sha256:1f3fbcb7ef1f16e48246f704ab79d79da8a46891e2da03f8783a5b6fa41a9532",
"sha256:2174c595a0d73a3080ca3257b40096db99799265e1c27cc5a610743acd86d62f",
"sha256:2b7c57a4dfc4f16f7142221afe5ba4e093e09e728ca65c51f5620c9aaeb9a617",
"sha256:2d2d793e36e230fd32babe143b04cec8a8b3eb8a3122d2aceb4a371e6b09b8df",
"sha256:30b600cf0a7ac9234b2638fbc0fb6158ba5bdcdf46aeb631ead21248b9affbc4",
"sha256:397081c1a0bfb5124355710fe79478cdbeb39626492b15d399526ae53422b906",
"sha256:3a57fdd7ce31c7ff06cdfbf31dafa96cc533c21e443d57f5b1ecc6cdc668ec7f",
"sha256:3c6b973f22eb18a789b1460b4b91bf04ae3f0c4234a0a6aa6b0a92f6f7b951d4",
"sha256:3e53af139f8579a6d5f7b76549125f0d94d7e630761a2111bc431fd820e163b8",
"sha256:4096e9de5c6fdf43fb4f04c26fb114f61ef0bf2e5604b6ee3019d51b69e8c371",
"sha256:4275d846e41ecefa46e2015117a9f491e57a71ddd59bbead77e904dc02b1bed2",
"sha256:4c31f53cdae6ecfa91a77820e8b151dba54ab528ba65dfd235c80b086d68a465",
"sha256:4f11aa001c540f62c6166c7726f71f7573b52c68c31f014c25cc7901deea0b52",
"sha256:5049256f536511ee3f7e1b3f87d1d1209d327e818e6ae1365e8653d7e3abb6a6",
"sha256:58c98fee265677f63a4385256a6d7683ab1832f3ddd1e66fe948d5880c21a169",
"sha256:598e3276b64aff0e7b3451b72e94fa3c238d452e7ddcd893c3ab324717456bad",
"sha256:5b7b716f97b52c5a14bffdf688f971b2d5ef4029127f1ad7a513973cfd818df2",
"sha256:5dedb4db619ba5a2787a94d877bc8ffc0566f92a01c0ef214865e54ecc9ee5e0",
"sha256:619bc166c4f2de5caa5a633b8b7326fbe98e0ccbfacabd87268a2b15ff73a029",
"sha256:629ddd2ca402ae6dbedfceeba9c46d5f7b2a61d9749597d4307f943ef198fc1f",
"sha256:656f7526c69fac7f600bd1f400991cc282b417d17539a1b228617081106feb4a",
"sha256:6ec585f69cec0aa07d945b20805be741395e28ac1627333b1c5b0105962ffced",
"sha256:72b6be590cc35924b02c78ef34b467da4ba07e4e0f0454a2c5907f473fc50ce5",
"sha256:7502934a33b54030eaf1194c21c692a534196063db72176b0c4028e140f8f32c",
"sha256:7a68b554d356a91cce1236aa7682dc01df0edba8d043fd1ce607c49dd3c1edcf",
"sha256:7b2e5a267c855eea6b4283940daa6e88a285f5f2a67f2220203786dfa59b37e9",
"sha256:823b65d8706e32ad2df51ed89496147a42a2a6e01c13cfb6ffb8b1e92bc910bb",
"sha256:8590b4ae07a35970728874632fed7bd57b26b0102df2d2b233b6d9d82f6c62ad",
"sha256:8dd717634f5a044f860435c1d8c16a270ddf0ef8588d4887037c5028b859b0c3",
"sha256:8dec4936e9c3100156f8a2dc89c4b88d5c435175ff03413b443469c7c8c5f4d1",
"sha256:97cafb1f3cbcd3fd2b6fbfb99ae11cdb14deea0736fc2b0952ee177f2b813a46",
"sha256:a17a92de5231666cfbe003f0e4b9b3a7ae3afb1ec2845aadc2bacc93ff85febc",
"sha256:a549b9c31bec33820e885335b451286e2969a2d9e24879f83fe904a5ce59d70a",
"sha256:ac07bad82163452a6884fe8fa0963fb98c2346ba78d779ec06bd7a6262132aee",
"sha256:ae2ad8ae6ebee9d2d94b17fb62763125f3f374c25618198f40cbb8b525411900",
"sha256:b91c037585eba9095565a3556f611e3cbfaa42ca1e865f7b8015fe5c7336d5a5",
"sha256:bc1667f8b83f48511b94671e0e441401371dfd0f0a795c7daa4a3cd1dde55bea",
"sha256:bec0a414d016ac1a18862a519e54b2fd0fc8bbfd6890376898a6c0891dd82e9f",
"sha256:bf50cd79a75d181c9181df03572cdce0fbb75cc353bc350712073108cba98de5",
"sha256:bff1b4290a66b490a2f4719358c0cdcd9bafb6b8f061e45c7a2460866bf50c2e",
"sha256:c061bb86a71b42465156a3ee7bd58c8c2ceacdbeb95d05a99893e08b8467359a",
"sha256:c8b29db45f8fe46ad280a7294f5c3ec36dbac9491f2d1c17345be8e69cc5928f",
"sha256:ce409136744f6521e39fd8e2a24c53fa18ad67aa5bc7c2cf83645cce5b5c4e50",
"sha256:d050b3361367a06d752db6ead6e7edeb0009be66bc3bae0ee9d97fb326badc2a",
"sha256:d283d37a890ba4c1ae73ffadf8046435c76e7bc2247bbb63c00bd1a709c6544b",
"sha256:d9fad5155d72433c921b782e58892377c44bd6252b5af2f67f16b194987338a4",
"sha256:daa4ee5a243f0f20d528d939d06670a298dd39b1ad5f8a72a4275124a7819eff",
"sha256:db0b55e0f3cc0be60c1f19efdde9a637c32740486004f20d1cff53c3c0ece4d2",
"sha256:e61659ba32cf2cf1481e575d0462554625196a1f2fc06a1c777d3f48e8865d46",
"sha256:ea3d8a3d18833cf4304cd2fc9cbb1efe188ca9b5efef2bdac7adc20594a0e46b",
"sha256:ec6a563cff360b50eed26f13adc43e61bc0c04d94b8be985e6fb24b81f6dcfdf",
"sha256:f5dfb42c4604dddc8e4305050aa6deb084540643ed5804d7455b5df8fe16f5e5",
"sha256:fa173ec60341d6bb97a89f5ea19c85c5643c1e7dedebc22f5181eb73573142c5",
"sha256:fa9db3f79de01457b03d4f01b34cf91bc0048eb2c3846ff26f66687c2f6d16ab",
"sha256:fce659a462a1be54d2ffcacea5e3ba2d74daa74f30f5f143fe0c58636e355fdd",
"sha256:ffee1f21e5ef0d712f9033568f8344d5da8cc2869dbd08d87c84656e6a2d2f68"
],
"markers": "python_version >= '3.9'",
"version": "==3.0.2"
"markers": "python_version >= '3.7'",
"version": "==2.1.5"
},
"mitmproxy": {
"editable": true,

2
.gitattributes vendored
View File

@ -25,6 +25,8 @@ configure -whitespace
# except these exceptions...
src/backend/distributed/utils/citus_outfuncs.c -citus-style
src/backend/distributed/deparser/ruleutils_13.c -citus-style
src/backend/distributed/deparser/ruleutils_14.c -citus-style
src/backend/distributed/deparser/ruleutils_15.c -citus-style
src/backend/distributed/deparser/ruleutils_16.c -citus-style
src/backend/distributed/deparser/ruleutils_17.c -citus-style

View File

@ -6,7 +6,7 @@ inputs:
runs:
using: composite
steps:
- uses: actions/upload-artifact@v4.6.0
- uses: actions/upload-artifact@v3.1.1
name: Upload logs
with:
name: ${{ inputs.folder }}

View File

@ -17,7 +17,7 @@ runs:
echo "PG_MAJOR=${{ inputs.pg_major }}" >> $GITHUB_ENV
fi
shell: bash
- uses: actions/download-artifact@v4.1.8
- uses: actions/download-artifact@v3.0.1
with:
name: build-${{ env.PG_MAJOR }}
- name: Install Extension

View File

@ -21,7 +21,7 @@ runs:
mkdir -p /tmp/codeclimate
cc-test-reporter format-coverage -t lcov -o /tmp/codeclimate/${{ inputs.flags }}.json lcov.info
shell: bash
- uses: actions/upload-artifact@v4.6.0
- uses: actions/upload-artifact@v3.1.1
with:
path: "/tmp/codeclimate/*.json"
name: codeclimate-${{ inputs.flags }}
name: codeclimate

View File

@ -10,13 +10,8 @@ on:
required: false
default: false
type: boolean
push:
branches:
- "main"
- "release-*"
pull_request:
types: [opened, reopened,synchronize]
merge_group:
jobs:
# Since GHA does not interpolate env varibles in matrix context, we need to
# define them in a separate job and use them in other jobs.
@ -31,38 +26,39 @@ jobs:
pgupgrade_image_name: "ghcr.io/citusdata/pgupgradetester"
style_checker_image_name: "ghcr.io/citusdata/stylechecker"
style_checker_tools_version: "0.8.18"
sql_snapshot_pg_version: "17.5"
image_suffix: "-dev-d28f316"
pg15_version: '{ "major": "15", "full": "15.13" }'
pg16_version: '{ "major": "16", "full": "16.9" }'
pg17_version: '{ "major": "17", "full": "17.5" }'
upgrade_pg_versions: "15.13-16.9-17.5"
sql_snapshot_pg_version: "17.2"
image_suffix: "-v889e4c1"
pg14_version: '{ "major": "14", "full": "14.15" }'
pg15_version: '{ "major": "15", "full": "15.10" }'
pg16_version: '{ "major": "16", "full": "16.6" }'
pg17_version: '{ "major": "17", "full": "17.2" }'
upgrade_pg_versions: "14.15-15.10-16.6-17.2"
steps:
# Since GHA jobs need at least one step we use a noop step here.
- name: Set up parameters
run: echo 'noop'
check-sql-snapshots:
needs: params
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: ${{ needs.params.outputs.build_image_name }}:${{ needs.params.outputs.sql_snapshot_pg_version }}${{ needs.params.outputs.image_suffix }}
options: --user root
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- name: Check Snapshots
run: |
git config --global --add safe.directory ${GITHUB_WORKSPACE}
ci/check_sql_snapshots.sh
check-style:
needs: params
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: ${{ needs.params.outputs.style_checker_image_name }}:${{ needs.params.outputs.style_checker_tools_version }}${{ needs.params.outputs.image_suffix }}
steps:
- name: Check Snapshots
run: |
git config --global --add safe.directory ${GITHUB_WORKSPACE}
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
with:
fetch-depth: 0
- name: Check C Style
@ -110,22 +106,23 @@ jobs:
image_suffix:
- ${{ needs.params.outputs.image_suffix}}
pg_version:
- ${{ needs.params.outputs.pg14_version }}
- ${{ needs.params.outputs.pg15_version }}
- ${{ needs.params.outputs.pg16_version }}
- ${{ needs.params.outputs.pg17_version }}
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: "${{ matrix.image_name }}:${{ fromJson(matrix.pg_version).full }}${{ matrix.image_suffix }}"
options: --user root
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- name: Expose $PG_MAJOR to Github Env
run: echo "PG_MAJOR=${PG_MAJOR}" >> $GITHUB_ENV
shell: bash
- name: Build
run: "./ci/build-citus.sh"
shell: bash
- uses: actions/upload-artifact@v4.6.0
- uses: actions/upload-artifact@v3.1.1
with:
name: build-${{ env.PG_MAJOR }}
path: |-
@ -141,6 +138,7 @@ jobs:
image_name:
- ${{ needs.params.outputs.test_image_name }}
pg_version:
- ${{ needs.params.outputs.pg14_version }}
- ${{ needs.params.outputs.pg15_version }}
- ${{ needs.params.outputs.pg16_version }}
- ${{ needs.params.outputs.pg17_version }}
@ -161,6 +159,10 @@ jobs:
- check-enterprise-isolation-logicalrep-2
- check-enterprise-isolation-logicalrep-3
include:
- make: check-failure
pg_version: ${{ needs.params.outputs.pg14_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-failure
pg_version: ${{ needs.params.outputs.pg15_version }}
suite: regress
@ -173,6 +175,10 @@ jobs:
pg_version: ${{ needs.params.outputs.pg17_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-enterprise-failure
pg_version: ${{ needs.params.outputs.pg14_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-enterprise-failure
pg_version: ${{ needs.params.outputs.pg15_version }}
suite: regress
@ -185,6 +191,10 @@ jobs:
pg_version: ${{ needs.params.outputs.pg17_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-pytest
pg_version: ${{ needs.params.outputs.pg14_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-pytest
pg_version: ${{ needs.params.outputs.pg15_version }}
suite: regress
@ -209,6 +219,10 @@ jobs:
suite: cdc
image_name: ${{ needs.params.outputs.test_image_name }}
pg_version: ${{ needs.params.outputs.pg17_version }}
- make: check-query-generator
pg_version: ${{ needs.params.outputs.pg14_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
- make: check-query-generator
pg_version: ${{ needs.params.outputs.pg15_version }}
suite: regress
@ -221,7 +235,7 @@ jobs:
pg_version: ${{ needs.params.outputs.pg17_version }}
suite: regress
image_name: ${{ needs.params.outputs.fail_test_image_name }}
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: "${{ matrix.image_name }}:${{ fromJson(matrix.pg_version).full }}${{ needs.params.outputs.image_suffix }}"
options: --user root --dns=8.8.8.8
@ -232,7 +246,7 @@ jobs:
- params
- build
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/setup_extension"
- name: Run Test
run: gosu circleci make -C src/test/${{ matrix.suite }} ${{ matrix.make }}
@ -261,12 +275,13 @@ jobs:
image_name:
- ${{ needs.params.outputs.fail_test_image_name }}
pg_version:
- ${{ needs.params.outputs.pg14_version }}
- ${{ needs.params.outputs.pg15_version }}
- ${{ needs.params.outputs.pg16_version }}
- ${{ needs.params.outputs.pg17_version }}
parallel: [0,1,2,3,4,5] # workaround for running 6 parallel jobs
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/setup_extension"
- name: Test arbitrary configs
run: |-
@ -288,16 +303,14 @@ jobs:
check-arbitrary-configs parallel=4 CONFIGS=$TESTS
- uses: "./.github/actions/save_logs_and_results"
if: always()
with:
folder: ${{ env.PG_MAJOR }}_arbitrary_configs_${{ matrix.parallel }}
- uses: "./.github/actions/upload_coverage"
if: always()
with:
flags: ${{ env.PG_MAJOR }}_arbitrary_configs_${{ matrix.parallel }}
flags: ${{ env.pg_major }}_upgrade
codecov_token: ${{ secrets.CODECOV_TOKEN }}
test-pg-upgrade:
name: PG${{ matrix.old_pg_major }}-PG${{ matrix.new_pg_major }} - check-pg-upgrade
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: "${{ needs.params.outputs.pgupgrade_image_name }}:${{ needs.params.outputs.upgrade_pg_versions }}${{ needs.params.outputs.image_suffix }}"
options: --user root
@ -308,17 +321,23 @@ jobs:
fail-fast: false
matrix:
include:
- old_pg_major: 14
new_pg_major: 15
- old_pg_major: 15
new_pg_major: 16
- old_pg_major: 14
new_pg_major: 16
- old_pg_major: 16
new_pg_major: 17
- old_pg_major: 15
new_pg_major: 17
- old_pg_major: 14
new_pg_major: 17
env:
old_pg_major: ${{ matrix.old_pg_major }}
new_pg_major: ${{ matrix.new_pg_major }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/setup_extension"
with:
pg_major: "${{ env.old_pg_major }}"
@ -341,24 +360,22 @@ jobs:
if: failure()
- uses: "./.github/actions/save_logs_and_results"
if: always()
with:
folder: ${{ env.old_pg_major }}_${{ env.new_pg_major }}_upgrade
- uses: "./.github/actions/upload_coverage"
if: always()
with:
flags: ${{ env.old_pg_major }}_${{ env.new_pg_major }}_upgrade
codecov_token: ${{ secrets.CODECOV_TOKEN }}
test-citus-upgrade:
name: PG${{ fromJson(needs.params.outputs.pg15_version).major }} - check-citus-upgrade
runs-on: ubuntu-latest
name: PG${{ fromJson(needs.params.outputs.pg14_version).major }} - check-citus-upgrade
runs-on: ubuntu-20.04
container:
image: "${{ needs.params.outputs.citusupgrade_image_name }}:${{ fromJson(needs.params.outputs.pg15_version).full }}${{ needs.params.outputs.image_suffix }}"
image: "${{ needs.params.outputs.citusupgrade_image_name }}:${{ fromJson(needs.params.outputs.pg14_version).full }}${{ needs.params.outputs.image_suffix }}"
options: --user root
needs:
- params
- build
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/setup_extension"
with:
skip_installation: true
@ -388,18 +405,16 @@ jobs:
done;
- uses: "./.github/actions/save_logs_and_results"
if: always()
with:
folder: ${{ env.PG_MAJOR }}_citus_upgrade
- uses: "./.github/actions/upload_coverage"
if: always()
with:
flags: ${{ env.PG_MAJOR }}_citus_upgrade
flags: ${{ env.pg_major }}_upgrade
codecov_token: ${{ secrets.CODECOV_TOKEN }}
upload-coverage:
if: always()
env:
CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: ${{ needs.params.outputs.test_image_name }}:${{ fromJson(needs.params.outputs.pg17_version).full }}${{ needs.params.outputs.image_suffix }}
needs:
@ -409,11 +424,10 @@ jobs:
- test-citus-upgrade
- test-pg-upgrade
steps:
- uses: actions/download-artifact@v4.1.8
- uses: actions/download-artifact@v3.0.1
with:
pattern: codeclimate*
path: codeclimate
merge-multiple: true
name: "codeclimate"
path: "codeclimate"
- name: Upload coverage results to Code Climate
run: |-
cc-test-reporter sum-coverage codeclimate/*.json -o total.json
@ -421,11 +435,11 @@ jobs:
ch_benchmark:
name: CH Benchmark
if: startsWith(github.ref, 'refs/heads/ch_benchmark/')
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
needs:
- build
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
@ -439,11 +453,11 @@ jobs:
tpcc_benchmark:
name: TPCC Benchmark
if: startsWith(github.ref, 'refs/heads/tpcc_benchmark/')
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
needs:
- build
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
@ -455,14 +469,14 @@ jobs:
chmod +x run_hammerdb.sh
run_hammerdb.sh citusbot_tpcc_benchmark_rg
prepare_parallelization_matrix_32:
name: Prepare parallelization matrix
name: Parallel 32
if: ${{ needs.test-flakyness-pre.outputs.tests != ''}}
needs: test-flakyness-pre
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
outputs:
json: ${{ steps.parallelization.outputs.json }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/parallelization"
id: parallelization
with:
@ -470,50 +484,34 @@ jobs:
test-flakyness-pre:
name: Detect regression tests need to be ran
if: ${{ !inputs.skip_test_flakyness }}}
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
needs: build
outputs:
tests: ${{ steps.detect-regression-tests.outputs.tests }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
with:
fetch-depth: 0
- name: Detect regression tests need to be ran
id: detect-regression-tests
run: |-
detected_changes=$(git diff origin/main... --name-only --diff-filter=AM | (grep 'src/test/regress/sql/.*\.sql\|src/test/regress/spec/.*\.spec\|src/test/regress/citus_tests/test/test_.*\.py' || true))
detected_changes=$(git diff origin/release-12.1... --name-only --diff-filter=AM | (grep 'src/test/regress/sql/.*\.sql\|src/test/regress/spec/.*\.spec\|src/test/regress/citus_tests/test/test_.*\.py' || true))
tests=${detected_changes}
# split the tests to be skipped --today we only skip upgrade tests
skipped_tests=""
not_skipped_tests=""
for test in $tests; do
if [[ $test =~ ^src/test/regress/sql/upgrade_ ]]; then
skipped_tests="$skipped_tests $test"
else
not_skipped_tests="$not_skipped_tests $test"
fi
done
if [ ! -z "$skipped_tests" ]; then
echo "Skipped tests " $skipped_tests
fi
if [ -z "$not_skipped_tests" ]; then
echo "Not detected any tests that flaky test detection should run"
if [ -z "$tests" ]; then
echo "No test found."
else
echo "Detected tests " $not_skipped_tests
echo "Detected tests " $tests
fi
echo 'tests<<EOF' >> $GITHUB_OUTPUT
echo "$not_skipped_tests" >> "$GITHUB_OUTPUT"
echo "$tests" >> "$GITHUB_OUTPUT"
echo 'EOF' >> $GITHUB_OUTPUT
test-flakyness:
if: ${{ needs.test-flakyness-pre.outputs.tests != ''}}
if: false
name: Test flakyness
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
container:
image: ${{ needs.params.outputs.fail_test_image_name }}:${{ fromJson(needs.params.outputs.pg17_version).full }}${{ needs.params.outputs.image_suffix }}
image: ${{ needs.params.outputs.fail_test_image_name }}:${{ needs.params.outputs.pg17_version }}${{ needs.params.outputs.image_suffix }}
options: --user root
env:
runs: 8
@ -526,8 +524,8 @@ jobs:
fail-fast: false
matrix: ${{ fromJson(needs.prepare_parallelization_matrix_32.outputs.json) }}
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4.1.8
- uses: actions/checkout@v3.5.0
- uses: actions/download-artifact@v3.0.1
- uses: "./.github/actions/setup_extension"
- name: Run minimal tests
run: |-
@ -536,10 +534,8 @@ jobs:
for test in "${tests_array[@]}"
do
test_name=$(echo "$test" | sed -r "s/.+\/(.+)\..+/\1/")
gosu circleci src/test/regress/citus_tests/run_test.py $test_name --repeat ${{ env.runs }} --use-whole-schedule-line
gosu circleci src/test/regress/citus_tests/run_test.py $test_name --repeat ${{ env.runs }} --use-base-schedule --use-whole-schedule-line
done
shell: bash
- uses: "./.github/actions/save_logs_and_results"
if: always()
with:
folder: test_flakyness_parallel_${{ matrix.id }}

View File

@ -21,10 +21,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
@ -76,4 +76,4 @@ jobs:
sudo make install-all
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
uses: github/codeql-action/analyze@v2

View File

@ -1,54 +0,0 @@
name: "Build devcontainer"
# Since building of containers can be quite time consuming, and take up some storage,
# there is no need to finish a build for a tag if new changes are concurrently being made.
# This cancels any previous builds for the same tag, and only the latest one will be kept.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
push:
paths:
- ".devcontainer/**"
workflow_dispatch:
jobs:
docker:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
attestations: write
id-token: write
steps:
-
name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/citusdata/citus-devcontainer
tags: |
type=ref,event=branch
type=sha
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: 'Login to GitHub Container Registry'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{github.actor}}
password: ${{secrets.GITHUB_TOKEN}}
-
name: Build and push
uses: docker/build-push-action@v5
with:
context: "{{defaultContext}}:.devcontainer"
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max

View File

@ -28,13 +28,13 @@ jobs:
image: ${{ vars.build_image_name }}:${{ vars.pg15_version }}${{ vars.image_suffix }}
options: --user root
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- name: Configure, Build, and Install
run: |
echo "PG_MAJOR=${PG_MAJOR}" >> $GITHUB_ENV
./ci/build-citus.sh
shell: bash
- uses: actions/upload-artifact@v4.6.0
- uses: actions/upload-artifact@v3.1.1
with:
name: build-${{ env.PG_MAJOR }}
path: |-
@ -46,7 +46,7 @@ jobs:
outputs:
json: ${{ steps.parallelization.outputs.json }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/parallelization"
id: parallelization
with:
@ -67,13 +67,13 @@ jobs:
fail-fast: false
matrix: ${{ fromJson(needs.prepare_parallelization_matrix.outputs.json) }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3.5.0
- uses: "./.github/actions/setup_extension"
- name: Run minimal tests
run: |-
gosu circleci src/test/regress/citus_tests/run_test.py ${{ env.test }} --repeat ${{ env.runs }} --use-whole-schedule-line
gosu circleci src/test/regress/citus_tests/run_test.py ${{ env.test }} --repeat ${{ env.runs }} --use-base-schedule --use-whole-schedule-line
shell: bash
- uses: "./.github/actions/save_logs_and_results"
if: always()
with:
folder: check_flakyness_parallel_${{ matrix.id }}
folder: ${{ matrix.id }}

View File

@ -3,7 +3,6 @@ name: Build tests in packaging images
on:
pull_request:
types: [opened, reopened,synchronize]
merge_group:
workflow_dispatch:
@ -129,7 +128,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set pg_config path and python parameters for deb based distros
run: |

3
.gitignore vendored
View File

@ -55,6 +55,3 @@ lib*.pc
# style related temporary outputs
*.uncrustify
.venv
# added output when modifying check_gucs_are_alphabetically_sorted.sh
guc.out

View File

@ -1,188 +1,3 @@
### citus v13.1.0 (May 30th, 2025) ###
* Adds `citus_stat_counters` view that can be used to query
stat counters that Citus collects while the feature is enabled, which is
controlled by citus.enable_stat_counters. `citus_stat_counters()` can be
used to query the stat counters for the provided database oid and
`citus_stat_counters_reset()` can be used to reset them for the provided
database oid or for the current database if nothing or 0 is provided (#7917)
* Adds `citus_nodes` view that displays the node name, port role, and "active"
for nodes in the cluster (#7968)
* Adds `citus_is_primary_node()` UDF to determine if the current node is a
primary node in the cluster (#7720)
* Adds support for propagating `GRANT/REVOKE` rights on table columns (#7918)
* Adds support for propagating `REASSIGN OWNED BY` commands (#7319)
* Adds support for propagating `CREATE`/`DROP` database from all nodes (#7240,
#7253, #7359)
* Propagates `SECURITY LABEL ON ROLE` statement from any node (#7508)
* Adds support for issuing role management commands from worker nodes (#7278)
* Adds support for propagating `ALTER USER RENAME` commands (#7204)
* Adds support for propagating `ALTER DATABASE <db_name> SET ..` commands
(#7181)
* Adds support for propagating `SECURITY LABEL` on tables and columns (#7956)
* Adds support for propagating `COMMENT ON <database>/<role>` commands (#7388)
* Moves some of the internal citus functions from `pg_catalog` to
`citus_internal` schema (#7473, #7470, #7466, 7456, 7450)
* Adjusts `max_prepared_transactions` only when it's set to default on PG >= 16
(#7712)
* Adds skip_qualify_public param to shard_name() UDF to allow qualifying for
"public" schema when needed (#8014)
* Allows `citus_*_size` on indexes on a distributed tables (#7271)
* Allows `GRANT ADMIN` to now also be `INHERIT` or `SET` in support of PG16
* Makes sure `worker_copy_table_to_node` errors out with Citus tables (#7662)
* Adds information to explain output when using
`citus.explain_distributed_queries=false` (#7412)
* Logs username in the failed connection message (#7432)
* Makes sure to avoid incorrectly pushing-down the outer joins between
distributed tables and recurring relations (like reference tables, local
tables and `VALUES(..)` etc.) prior to PG 17 (#7937)
* Prevents incorrectly pushing `nextval()` call down to workers to avoid using
incorrect sequence value for some types of `INSERT .. SELECT`s (#7976)
* Makes sure to prevent `INSERT INTO ... SELECT` queries involving subfield or
sublink, to avoid crashes (#7912)
* Makes sure to take improvement_threshold into the account
in `citus_add_rebalance_strategy()` (#7247)
* Makes sure to disallow creating a replicated distributed
table concurrently (#7219)
* Fixes a bug that causes omitting `CASCADE` clause for the commands sent to
workers for `REVOKE` commands on tables (#7958)
* Fixes an issue detected using address sanitizer (#7948, #7949)
* Fixes a bug in deparsing of shard query in case of "output-table column" name
conflict (#7932)
* Fixes a crash in columnar custom scan that happens when a columnar table is
used in a join (#7703)
* Fixes `MERGE` command when insert value does not have source distributed
column (#7627)
* Fixes performance issue when using `\d tablename` on a server with many
tables (#7577)
* Fixes performance issue in `GetForeignKeyOids` on systems with many
constraints (#7580)
* Fixes performance issue when distributing a table that depends on an
extension (#7574)
* Fixes performance issue when creating distributed tables if many already
exist (#7575)
* Fixes a crash caused by some form of `ALTER TABLE ADD COLUMN` statements. When
adding multiple columns, if one of the `ADD COLUMN` statements contains a
`FOREIGN` constraint ommitting the referenced
columns in the statement, a `SEGFAULT` occurs (#7522)
* Fixes assertion failure in maintenance daemon during Citus upgrades (#7537)
* Fixes segmentation fault when using `CASE WHEN` in `DO` block functions
(#7554)
* Fixes undefined behavior in `master_disable_node` due to argument mismatch
(#7492)
* Fixes incorrect propagating of `GRANTED BY` and `CASCADE/RESTRICT` clauses
for `REVOKE` statements (#7451)
* Fixes the incorrect column count after `ALTER TABLE` (#7379)
* Fixes timeout when underlying socket is changed for an inter-node connection
(#7377)
* Fixes memory leaks (#7441, #7440)
* Fixes leaking of memory and memory contexts when tracking foreign keys between
Citus tables (#7236)
* Fixes a potential segfault for background rebalancer (#7694)
* Fixes potential `NULL` dereference in casual clocks (#7704)
### citus v13.0.4 (May 29th, 2025) ###
* Fixes an issue detected using address sanitizer (#7966)
* Error out for queries with outer joins and pseudoconstant quals in versions
prior to PG 17 (#7937)
### citus v12.1.8 (May 29, 2025) ###
* Fixes a crash in left outer joins that can happen when there is an an
aggregate on a column from the inner side of the join (#7904)
* Fixes an issue detected using address sanitizer (#7965)
* Fixes a crash when executing a prepared CALL, which is not pure SQL but
available with some drivers like npgsql and jpgdbc (#7288)
### citus v13.0.3 (March 20th, 2025) ###
* Fixes a version bump issue in 13.0.2
### citus v13.0.2 (March 12th, 2025) ###
* Fixes a crash in columnar custom scan that happens when a columnar table is
used in a join. (#7647)
* Fixes a bug that breaks `UPDATE SET (...) = (SELECT some_func(),... )`
type of queries on Citus tables (#7914)
* Fixes a planning error caused by a redundant WHERE clause (#7907)
* Fixes a crash in left outer joins that can happen when there is an aggregate
on a column from the inner side of the join. (#7901)
* Fixes deadlock with transaction recovery that is possible during Citus
upgrades. (#7910)
* Fixes a bug that prevents inserting into Citus tables that uses
a GENERATED ALWAYS AS IDENTITY column. (#7920)
* Ensures that a MERGE command on a distributed table with a WHEN NOT MATCHED BY
SOURCE clause runs against all shards of the distributed table. (#7900)
* Fixes a bug that breaks router updates on distributed tables
when a reference table is used in the subquery (#7897)
### citus v12.1.7 (Feb 6, 2025) ###
* Fixes a crash that happens because of unsafe catalog access when re-assigning
the global pid after `application_name` changes (#7791)
* Prevents crashes when another extension skips executing the
`ClientAuthentication_hook` of Citus. (#7836)
### citus v13.0.1 (February 4th, 2025) ###
* Drops support for PostgreSQL 14 (#7753)
### citus v13.0.0 (January 22, 2025) ###
* Adds support for PostgreSQL 17 (#7699, #7661)
@ -221,12 +36,6 @@ available with some drivers like npgsql and jpgdbc (#7288)
* Improves `citus_move_shard_placement()` to fail early if there is a new node
without reference tables yet (#7467)
### citus v12.1.6 (Nov 14, 2024) ###
* Propagates `SECURITY LABEL .. ON ROLE` statements (#7304)
* Fixes crash caused by running queries with window partition (#7718)
### citus v12.1.5 (July 17, 2024) ###
* Adds support for MERGE commands with single shard distributed target tables
@ -244,8 +53,9 @@ available with some drivers like npgsql and jpgdbc (#7288)
* Allows overwriting host name for all inter-node connections by
supporting "host" parameter in citus.node_conninfo (#7541)
* Avoids distributed deadlocks by changing the order in which the locks are
acquired for the target and reference tables (#7542)
* Changes the order in which the locks are acquired for the target and
reference tables, when a modify request is initiated from a worker
node that is not the "FirstWorkerNode" (#7542)
* Fixes a performance issue when distributing a table that depends on an
extension (#7574)
@ -278,120 +88,10 @@ available with some drivers like npgsql and jpgdbc (#7288)
* Logs username in the failed connection message (#7432)
### citus v11.0.10 (February 15, 2024) ###
* Removes pg_send_cancellation and all references (#7135)
### citus v12.1.2 (February 12, 2024) ###
* Fixes the incorrect column count after ALTER TABLE (#7379)
### citus v12.0.1 (July 11, 2023) ###
* Fixes incorrect default value assumption for VACUUM(PROCESS_TOAST) #7122)
* Fixes a bug that causes an unexpected error when adding a column
with a NULL constraint (#7093)
* Fixes a bug that could cause COPY logic to skip data in case of OOM (#7152)
* Fixes a bug with deleting colocation groups (#6929)
* Fixes memory and memory contexts leaks in Foreign Constraint Graphs (#7236)
* Fixes shard size bug with too many shards (#7018)
* Fixes the incorrect column count after ALTER TABLE (#7379)
* Improves citus_tables view performance (#7050)
* Makes sure to disallow creating a replicated distributed table
concurrently (#7219)
* Removes pg_send_cancellation and all references (#7135)
### citus v11.3.1 (February 12, 2024) ###
* Disallows MERGE when the query prunes down to zero shards (#6946)
* Fixes a bug related to non-existent objects in DDL commands (#6984)
* Fixes a bug that could cause COPY logic to skip data in case of OOM (#7152)
* Fixes a bug with deleting colocation groups (#6929)
* Fixes incorrect results on fetching scrollable with hold cursors (#7014)
* Fixes memory and memory context leaks in Foreign Constraint Graphs (#7236)
* Fixes replicate reference tables task fail when user is superuser (#6930)
* Fixes the incorrect column count after ALTER TABLE (#7379)
* Improves citus_shard_sizes performance (#7050)
* Makes sure to disallow creating a replicated distributed table
concurrently (#7219)
* Removes pg_send_cancellation and all references (#7135)
### citus v11.2.2 (February 12, 2024) ###
* Fixes a bug in background shard rebalancer where the replicate
reference tables task fails if the current user is not a superuser (#6930)
* Fixes a bug related to non-existent objects in DDL commands (#6984)
* Fixes a bug that could cause COPY logic to skip data in case of OOM (#7152)
* Fixes a bug with deleting colocation groups (#6929)
* Fixes incorrect results on fetching scrollable with hold cursors (#7014)
* Fixes memory and memory context leaks in Foreign Constraint Graphs (#7236)
* Fixes the incorrect column count after ALTER TABLE (#7379)
* Improves failure handling of distributed execution (#7090)
* Makes sure to disallow creating a replicated distributed table
concurrently (#7219)
* Removes pg_send_cancellation (#7135)
### citus v11.1.7 (February 12, 2024) ###
* Fixes memory and memory context leaks in Foreign Constraint Graphs (#7236)
* Fixes a bug related to non-existent objects in DDL commands (#6984)
* Fixes a bug that could cause COPY logic to skip data in case of OOM (#7152)
* Fixes a bug with deleting colocation groups (#6929)
* Fixes incorrect results on fetching scrollable with hold cursors (#7014)
* Fixes the incorrect column count after ALTER TABLE (#7379)
* Improves failure handling of distributed execution (#7090)
* Makes sure to disallow creating a replicated distributed table
concurrently (#7219)
* Removes pg_send_cancellation and all references (#7135)
### citus v11.0.9 (February 12, 2024) ###
* Fixes a bug that could cause COPY logic to skip data in case of OOM (#7152)
* Fixes a bug with deleting colocation groups (#6929)
* Fixes memory and memory context leaks in Foreign Constraint Graphs (#7236)
* Fixes the incorrect column count after ALTER TABLE (#7462)
* Improve failure handling of distributed execution (#7090)
### citus v12.1.1 (November 9, 2023) ###
* Fixes leaking of memory and memory contexts in Citus foreign key cache

View File

@ -11,52 +11,6 @@ sign a Contributor License Agreement (CLA). For an explanation of
why we ask this as well as instructions for how to proceed, see the
[Microsoft CLA](https://cla.opensource.microsoft.com/).
### Devcontainer / Github Codespaces
The easiest way to start contributing is via our devcontainer. This container works both locally in visual studio code with docker-desktop/docker-for-mac as well as [Github Codespaces](https://github.com/features/codespaces). To open the project in vscode you will need the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). For codespaces you will need to [create a new codespace](https://codespace.new/citusdata/citus).
With the extension installed you can run the following from the command pallet to get started
```
> Dev Containers: Clone Repository in Container Volume...
```
In the subsequent popup paste the url to the repo and hit enter.
```
https://github.com/citusdata/citus
```
This will create an isolated Workspace in vscode, complete with all tools required to build, test and run the Citus extension. We keep this container up to date with the supported postgres versions as well as the exact versions of tooling we use.
To quickly start we suggest splitting your terminal once to have two shells. The left one in the `/workspaces/citus`, the second one changed to `/data`. The left terminal will be used to interact with the project, the right one with a testing cluster.
To get citus installed from source we run `make install -s` in the first terminal. Once installed you can start a Citus cluster in the second terminal via `citus_dev make citus`. The cluster will run in the background, and can be interacted with via `citus_dev`. To get an overview of the available commands.
With the Citus cluster running you can connect to the coordinator in the first terminal via `psql -p9700`. Because the coordinator is the most common entrypoint the `PGPORT` environment is set accordingly, so a simple `psql` will connect directly to the coordinator.
### Debugging in the VS code
1. Start Debugging: Press F5 in VS Code to start debugging. When prompted, you'll need to attach the debugger to the appropriate PostgreSQL process.
2. Identify the Process: If you're running a psql command, take note of the PID that appears in your psql prompt. For example:
```
[local] citus@citus:9700 (PID: 5436)=#
```
This PID (5436 in this case) indicates the process that you should attach the debugger to.
If you are uncertain about which process to attach, you can list all running PostgreSQL processes using the following command:
```
ps aux | grep postgres
```
Look for the process associated with the PID you noted. For example:
```
citus 5436 0.0 0.0 0 0 ? S 14:00 0:00 postgres: citus citus
```
4. Attach the Debugger: Once you've identified the correct PID, select that process when prompted in VS Code to attach the debugger. You should now be able to debug the PostgreSQL session tied to the psql command.
5. Set Breakpoints and Debug: With the debugger attached, you can set breakpoints within the code. This allows you to step through the code execution, inspect variables, and fully debug the PostgreSQL instance running in your container.
### Getting and building
[PostgreSQL documentation](https://www.postgresql.org/support/versioning/) has a
@ -87,8 +41,6 @@ that are missing in earlier minor versions.
cd citus
./configure
# If you have already installed the project, you need to clean it first
make clean
make
make install
# Optionally, you might instead want to use `make install-all`
@ -127,8 +79,6 @@ that are missing in earlier minor versions.
git clone https://github.com/citusdata/citus.git
cd citus
./configure
# If you have already installed the project previously, you need to clean it first
make clean
make
sudo make install
# Optionally, you might instead want to use `sudo make install-all`
@ -179,8 +129,6 @@ that are missing in earlier minor versions.
git clone https://github.com/citusdata/citus.git
cd citus
PG_CONFIG=/usr/pgsql-14/bin/pg_config ./configure
# If you have already installed the project previously, you need to clean it first
make clean
make
sudo make install
# Optionally, you might instead want to use `sudo make install-all`
@ -197,7 +145,43 @@ that are missing in earlier minor versions.
### Following our coding conventions
Our coding conventions are documented in [STYLEGUIDE.md](STYLEGUIDE.md).
CircleCI will automatically reject any PRs which do not follow our coding
conventions. The easiest way to ensure your PR adheres to those conventions is
to use the [citus_indent](https://github.com/citusdata/tools/tree/develop/uncrustify)
tool. This tool uses `uncrustify` under the hood.
```bash
# Uncrustify changes the way it formats code every release a bit. To make sure
# everyone formats consistently we use version 0.68.1:
curl -L https://github.com/uncrustify/uncrustify/archive/uncrustify-0.68.1.tar.gz | tar xz
cd uncrustify-uncrustify-0.68.1/
mkdir build
cd build
cmake ..
make -j5
sudo make install
cd ../..
git clone https://github.com/citusdata/tools.git
cd tools
make uncrustify/.install
```
Once you've done that, you can run the `make reindent` command from the top
directory to recursively check and correct the style of any source files in the
current directory. Under the hood, `make reindent` will run `citus_indent` and
some other style corrections for you.
You can also run the following in the directory of this repository to
automatically format all the files that you have changed before committing:
```bash
cat > .git/hooks/pre-commit << __EOF__
#!/bin/bash
citus_indent --check --diff || { citus_indent --diff; exit 1; }
__EOF__
chmod +x .git/hooks/pre-commit
```
### Making SQL changes
@ -250,34 +234,3 @@ Any other SQL you can put directly in the main sql file, e.g.
### Running tests
See [`src/test/regress/README.md`](https://github.com/citusdata/citus/blob/master/src/test/regress/README.md)
### Documentation
User-facing documentation is published on [docs.citusdata.com](https://docs.citusdata.com/). When adding a new feature, function, or setting, you can open a pull request or issue against the [Citus docs repo](https://github.com/citusdata/citus_docs/).
Detailed descriptions of the implementation for Citus developers are provided in the [Citus Technical Documentation](src/backend/distributed/README.md). It is currently a single file for ease of searching. Please update the documentation if you make any changes that affect the design or add major new features.
# Making a pull request ready for reviews
Asking for help and asking for reviews are two different things. When you're asking for help, you're asking for someone to help you with something that you're not expected to know.
But when you're asking for a review, you're asking for someone to review your work and provide feedback. So, when you're asking for a review, you're expected to make sure that:
* Your changes don't perform **unnecessary line addition / deletions / style changes on unrelated files / lines**.
* All CI jobs are **passing**, including **style checks** and **flaky test detection jobs**. Note that if you're an external contributor, you don't have to wait CI jobs to run (and finish) because they don't get automatically triggered for external contributors.
* Your PR has necessary amount of **tests** and that they're passing.
* You separated as much as possible work into **separate PRs**, e.g., a prerequisite bugfix, a refactoring etc..
* Your PR doesn't introduce a typo or something that you can easily fix yourself.
* After all CI jobs pass, code-coverage measurement job (CodeCov as of today) then kicks in. That's why it's important to make the **tests passing** first. At that point, you're expected to check **CodeCov annotations** that can be seen in the **Files Changed** tab and expected to make sure that it doesn't complain about any lines that are not covered. For example, it's ok if CodeCov complains about an `ereport()` call that you put for an "unexpected-but-better-than-crashing" case, but it's not ok if it complains about an uncovered `if` branch that you added.
* And finally, perform a **self-review** to make sure that:
* Code and code-comments reflects the idea **without requiring an extra explanation** via a chat message / email / PR comment.
This is important because we don't expect developers to reach out to author / read about the whole discussion in the PR to understand the idea behind a commit merged into `main` branch.
* PR description is clear enough.
* If-and-only-if you're **introducing a user facing change / bugfix**, your PR has a line that starts with `DESCRIPTION: <Present simple tense word that starts with a capital letter, e.g., Adds support for / Fixes / Disallows>`.
* **Commit messages** are clear enough if the commits are doing logically different things.

View File

@ -1,43 +0,0 @@
# Devcontainer
## Coredumps
When postgres/citus crashes, there is the option to create a coredump. This is useful for debugging the issue. Coredumps are enabled in the devcontainer by default. However, not all environments are configured correctly out of the box. The most important configuration that is not standardized is the `core_pattern`. The configuration can be verified from the container, however, you cannot change this setting from inside the container as the filesystem containing this setting is in read only mode while inside the container.
To verify if corefiles are written run the following command in a terminal. This shows the filename pattern with which the corefile will be written.
```bash
cat /proc/sys/kernel/core_pattern
```
This should be configured with a relative path or simply a simple filename, such as `core`. When your environment shows an absolute path you will need to change this setting. How to change this setting depends highly on the underlying system as the setting needs to be changed on the kernel of the host running the container.
You can put any pattern in `/proc/sys/kernel/core_pattern` as you see fit. eg. You can add the PID to the core pattern in one of two ways;
- You either include `%p` in the core_pattern. This gets substituted with the PID of the crashing process.
- Alternatively you could set `/proc/sys/kernel/core_uses_pid` to `1` in the same way as you set `core_pattern`. This will append the PID to the corefile if `%p` is not explicitly contained in the core_pattern.
When a coredump is written you can use the debug/launch configuration `Open core file` which is preconfigured in the devcontainer. This will open a fileprompt that lists all coredumps that are found in your workspace. When you want to debug coredumps from `citus_dev` that are run in your `/data` directory, you can add the data directory to your workspace. In the command pallet of vscode you can run `>Workspace: Add Folder to Workspace...` and select the `/data` directory. This will allow you to open the coredumps from the `/data` directory in the `Open core file` debug configuration.
### Windows (docker desktop)
When running in docker desktop on windows you will most likely need to change this setting. The linux guest in WSL2 that runs your container is the `docker-desktop` environment. The easiest way to get onto the host, where you can change this setting, is to open a powershell window and verify you have the docker-desktop environment listed.
```powershell
wsl --list
```
Among others this should list both `docker-desktop` and `docker-desktop-data`. You can then open a shell in the `docker-desktop` environment.
```powershell
wsl -d docker-desktop
```
Inside this shell you can verify that you have the right environment by running
```bash
cat /proc/sys/kernel/core_pattern
```
This should show the same configuration as the one you see inside the devcontainer. You can then change the setting by running the following command.
This will change the setting for the current session. If you want to make the change permanent you will need to add this to a startup script.
```bash
echo "core" > /proc/sys/kernel/core_pattern
```

View File

@ -61,7 +61,6 @@ check-style:
# depend on install-all so that downgrade scripts are installed as well
check: all install-all
# explicetely does not use $(MAKE) to avoid parallelism
make -C src/test/regress check
$(MAKE) -C src/test/regress check-full
.PHONY: all check clean install install-downgrades install-all

View File

@ -1,10 +1,10 @@
| **<br/>The Citus database is 100% open source.<br/><img width=1000/><br/>Learn what's new in the [Citus 13.0 release blog](https://www.citusdata.com/blog/2025/02/06/distribute-postgresql-17-with-citus-13/) and the [Citus Updates page](https://www.citusdata.com/updates/).<br/><br/>**|
| **<br/>The Citus database is 100% open source.<br/><img width=1000/><br/>Learn what's new in the [Citus 12.0 release blog](https://www.citusdata.com/blog/2023/07/18/citus-12-schema-based-sharding-comes-to-postgres/) and the [Citus Updates page](https://www.citusdata.com/updates/).<br/><br/>**|
|---|
<br/>
![Citus Banner](images/citus-readme-banner.png)
![Citus Banner](/citus-readme-banner.png)
[![Latest Docs](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://docs.citusdata.com/)
[![Stack Overflow](https://img.shields.io/badge/Stack%20Overflow-%20-545353?logo=Stack%20Overflow)](https://stackoverflow.com/questions/tagged/citus)
@ -31,7 +31,7 @@ You can use these Citus superpowers to make your Postgres database scale-out rea
Our [SIGMOD '21](https://2021.sigmod.org/) paper [Citus: Distributed PostgreSQL for Data-Intensive Applications](https://doi.org/10.1145/3448016.3457551) gives a more detailed look into what Citus is, how it works, and why it works that way.
![Citus scales out from a single node](images/citus-scale-out.png)
![Citus scales out from a single node](/citus-scale-out.png)
Since Citus is an extension to Postgres, you can use Citus with the latest Postgres versions. And Citus works seamlessly with the PostgreSQL tools and extensions you are already familiar with.
@ -95,14 +95,14 @@ Install packages on Ubuntu / Debian:
```bash
curl https://install.citusdata.com/community/deb.sh > add-citus-repo.sh
sudo bash add-citus-repo.sh
sudo apt-get -y install postgresql-17-citus-13.0
sudo apt-get -y install postgresql-15-citus-12.0
```
Install packages on Red Hat:
Install packages on CentOS / Red Hat:
```bash
curl https://install.citusdata.com/community/rpm.sh > add-citus-repo.sh
sudo bash add-citus-repo.sh
sudo yum install -y citus130_17
sudo yum install -y citus120_15
```
To add Citus to your local PostgreSQL database, add the following to `postgresql.conf`:
@ -423,14 +423,12 @@ A Citus database cluster grows from a single PostgreSQL node into a cluster by a
Data in distributed tables is stored in “shards”, which are actually just regular PostgreSQL tables on the worker nodes. When querying a distributed table on the coordinator node, Citus will send regular SQL queries to the worker nodes. That way, all the usual PostgreSQL optimizations and extensions can automatically be used with Citus.
![Citus architecture](images/citus-architecture.png)
![Citus architecture](/citus-architecture.png)
When you send a query in which all (co-located) distributed tables have the same filter on the distribution column, Citus will automatically detect that and send the whole query to the worker node that stores the data. That way, arbitrarily complex queries are supported with minimal routing overhead, which is especially useful for scaling transactional workloads. If queries do not have a specific filter, each shard is queried in parallel, which is especially useful in analytical workloads. The Citus distributed executor is adaptive and is designed to handle both query types at the same time on the same system under high concurrency, which enables large-scale mixed workloads.
The schema and metadata of distributed tables and reference tables are automatically synchronized to all the nodes in the cluster. That way, you can connect to any node to run distributed queries. Schema changes and cluster administration still need to go through the coordinator.
Detailed descriptions of the implementation for Citus developers are provided in the [Citus Technical Documentation](src/backend/distributed/README.md).
## When to use Citus
Citus is uniquely capable of scaling both analytical and transactional workloads with up to petabytes of data. Use cases in which Citus is commonly used:
@ -440,21 +438,21 @@ Citus is uniquely capable of scaling both analytical and transactional workloads
The advanced parallel, distributed query engine in Citus combined with PostgreSQL features such as [array types](https://www.postgresql.org/docs/current/arrays.html), [JSONB](https://www.postgresql.org/docs/current/datatype-json.html), [lateral joins](https://heap.io/blog/engineering/postgresqls-powerful-new-join-type-lateral), and extensions like [HyperLogLog](https://github.com/citusdata/postgresql-hll) and [TopN](https://github.com/citusdata/postgresql-topn) allow you to build responsive analytics dashboards no matter how many customers or how much data you have.
Example real-time analytics users: [Algolia](https://www.citusdata.com/customers/algolia)
Example real-time analytics users: [Algolia](https://www.citusdata.com/customers/algolia), [Heap](https://www.citusdata.com/customers/heap)
- **[Time series data](http://docs.citusdata.com/en/stable/use_cases/timeseries.html)**:
Citus enables you to process and analyze very large amounts of time series data. The biggest Citus clusters store well over a petabyte of time series data and ingest terabytes per day.
Citus integrates seamlessly with [Postgres table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) and has [built-in functions for partitioning by time](https://www.citusdata.com/blog/2021/10/22/how-to-scale-postgres-for-time-series-data-with-citus/), which can speed up queries and writes on time series tables. You can take advantage of Cituss parallel, distributed query engine for fast analytical queries, and use the built-in *columnar storage* to compress old partitions.
Example users: [MixRank](https://www.citusdata.com/customers/mixrank)
Example users: [MixRank](https://www.citusdata.com/customers/mixrank), [Windows team](https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/architecting-petabyte-scale-analytics-by-scaling-out-postgres-on/ba-p/969685)
- **[Software-as-a-service (SaaS) applications](http://docs.citusdata.com/en/stable/use_cases/multi_tenant.html)**:
SaaS and other multi-tenant applications need to be able to scale their database as the number of tenants/customers grows. Citus enables you to transparently shard a complex data model by the tenant dimension, so your database can grow along with your business.
By distributing tables along a tenant ID column and co-locating data for the same tenant, Citus can horizontally scale complex (tenant-scoped) queries, transactions, and foreign key graphs. Reference tables and distributed DDL commands make database management a breeze compared to manual sharding. On top of that, you have a built-in distributed query engine for doing cross-tenant analytics inside the database.
Example multi-tenant SaaS users: [Salesloft](https://fivetran.com/case-studies/replicating-sharded-databases-a-case-study-of-salesloft-citus-data-and-fivetran), [ConvertFlow](https://www.citusdata.com/customers/convertflow)
Example multi-tenant SaaS users: [Copper](https://www.citusdata.com/customers/copper), [Salesloft](https://fivetran.com/case-studies/replicating-sharded-databases-a-case-study-of-salesloft-citus-data-and-fivetran), [ConvertFlow](https://www.citusdata.com/customers/convertflow)
- **[Microservices](https://docs.citusdata.com/en/stable/get_started/tutorial_microservices.html)**: Citus supports schema based sharding, which allows distributing regular database schemas across many machines. This sharding methodology fits nicely with typical Microservices architecture, where storage is fully owned by the service hence cant share the same schema definition with other tenants. Citus allows distributing horizontally scalable state across services, solving one of the [main problems](https://stackoverflow.blog/2020/11/23/the-macro-problem-with-microservices/) of microservices.

View File

@ -1,160 +0,0 @@
# Coding style
The existing code-style in our code-base is not super consistent. There are multiple reasons for that. One big reason is because our code-base is relatively old and our standards have changed over time. The second big reason is that our style-guide is different from style-guide of Postgres and some code is copied from Postgres source code and is slightly modified. The below rules are for new code. If you're changing existing code that uses a different style, use your best judgement to decide if you use the rules here or if you match the existing style.
## Using citus_indent
CI pipeline will automatically reject any PRs which do not follow our coding
conventions. The easiest way to ensure your PR adheres to those conventions is
to use the [citus_indent](https://github.com/citusdata/tools/tree/develop/uncrustify)
tool. This tool uses `uncrustify` under the hood.
```bash
# Uncrustify changes the way it formats code every release a bit. To make sure
# everyone formats consistently we use version 0.68.1:
curl -L https://github.com/uncrustify/uncrustify/archive/uncrustify-0.68.1.tar.gz | tar xz
cd uncrustify-uncrustify-0.68.1/
mkdir build
cd build
cmake ..
make -j5
sudo make install
cd ../..
git clone https://github.com/citusdata/tools.git
cd tools
make uncrustify/.install
```
Once you've done that, you can run the `make reindent` command from the top
directory to recursively check and correct the style of any source files in the
current directory. Under the hood, `make reindent` will run `citus_indent` and
some other style corrections for you.
You can also run the following in the directory of this repository to
automatically format all the files that you have changed before committing:
```bash
cat > .git/hooks/pre-commit << __EOF__
#!/bin/bash
citus_indent --check --diff || { citus_indent --diff; exit 1; }
__EOF__
chmod +x .git/hooks/pre-commit
```
## Other rules we follow that citus_indent does not enforce
* We almost always use **CamelCase**, when naming functions, variables etc., **not snake_case**.
* We also have the habits of using a **lowerCamelCase** for some variables named from their type or from their function name, as shown in the examples:
```c
bool IsCitusExtensionLoaded = false;
bool
IsAlterTableRenameStmt(RenameStmt *renameStmt)
{
AlterTableCmd *alterTableCommand = NULL;
..
..
bool isAlterTableRenameStmt = false;
..
}
```
* We **start functions with a comment**:
```c
/*
* MyNiceFunction <something in present simple tense, e.g., processes / returns / checks / takes X as input / does Y> ..
* <some more nice words> ..
* <some more nice words> ..
*/
<static?> <return type>
MyNiceFunction(..)
{
..
..
}
```
* `#includes` needs to be sorted based on below ordering and then alphabetically and we should not include what we don't need in a file:
* System includes (eg. #include<...>)
* Postgres.h (eg. #include "postgres.h")
* Toplevel imports from postgres, not contained in a directory (eg. #include "miscadmin.h")
* General postgres includes (eg . #include "nodes/...")
* Toplevel citus includes, not contained in a directory (eg. #include "citus_verion.h")
* Columnar includes (eg. #include "columnar/...")
* Distributed includes (eg. #include "distributed/...")
* Comments:
```c
/* single line comments start with a lower-case */
/*
* We start multi-line comments with a capital letter
* and keep adding a star to the beginning of each line
* until we close the comment with a star and a slash.
*/
```
* Order of function implementations and their declarations in a file:
We define static functions after the functions that call them. For example:
```c
#include<..>
#include<..>
..
..
typedef struct
{
..
..
} MyNiceStruct;
..
..
PG_FUNCTION_INFO_V1(my_nice_udf1);
PG_FUNCTION_INFO_V1(my_nice_udf2);
..
..
// .. somewhere on top of the file …
static void MyNiceStaticlyDeclaredFunction1(…);
static void MyNiceStaticlyDeclaredFunction2(…);
..
..
void
MyNiceFunctionExternedViaHeaderFile(..)
{
..
..
MyNiceStaticlyDeclaredFunction1(..);
..
..
MyNiceStaticlyDeclaredFunction2(..);
..
}
..
..
// we define this first because it's called by MyNiceFunctionExternedViaHeaderFile()
// before MyNiceStaticlyDeclaredFunction2()
static void
MyNiceStaticlyDeclaredFunction1(…)
{
}
..
..
// then we define this
static void
MyNiceStaticlyDeclaredFunction2(…)
{
}
```

View File

@ -4,22 +4,7 @@ set -euo pipefail
# shellcheck disable=SC1091
source ci/ci_helpers.sh
# Find the line that exactly matches "RegisterCitusConfigVariables(void)" in
# shared_library_init.c. grep command returns something like
# "934:RegisterCitusConfigVariables(void)" and we extract the line number
# with cut.
RegisterCitusConfigVariables_begin_linenumber=$(grep -n "^RegisterCitusConfigVariables(void)$" src/backend/distributed/shared_library_init.c | cut -d: -f1)
# Consider the lines starting from $RegisterCitusConfigVariables_begin_linenumber,
# grep the first line that starts with "}" and extract the line number with cut
# as in the previous step.
RegisterCitusConfigVariables_length=$(tail -n +$RegisterCitusConfigVariables_begin_linenumber src/backend/distributed/shared_library_init.c | grep -n -m 1 "^}$" | cut -d: -f1)
# extract the function definition of RegisterCitusConfigVariables into a temp file
tail -n +$RegisterCitusConfigVariables_begin_linenumber src/backend/distributed/shared_library_init.c | head -n $(($RegisterCitusConfigVariables_length)) > RegisterCitusConfigVariables_func_def.out
# extract citus gucs in the form of <tab><tab>"citus.X"
grep -P "^[\t][\t]\"citus\.[a-zA-Z_0-9]+\"" RegisterCitusConfigVariables_func_def.out > gucs.out
LC_COLLATE=C sort -c gucs.out
# extract citus gucs in the form of "citus.X"
grep -o -E "(\.*\"citus\.\w+\")," src/backend/distributed/shared_library_init.c > gucs.out
sort -c gucs.out
rm gucs.out
rm RegisterCitusConfigVariables_func_def.out

View File

Before

Width:  |  Height:  |  Size: 94 KiB

After

Width:  |  Height:  |  Size: 94 KiB

View File

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 22 KiB

View File

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

20
configure vendored
View File

@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for Citus 13.2devel.
# Generated by GNU Autoconf 2.69 for Citus 13.0.0.
#
#
# Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='Citus'
PACKAGE_TARNAME='citus'
PACKAGE_VERSION='13.2devel'
PACKAGE_STRING='Citus 13.2devel'
PACKAGE_VERSION='13.0.0'
PACKAGE_STRING='Citus 13.0.0'
PACKAGE_BUGREPORT=''
PACKAGE_URL=''
@ -1262,7 +1262,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
\`configure' configures Citus 13.2devel to adapt to many kinds of systems.
\`configure' configures Citus 13.0.0 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1324,7 +1324,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
short | recursive ) echo "Configuration of Citus 13.2devel:";;
short | recursive ) echo "Configuration of Citus 13.0.0:";;
esac
cat <<\_ACEOF
@ -1429,7 +1429,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
Citus configure 13.2devel
Citus configure 13.0.0
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@ -1912,7 +1912,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
It was created by Citus $as_me 13.2devel, which was
It was created by Citus $as_me 13.0.0, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@ -2588,7 +2588,7 @@ fi
if test "$with_pg_version_check" = no; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: building against PostgreSQL $version_num (skipped compatibility check)" >&5
$as_echo "$as_me: building against PostgreSQL $version_num (skipped compatibility check)" >&6;}
elif test "$version_num" != '15' -a "$version_num" != '16' -a "$version_num" != '17'; then
elif test "$version_num" != '14' -a "$version_num" != '15' -a "$version_num" != '16' -a "$version_num" != '17'; then
as_fn_error $? "Citus is not compatible with the detected PostgreSQL version ${version_num}." "$LINENO" 5
else
{ $as_echo "$as_me:${as_lineno-$LINENO}: building against PostgreSQL $version_num" >&5
@ -5393,7 +5393,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
This file was extended by Citus $as_me 13.2devel, which was
This file was extended by Citus $as_me 13.0.0, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@ -5455,7 +5455,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
Citus config.status 13.2devel
Citus config.status 13.0.0
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"

View File

@ -5,7 +5,7 @@
# everyone needing autoconf installed, the resulting files are checked
# into the SCM.
AC_INIT([Citus], [13.2devel])
AC_INIT([Citus], [13.0.0])
AC_COPYRIGHT([Copyright (c) Citus Data, Inc.])
# we'll need sed and awk for some of the version commands
@ -80,7 +80,7 @@ AC_SUBST(with_pg_version_check)
if test "$with_pg_version_check" = no; then
AC_MSG_NOTICE([building against PostgreSQL $version_num (skipped compatibility check)])
elif test "$version_num" != '15' -a "$version_num" != '16' -a "$version_num" != '17'; then
elif test "$version_num" != '14' -a "$version_num" != '15' -a "$version_num" != '16' -a "$version_num" != '17'; then
AC_MSG_ERROR([Citus is not compatible with the detected PostgreSQL version ${version_num}.])
else
AC_MSG_NOTICE([building against PostgreSQL $version_num])

133
gucs.out Normal file
View File

@ -0,0 +1,133 @@
"citus.all_modifications_commutative",
"citus.allow_modifications_from_workers_to_replicated_tables",
"citus.allow_nested_distributed_execution",
"citus.allow_unsafe_constraints",
"citus.allow_unsafe_locks_from_workers",
"citus.background_task_queue_interval",
"citus.check_available_space_before_move",
"citus.cluster_name",
"citus.coordinator_aggregation_strategy",
"citus.copy_switchover_threshold",
"citus.count_distinct_error_rate",
"citus.cpu_priority",
"citus.cpu_priority_for_logical_replication_senders",
"citus.create_object_propagation",
"citus.defer_drop_after_shard_move",
"citus.defer_drop_after_shard_split",
"citus.defer_shard_delete_interval",
"citus.desired_percent_disk_available_after_move",
"citus.distributed_deadlock_detection_factor",
"citus.enable_alter_database_owner",
"citus.enable_alter_role_propagation",
"citus.enable_alter_role_set_propagation",
"citus.enable_binary_protocol",
"citus.enable_change_data_capture",
"citus.enable_cluster_clock",
"citus.enable_cost_based_connection_establishment",
"citus.enable_create_role_propagation",
"citus.enable_create_type_propagation",
"citus.enable_ddl_propagation",
"citus.enable_deadlock_prevention",
"citus.enable_fast_path_router_planner",
"citus.enable_local_execution",
"citus.enable_local_reference_table_foreign_keys",
"citus.enable_manual_changes_to_shards",
"citus.enable_manual_metadata_changes_for_user",
"citus.enable_metadata_sync",
"citus.enable_non_colocated_router_query_pushdown",
"citus.enable_repartition_joins",
"citus.enable_repartitioned_insert_select",
"citus.enable_router_execution",
"citus.enable_schema_based_sharding",
"citus.enable_single_hash_repartition_joins",
"citus.enable_statistics_collection",
"citus.enable_unique_job_ids",
"citus.enable_unsafe_triggers",
"citus.enable_unsupported_feature_messages",
"citus.enable_version_checks",
"citus.enforce_foreign_key_restrictions",
"citus.enforce_object_restrictions_for_local_objects",
"citus.executor_slow_start_interval",
"citus.explain_all_tasks",
"citus.explain_analyze_sort_method",
"citus.explain_distributed_queries",
"citus.force_max_query_parallelization",
"citus.function_opens_transaction_block",
"citus.grep_remote_commands",
"citus.hide_citus_dependent_objects",
"citus.hide_shards_from_app_name_prefixes",
"citus.isolation_test_session_process_id",
"citus.isolation_test_session_remote_process_id",
"citus.limit_clause_row_fetch_count",
"citus.local_copy_flush_threshold",
"citus.local_hostname",
"citus.local_shared_pool_size",
"citus.local_table_join_policy",
"citus.log_distributed_deadlock_detection",
"citus.log_intermediate_results",
"citus.log_local_commands",
"citus.log_multi_join_order",
"citus.log_remote_commands",
"citus.logical_replication_timeout",
"citus.main_db",
"citus.max_adaptive_executor_pool_size",
"citus.max_background_task_executors",
"citus.max_background_task_executors_per_node",
"citus.max_cached_connection_lifetime",
"citus.max_cached_conns_per_worker",
"citus.max_client_connections",
"citus.max_high_priority_background_processes",
"citus.max_intermediate_result_size",
"citus.max_matview_size_to_auto_recreate",
"citus.max_rebalancer_logged_ignored_moves",
"citus.max_shared_pool_size",
"citus.max_worker_nodes_tracked",
"citus.metadata_sync_interval",
"citus.metadata_sync_mode",
"citus.metadata_sync_retry_interval",
"citus.mitmfifo",
"citus.multi_shard_modify_mode",
"citus.multi_task_query_log_level",
"citus.next_cleanup_record_id",
"citus.next_operation_id",
"citus.next_placement_id",
"citus.next_shard_id",
"citus.node_connection_timeout",
"citus.node_conninfo",
"citus.override_table_visibility",
"citus.prevent_incomplete_connection_establishment",
"citus.propagate_session_settings_for_loopback_connection",
"citus.propagate_set_commands",
"citus.rebalancer_by_disk_size_base_cost",
"citus.recover_2pc_interval",
"citus.remote_copy_flush_threshold",
"citus.remote_task_check_interval",
"citus.repartition_join_bucket_count_per_node",
"citus.replicate_reference_tables_on_activate",
"citus.replication_model",
"citus.running_under_citus_test_suite",
"citus.select_opens_transaction_block",
"citus.shard_count",
"citus.shard_replication_factor",
"citus.show_shards_for_app_name_prefixes",
"citus.skip_advisory_lock_permission_checks",
"citus.skip_constraint_validation",
"citus.skip_jsonb_validation_in_copy",
"citus.sort_returning",
"citus.stat_statements_max",
"citus.stat_statements_purge_interval",
"citus.stat_statements_track",
"citus.stat_tenants_limit",
"citus.stat_tenants_log_level",
"citus.stat_tenants_period",
"citus.stat_tenants_track",
"citus.stat_tenants_untracked_sample_rate",
"citus.subquery_pushdown",
"citus.task_assignment_policy",
"citus.task_executor_type",
"citus.use_citus_managed_tables",
"citus.use_secondary_nodes",
"citus.values_materialization_threshold",
"citus.version",
"citus.worker_min_messages",
"citus.writable_standby_coordinator",

Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 111 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 168 KiB

View File

@ -1,6 +1,6 @@
# Columnar extension
comment = 'Citus Columnar extension'
default_version = '12.2-1'
default_version = '11.3-1'
module_pathname = '$libdir/citus_columnar'
relocatable = false
schema = pg_catalog

View File

@ -32,6 +32,8 @@
#include "optimizer/paths.h"
#include "optimizer/plancat.h"
#include "optimizer/restrictinfo.h"
#include "citus_version.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "parser/parse_relation.h"
#include "parser/parsetree.h"
@ -43,8 +45,6 @@
#include "utils/selfuncs.h"
#include "utils/spccache.h"
#include "citus_version.h"
#include "columnar/columnar.h"
#include "columnar/columnar_customscan.h"
#include "columnar/columnar_metadata.h"
@ -1051,15 +1051,6 @@ FindCandidateRelids(PlannerInfo *root, RelOptInfo *rel, List *joinClauses)
candidateRelids = bms_del_members(candidateRelids, rel->relids);
candidateRelids = bms_del_members(candidateRelids, rel->lateral_relids);
/*
* For the relevant PG16 commit requiring this addition:
* postgres/postgres@2489d76
*/
#if PG_VERSION_NUM >= PG_VERSION_16
candidateRelids = bms_del_members(candidateRelids, root->outer_join_rels);
#endif
return candidateRelids;
}
@ -1321,8 +1312,11 @@ AddColumnarScanPath(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte,
cpath->methods = &ColumnarScanPathMethods;
#if (PG_VERSION_NUM >= PG_VERSION_15)
/* necessary to avoid extra Result node in PG15 */
cpath->flags = CUSTOMPATH_SUPPORT_PROJECTION;
#endif
/*
* populate generic path information

View File

@ -24,7 +24,6 @@
#include "postgres.h"
#include "miscadmin.h"
#include "port.h"
#include "safe_lib.h"
#include "access/heapam.h"
@ -43,6 +42,19 @@
#include "executor/spi.h"
#include "lib/stringinfo.h"
#include "nodes/execnodes.h"
#include "citus_version.h"
#include "columnar/columnar.h"
#include "columnar/columnar_storage.h"
#include "columnar/columnar_version_compat.h"
#include "distributed/listutils.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "parser/parse_relation.h"
#endif
#include "port.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
#include "storage/procarray.h"
@ -52,18 +64,7 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "citus_version.h"
#include "pg_version_constants.h"
#include "columnar/columnar.h"
#include "columnar/columnar_storage.h"
#include "columnar/columnar_version_compat.h"
#include "distributed/listutils.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "parser/parse_relation.h"
#include "storage/relfilelocator.h"
#include "utils/relfilenumbermap.h"
#else
@ -1685,7 +1686,7 @@ DeleteTupleAndEnforceConstraints(ModifyState *state, HeapTuple heapTuple)
simple_heap_delete(state->rel, tid);
/* execute AFTER ROW DELETE Triggers to enforce constraints */
ExecARDeleteTriggers(estate, resultRelInfo, tid, NULL, NULL, false);
ExecARDeleteTriggers_compat(estate, resultRelInfo, tid, NULL, NULL, false);
}

View File

@ -877,7 +877,7 @@ columnar_relation_set_new_filelocator(Relation rel,
*freezeXid = RecentXmin;
*minmulti = GetOldestMultiXactId();
SMgrRelation srel = RelationCreateStorage(*newrlocator, persistence, true);
SMgrRelation srel = RelationCreateStorage_compat(*newrlocator, persistence, true);
ColumnarStorageInit(srel, ColumnarMetadataNewStorageId());
InitColumnarOptions(rel->rd_id);
@ -2245,6 +2245,7 @@ ColumnarProcessAlterTable(AlterTableStmt *alterTableStmt, List **columnarOptions
columnarRangeVar = alterTableStmt->relation;
}
}
#if PG_VERSION_NUM >= PG_VERSION_15
else if (alterTableCmd->subtype == AT_SetAccessMethod)
{
if (columnarRangeVar || *columnarOptions)
@ -2264,6 +2265,7 @@ ColumnarProcessAlterTable(AlterTableStmt *alterTableStmt, List **columnarOptions
DeleteColumnarTableOptions(RelationGetRelid(rel), true);
}
}
#endif /* PG_VERSION_15 */
}
relation_close(rel, NoLock);
@ -2647,12 +2649,21 @@ ColumnarCheckLogicalReplication(Relation rel)
return;
}
#if PG_VERSION_NUM >= PG_VERSION_15
{
PublicationDesc pubdesc;
RelationBuildPublicationDesc(rel, &pubdesc);
pubActionInsert = pubdesc.pubactions.pubinsert;
}
#else
if (rel->rd_pubactions == NULL)
{
GetRelationPublicationActions(rel);
Assert(rel->rd_pubactions != NULL);
}
pubActionInsert = rel->rd_pubactions->pubinsert;
#endif
if (pubActionInsert)
{

View File

@ -29,12 +29,6 @@
#include "utils/rel.h"
#include "pg_version_compat.h"
#include "pg_version_constants.h"
#include "columnar/columnar.h"
#include "columnar/columnar_storage.h"
#include "columnar/columnar_version_compat.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "storage/relfilelocator.h"
#include "utils/relfilenumbermap.h"
@ -42,6 +36,10 @@
#include "utils/relfilenodemap.h"
#endif
#include "columnar/columnar.h"
#include "columnar/columnar_storage.h"
#include "columnar/columnar_version_compat.h"
struct ColumnarWriteState
{
TupleDesc tupleDescriptor;

View File

@ -1 +0,0 @@
-- citus_columnar--11.3-1--12.2-1

View File

@ -1 +0,0 @@
-- citus_columnar--12.2-1--11.3-1

View File

@ -18,7 +18,7 @@ generated_downgrade_sql_files += $(patsubst %,$(citus_abs_srcdir)/build/sql/%,$(
DATA_built = $(generated_sql_files)
# directories with source files
SUBDIRS = . commands connection ddl deparser executor metadata operations planner progress relay safeclib shardsplit stats test transaction utils worker clock
SUBDIRS = . commands connection ddl deparser executor metadata operations planner progress relay safeclib shardsplit test transaction utils worker clock
# enterprise modules
SUBDIRS += replication

File diff suppressed because it is too large Load Diff

View File

@ -14,6 +14,11 @@ override CPPFLAGS += -DDECODER=\"$(DECODER)\" -I$(citus_abs_top_srcdir)/include
install: install-cdc
clean: clean-cdc
install-cdc:
mkdir -p '$(citus_decoders_dir)'
$(INSTALL_SHLIB) citus_$(DECODER)$(DLSUFFIX) '$(citus_decoders_dir)/$(DECODER)$(DLSUFFIX)'
$(INSTALL_SHLIB) citus_$(DECODER).so '$(citus_decoders_dir)/$(DECODER).so'
clean-cdc:
rm -f '$(DESTDIR)$(datadir)/$(datamoduledir)/citus_decoders/$(DECODER).so'

View File

@ -1,6 +1,6 @@
# Citus extension
comment = 'Citus distributed database'
default_version = '13.2-1'
default_version = '13.0-1'
module_pathname = '$libdir/citus'
relocatable = false
schema = pg_catalog

View File

@ -145,6 +145,17 @@ LogicalClockShmemSize(void)
void
InitializeClusterClockMem(void)
{
/* On PG 15 and above, we use shmem_request_hook_type */
#if PG_VERSION_NUM < PG_VERSION_15
/* allocate shared memory for pre PG-15 versions */
if (!IsUnderPostmaster)
{
RequestAddinShmemSpace(LogicalClockShmemSize());
}
#endif
prev_shmem_startup_hook = shmem_startup_hook;
shmem_startup_hook = LogicalClockShmemInit;
}
@ -386,7 +397,7 @@ AdjustClocksToTransactionHighest(List *nodeConnectionList,
/* Set the clock value on participating worker nodes */
appendStringInfo(queryToSend,
"SELECT citus_internal.adjust_local_clock_to_remote"
"SELECT pg_catalog.citus_internal_adjust_local_clock_to_remote"
"('(%lu, %u)'::pg_catalog.cluster_clock);",
transactionClockValue->logical, transactionClockValue->counter);
@ -420,11 +431,6 @@ PrepareAndSetTransactionClock(void)
MultiConnection *connection = dlist_container(MultiConnection, transactionNode,
iter.cur);
WorkerNode *workerNode = FindWorkerNode(connection->hostname, connection->port);
if (!workerNode)
{
ereport(WARNING, errmsg("Worker node is missing"));
continue;
}
/* Skip the node if we already in the list */
if (list_member_int(nodeList, workerNode->groupId))

View File

@ -68,6 +68,8 @@ CreateCollationDDLInternal(Oid collationId, Oid *collowner, char **quotedCollati
char *collcollate;
char *collctype;
#if PG_VERSION_NUM >= PG_VERSION_15
/*
* In PG15, there is an added option to use ICU as global locale provider.
* pg_collation has three locale-related fields: collcollate and collctype,
@ -110,6 +112,16 @@ CreateCollationDDLInternal(Oid collationId, Oid *collowner, char **quotedCollati
}
Assert((collcollate && collctype) || colllocale);
#else
/*
* In versions before 15, collcollate and collctype were type "name". Use
* pstrdup() to match the interface of 15 so that we consistently free the
* result later.
*/
collcollate = pstrdup(NameStr(collationForm->collcollate));
collctype = pstrdup(NameStr(collationForm->collctype));
#endif
if (collowner != NULL)
{
@ -135,6 +147,7 @@ CreateCollationDDLInternal(Oid collationId, Oid *collowner, char **quotedCollati
"CREATE COLLATION %s (provider = '%s'",
*quotedCollationName, providerString);
#if PG_VERSION_NUM >= PG_VERSION_15
if (colllocale)
{
appendStringInfo(&collationNameDef,
@ -160,7 +173,24 @@ CreateCollationDDLInternal(Oid collationId, Oid *collowner, char **quotedCollati
pfree(collcollate);
pfree(collctype);
}
#else
if (strcmp(collcollate, collctype) == 0)
{
appendStringInfo(&collationNameDef,
", locale = %s",
quote_literal_cstr(collcollate));
}
else
{
appendStringInfo(&collationNameDef,
", lc_collate = %s, lc_ctype = %s",
quote_literal_cstr(collcollate),
quote_literal_cstr(collctype));
}
pfree(collcollate);
pfree(collctype);
#endif
#if PG_VERSION_NUM >= PG_VERSION_16
char *collicurules = NULL;
datum = SysCacheGetAttr(COLLOID, heapTuple, Anum_pg_collation_collicurules, &isnull);

View File

@ -1,131 +0,0 @@
/*-------------------------------------------------------------------------
*
* comment.c
* Commands to interact with the comments for all database
* object types.
*
* Copyright (c) Citus Data, Inc.
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "catalog/pg_shdescription.h"
#include "nodes/parsenodes.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "distributed/comment.h"
static char * GetCommentForObject(Oid classOid, Oid objectOid);
List *
GetCommentPropagationCommands(Oid classOid, Oid objOoid, char *objectName, ObjectType
objectType)
{
List *commands = NIL;
StringInfo commentStmt = makeStringInfo();
/* Get the comment for the database */
char *comment = GetCommentForObject(classOid, objOoid);
char const *commentObjectType = ObjectTypeNames[objectType];
/* Create the SQL command to propagate the comment to other nodes */
if (comment != NULL)
{
appendStringInfo(commentStmt, "COMMENT ON %s %s IS %s;", commentObjectType,
quote_identifier(objectName),
quote_literal_cstr(comment));
}
/* Add the command to the list */
if (commentStmt->len > 0)
{
commands = list_make1(commentStmt->data);
}
return commands;
}
static char *
GetCommentForObject(Oid classOid, Oid objectOid)
{
HeapTuple tuple;
char *comment = NULL;
/* Open pg_shdescription catalog */
Relation shdescRelation = table_open(SharedDescriptionRelationId, AccessShareLock);
/* Scan the table */
ScanKeyData scanKey[2];
ScanKeyInit(&scanKey[0],
Anum_pg_shdescription_objoid,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(objectOid));
ScanKeyInit(&scanKey[1],
Anum_pg_shdescription_classoid,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(classOid));
bool indexOk = true;
int scanKeyCount = 2;
SysScanDesc scan = systable_beginscan(shdescRelation, SharedDescriptionObjIndexId,
indexOk, NULL, scanKeyCount,
scanKey);
if ((tuple = systable_getnext(scan)) != NULL)
{
bool isNull = false;
TupleDesc tupdesc = RelationGetDescr(shdescRelation);
Datum descDatum = heap_getattr(tuple, Anum_pg_shdescription_description, tupdesc,
&isNull);
/* Add the command to the list */
if (!isNull)
{
comment = TextDatumGetCString(descDatum);
}
else
{
comment = NULL;
}
}
/* End the scan and close the catalog */
systable_endscan(scan);
table_close(shdescRelation, AccessShareLock);
return comment;
}
/*
* CommentObjectAddress resolves the ObjectAddress for the object
* on which the comment is placed. Optionally errors if the object does not
* exist based on the missing_ok flag passed in by the caller.
*/
List *
CommentObjectAddress(Node *node, bool missing_ok, bool isPostprocess)
{
CommentStmt *stmt = castNode(CommentStmt, node);
Relation relation;
ObjectAddress objectAddress = get_object_address(stmt->objtype, stmt->object,
&relation, AccessExclusiveLock,
missing_ok);
ObjectAddress *objectAddressCopy = palloc0(sizeof(ObjectAddress));
*objectAddressCopy = objectAddress;
return list_make1(objectAddressCopy);
}

View File

@ -170,10 +170,12 @@ static void EnsureDistributedSequencesHaveOneType(Oid relationId,
static void CopyLocalDataIntoShards(Oid distributedTableId);
static List * TupleDescColumnNameList(TupleDesc tupleDescriptor);
#if (PG_VERSION_NUM >= PG_VERSION_15)
static bool DistributionColumnUsesNumericColumnNegativeScale(TupleDesc relationDesc,
Var *distributionColumn);
static int numeric_typmod_scale(int32 typmod);
static bool is_valid_numeric_typmod(int32 typmod);
#endif
static bool DistributionColumnUsesGeneratedStoredColumn(TupleDesc relationDesc,
Var *distributionColumn);
@ -2112,6 +2114,8 @@ EnsureRelationCanBeDistributed(Oid relationId, Var *distributionColumn,
"AS (...) STORED.")));
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/* verify target relation is not distributed by a column of type numeric with negative scale */
if (distributionMethod != DISTRIBUTE_BY_NONE &&
DistributionColumnUsesNumericColumnNegativeScale(relationDesc,
@ -2122,6 +2126,7 @@ EnsureRelationCanBeDistributed(Oid relationId, Var *distributionColumn,
errdetail("Distribution column must not use numeric type "
"with negative scale")));
}
#endif
/* check for support function needed by specified partition method */
if (distributionMethod == DISTRIBUTE_BY_HASH)
@ -2727,15 +2732,11 @@ CopyFromLocalTableIntoDistTable(Oid localTableId, Oid distributedTableId)
ExprContext *econtext = GetPerTupleExprContext(estate);
econtext->ecxt_scantuple = slot;
const bool nonPublishableData = false;
/* we don't track query counters when distributing a table */
const bool trackQueryCounters = false;
DestReceiver *copyDest =
(DestReceiver *) CreateCitusCopyDestReceiver(distributedTableId,
columnNameList,
partitionColumnIndex,
estate, NULL, nonPublishableData,
trackQueryCounters);
estate, NULL, nonPublishableData);
/* initialise state for writing to shards, we'll open connections on demand */
copyDest->rStartup(copyDest, 0, sourceTupleDescriptor);
@ -2843,6 +2844,8 @@ TupleDescColumnNameList(TupleDesc tupleDescriptor)
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* is_valid_numeric_typmod checks if the typmod value is valid
*
@ -2892,6 +2895,8 @@ DistributionColumnUsesNumericColumnNegativeScale(TupleDesc relationDesc,
}
#endif
/*
* DistributionColumnUsesGeneratedStoredColumn returns whether a given relation uses
* GENERATED ALWAYS AS (...) STORED on distribution column

View File

@ -13,97 +13,35 @@
#include "miscadmin.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/xact.h"
#include "catalog/objectaddress.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_database.h"
#include "catalog/pg_database_d.h"
#include "catalog/pg_tablespace.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "nodes/parsenodes.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/syscache.h"
#include "distributed/adaptive_executor.h"
#include "distributed/commands.h"
#include "distributed/commands/serialize_distributed_ddls.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/comment.h"
#include "distributed/deparse_shard_query.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#include "distributed/local_executor.h"
#include "distributed/metadata/distobject.h"
#include "distributed/metadata_sync.h"
#include "distributed/metadata_utility.h"
#include "distributed/multi_executor.h"
#include "distributed/relation_access_tracking.h"
#include "distributed/shard_cleaner.h"
#include "distributed/worker_protocol.h"
#include "distributed/worker_transaction.h"
/*
* Used to save original name of the database before it is replaced with a
* temporary name for failure handling purposes in PreprocessCreateDatabaseStmt().
*/
static char *CreateDatabaseCommandOriginalDbName = NULL;
/*
* The format string used when creating a temporary databases for failure
* handling purposes.
*
* The fields are as follows to ensure using a unique name for each temporary
* database:
* - operationId: The operation id returned by RegisterOperationNeedingCleanup().
* - groupId: The group id of the worker node where CREATE DATABASE command
* is issued from.
*/
#define TEMP_DATABASE_NAME_FMT "citus_temp_database_%lu_%d"
/*
* DatabaseCollationInfo is used to store collation related information of a database.
*/
typedef struct DatabaseCollationInfo
{
char *datcollate;
char *datctype;
char *daticulocale;
char *datcollversion;
#if PG_VERSION_NUM >= PG_VERSION_16
char *daticurules;
#endif
} DatabaseCollationInfo;
static char * GenerateCreateDatabaseStatementFromPgDatabase(Form_pg_database
databaseForm);
static DatabaseCollationInfo GetDatabaseCollation(Oid dbOid);
static AlterOwnerStmt * RecreateAlterDatabaseOwnerStmt(Oid databaseOid);
static char * GetLocaleProviderString(char datlocprovider);
static char * GetTablespaceName(Oid tablespaceOid);
static Oid get_database_owner(Oid db_oid);
static ObjectAddress * GetDatabaseAddressFromDatabaseName(char *databaseName,
bool missingOk);
static List * FilterDistributedDatabases(List *databases);
static Oid get_database_owner(Oid dbId);
List * PreprocessGrantOnDatabaseStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext);
/* controlled via GUC */
bool EnableCreateDatabasePropagation = false;
bool EnableAlterDatabaseOwner = true;
/*
* AlterDatabaseOwnerObjectAddress returns the ObjectAddress of the database that is the
* object of the AlterOwnerStmt. Errors if missing_ok is false.
@ -160,13 +98,13 @@ RecreateAlterDatabaseOwnerStmt(Oid databaseOid)
* get_database_owner returns the Oid of the role owning the database
*/
static Oid
get_database_owner(Oid dbId)
get_database_owner(Oid db_oid)
{
HeapTuple tuple = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(dbId));
HeapTuple tuple = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(db_oid));
if (!HeapTupleIsValid(tuple))
{
ereport(ERROR, (errcode(ERRCODE_UNDEFINED_DATABASE),
errmsg("database with OID %u does not exist", dbId)));
errmsg("database with OID %u does not exist", db_oid)));
}
Oid dba = ((Form_pg_database) GETSTRUCT(tuple))->datdba;
@ -196,23 +134,17 @@ PreprocessGrantOnDatabaseStmt(Node *node, const char *queryString,
GrantStmt *stmt = castNode(GrantStmt, node);
Assert(stmt->objtype == OBJECT_DATABASE);
List *distributedDatabases = FilterDistributedDatabases(stmt->objects);
List *databaseList = stmt->objects;
if (list_length(distributedDatabases) == 0)
if (list_length(databaseList) == 0)
{
return NIL;
}
EnsureCoordinator();
List *originalObjects = stmt->objects;
stmt->objects = distributedDatabases;
char *sql = DeparseTreeNode((Node *) stmt);
stmt->objects = originalObjects;
List *commands = list_make3(DISABLE_DDL_PROPAGATION,
(void *) sql,
ENABLE_DDL_PROPAGATION);
@ -221,192 +153,21 @@ PreprocessGrantOnDatabaseStmt(Node *node, const char *queryString,
}
/*
* FilterDistributedDatabases filters the database list and returns the distributed ones,
* as a list.
*/
static List *
FilterDistributedDatabases(List *databases)
{
List *distributedDatabases = NIL;
String *databaseName = NULL;
foreach_declared_ptr(databaseName, databases)
{
bool missingOk = true;
ObjectAddress *dbAddress =
GetDatabaseAddressFromDatabaseName(strVal(databaseName), missingOk);
if (IsAnyObjectDistributed(list_make1(dbAddress)))
{
distributedDatabases = lappend(distributedDatabases, databaseName);
}
}
return distributedDatabases;
}
/*
* IsSetTablespaceStatement returns true if the statement is a SET TABLESPACE statement,
* false otherwise.
*/
static bool
IsSetTablespaceStatement(AlterDatabaseStmt *stmt)
{
DefElem *def = NULL;
foreach_declared_ptr(def, stmt->options)
{
if (strcmp(def->defname, "tablespace") == 0)
{
return true;
}
}
return false;
}
/*
* PreprocessAlterDatabaseStmt is executed before the statement is applied to the local
* postgres instance.
*
* In this stage we can prepare the commands that need to be run on all workers to grant
* on databases.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*/
List *
PreprocessAlterDatabaseStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
bool missingOk = false;
AlterDatabaseStmt *stmt = castNode(AlterDatabaseStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
if (!ShouldPropagate() || !IsAnyObjectDistributed(list_make1(dbAddress)))
{
return NIL;
}
EnsureCoordinator();
SerializeDistributedDDLsOnObjectClassObject(OCLASS_DATABASE, stmt->dbname);
char *sql = DeparseTreeNode((Node *) stmt);
List *commands = list_make3(DISABLE_DDL_PROPAGATION,
sql,
ENABLE_DDL_PROPAGATION);
if (IsSetTablespaceStatement(stmt))
{
/*
* Set tablespace does not work inside a transaction.Therefore, we need to use
* NontransactionalNodeDDLTask to run the command on the workers outside
* the transaction block.
*/
bool warnForPartialFailure = true;
return NontransactionalNodeDDLTaskList(NON_COORDINATOR_NODES, commands,
warnForPartialFailure);
}
else
{
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
}
/*
* PreprocessAlterDatabaseRefreshCollStmt is executed before the statement is applied to
* the local postgres instance.
*
* In this stage we can prepare the commands that need to be run on all workers to grant
* on databases.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*/
List *
PreprocessAlterDatabaseRefreshCollStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
bool missingOk = true;
AlterDatabaseRefreshCollStmt *stmt = castNode(AlterDatabaseRefreshCollStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
if (!ShouldPropagate() || !IsAnyObjectDistributed(list_make1(dbAddress)))
{
return NIL;
}
EnsureCoordinator();
SerializeDistributedDDLsOnObjectClassObject(OCLASS_DATABASE, stmt->dbname);
char *sql = DeparseTreeNode((Node *) stmt);
List *commands = list_make3(DISABLE_DDL_PROPAGATION,
(void *) sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
/*
* PreprocessAlterDatabaseRenameStmt is executed before the statement is applied to
* the local postgres instance.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*
* We acquire this lock here instead of PostprocessAlterDatabaseRenameStmt because the
* command renames the database and SerializeDistributedDDLsOnObjectClass resolves the
* object on workers based on database name. For this reason, we need to acquire the lock
* before the command is applied to the local postgres instance.
*/
List *
PreprocessAlterDatabaseRenameStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
bool missingOk = true;
RenameStmt *stmt = castNode(RenameStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->subname,
missingOk);
if (!ShouldPropagate() || !IsAnyObjectDistributed(list_make1(dbAddress)))
{
return NIL;
}
EnsureCoordinator();
/*
* Different than other ALTER DATABASE commands, we first acquire a lock
* by providing InvalidOid because we want ALTER TABLE .. RENAME TO ..
* commands to block not only with ALTER DATABASE operations but also
* with CREATE DATABASE operations because they might cause name conflicts
* and that could also cause deadlocks too.
*/
SerializeDistributedDDLsOnObjectClass(OCLASS_DATABASE);
SerializeDistributedDDLsOnObjectClassObject(OCLASS_DATABASE, stmt->subname);
return NIL;
}
/*
* PostprocessAlterDatabaseRenameStmt is executed after the statement is applied to the local
* postgres instance. In this stage we prepare ALTER DATABASE RENAME statement to be run on
* all workers.
*/
List *
PostprocessAlterDatabaseRenameStmt(Node *node, const char *queryString)
{
bool missingOk = false;
RenameStmt *stmt = castNode(RenameStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->newname,
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
if (!ShouldPropagate() || !IsAnyObjectDistributed(list_make1(dbAddress)))
{
return NIL;
@ -424,32 +185,27 @@ PostprocessAlterDatabaseRenameStmt(Node *node, const char *queryString)
}
#if PG_VERSION_NUM >= PG_VERSION_15
/*
* PreprocessAlterDatabaseSetStmt is executed before the statement is applied to the local
* postgres instance.
*
* In this stage we can prepare the commands that need to be run on all workers to grant
* on databases.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*/
List *
PreprocessAlterDatabaseSetStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
PreprocessAlterDatabaseRefreshCollStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
AlterDatabaseSetStmt *stmt = castNode(AlterDatabaseSetStmt, node);
bool missingOk = true;
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
if (!ShouldPropagate() || !IsAnyObjectDistributed(list_make1(dbAddress)))
if (!ShouldPropagate())
{
return NIL;
}
AlterDatabaseRefreshCollStmt *stmt = castNode(AlterDatabaseRefreshCollStmt, node);
EnsureCoordinator();
SerializeDistributedDDLsOnObjectClassObject(OCLASS_DATABASE, stmt->dbname);
char *sql = DeparseTreeNode((Node *) stmt);
@ -461,294 +217,7 @@ PreprocessAlterDatabaseSetStmt(Node *node, const char *queryString,
}
/*
* PreprocessCreateDatabaseStmt is executed before the statement is applied to the local
* Postgres instance.
*
* In this stage, we perform validations that we want to ensure before delegating to
* previous utility hooks because it might not be convenient to throw an error in an
* implicit transaction that creates a database. Also in this stage, we save the original
* database name and replace dbname field with a temporary name for failure handling
* purposes. We let Postgres create the database with the temporary name, insert a cleanup
* record for the temporary database name on all nodes and let PostprocessCreateDatabaseStmt()
* to return the distributed DDL job that both creates the database with the temporary name
* and then renames it back to its original name.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*/
List *
PreprocessCreateDatabaseStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
if (!EnableCreateDatabasePropagation || !ShouldPropagate())
{
return NIL;
}
EnsureCoordinatorIsInMetadata();
CreatedbStmt *stmt = castNode(CreatedbStmt, node);
EnsureSupportedCreateDatabaseCommand(stmt);
SerializeDistributedDDLsOnObjectClass(OCLASS_DATABASE);
OperationId operationId = RegisterOperationNeedingCleanup();
char *tempDatabaseName = psprintf(TEMP_DATABASE_NAME_FMT,
operationId, GetLocalGroupId());
List *remoteNodes = TargetWorkerSetNodeList(ALL_SHARD_NODES, RowShareLock);
WorkerNode *remoteNode = NULL;
foreach_declared_ptr(remoteNode, remoteNodes)
{
InsertCleanupRecordOutsideTransaction(
CLEANUP_OBJECT_DATABASE,
pstrdup(quote_identifier(tempDatabaseName)),
remoteNode->groupId,
CLEANUP_ON_FAILURE
);
}
CreateDatabaseCommandOriginalDbName = stmt->dbname;
stmt->dbname = tempDatabaseName;
/*
* Delete cleanup records in the same transaction so that if the current
* transactions fails for some reason, then the cleanup records won't be
* deleted. In the happy path, we will delete the cleanup records without
* deferring them to the background worker.
*/
FinalizeOperationNeedingCleanupOnSuccess("create database");
return NIL;
}
/*
* PostprocessCreateDatabaseStmt is executed after the statement is applied to the local
* postgres instance.
*
* In this stage, we first rename the temporary database back to its original name for
* local node and then return a list of distributed DDL jobs to create the database with
* the temporary name and then to rename it back to its original name. That way, if CREATE
* DATABASE fails on any of the nodes, the temporary database will be cleaned up by the
* cleanup records that we inserted in PreprocessCreateDatabaseStmt() and in case of a
* failure, we won't leak any databases called as the name that user intended to use for
* the database.
*/
List *
PostprocessCreateDatabaseStmt(Node *node, const char *queryString)
{
if (!EnableCreateDatabasePropagation || !ShouldPropagate())
{
return NIL;
}
EnsurePropagationToCoordinator();
/*
* Given that CREATE DATABASE doesn't support "IF NOT EXISTS" and we're
* in the post-process, database must exist, hence missingOk = false.
*/
bool missingOk = false;
bool isPostProcess = true;
List *addresses = GetObjectAddressListFromParseTree(node, missingOk,
isPostProcess);
EnsureAllObjectDependenciesExistOnAllNodes(addresses);
char *createDatabaseCommand = DeparseTreeNode(node);
List *createDatabaseCommands = list_make3(DISABLE_DDL_PROPAGATION,
(void *) createDatabaseCommand,
ENABLE_DDL_PROPAGATION);
/*
* Since the CREATE DATABASE statements cannot be executed in a transaction
* block, we need to use NontransactionalNodeDDLTaskList() to send the CREATE
* DATABASE statement to the workers.
*/
bool warnForPartialFailure = false;
List *createDatabaseDDLJobList =
NontransactionalNodeDDLTaskList(REMOTE_NODES, createDatabaseCommands,
warnForPartialFailure);
CreatedbStmt *stmt = castNode(CreatedbStmt, node);
char *renameDatabaseCommand =
psprintf("ALTER DATABASE %s RENAME TO %s",
quote_identifier(stmt->dbname),
quote_identifier(CreateDatabaseCommandOriginalDbName));
List *renameDatabaseCommands = list_make3(DISABLE_DDL_PROPAGATION,
renameDatabaseCommand,
ENABLE_DDL_PROPAGATION);
/*
* We use NodeDDLTaskList() to send the RENAME DATABASE statement to the
* workers because we want to execute it in a coordinated transaction.
*/
List *renameDatabaseDDLJobList =
NodeDDLTaskList(REMOTE_NODES, renameDatabaseCommands);
/*
* Temporarily disable citus.enable_ddl_propagation before issuing
* rename command locally because we don't want to execute it on remote
* nodes yet. We will execute it on remote nodes by returning it as a
* distributed DDL job.
*
* The reason why we don't want to execute it on remote nodes yet is that
* the database is not created on remote nodes yet.
*/
int saveNestLevel = NewGUCNestLevel();
set_config_option("citus.enable_ddl_propagation", "off",
(superuser() ? PGC_SUSET : PGC_USERSET), PGC_S_SESSION,
GUC_ACTION_LOCAL, true, 0, false);
ExecuteUtilityCommand(renameDatabaseCommand);
AtEOXact_GUC(true, saveNestLevel);
/*
* Restore the original database name because MarkObjectDistributed()
* resolves oid of the object based on the database name and is called
* after executing the distributed DDL job that renames temporary database.
*/
stmt->dbname = CreateDatabaseCommandOriginalDbName;
return list_concat(createDatabaseDDLJobList, renameDatabaseDDLJobList);
}
/*
* PreprocessDropDatabaseStmt is executed before the statement is applied to the local
* postgres instance. In this stage we can prepare the commands that need to be run on
* all workers to drop the database.
*
* We also serialize database commands globally by acquiring a Citus specific advisory
* lock based on OCLASS_DATABASE on the first primary worker node.
*/
List *
PreprocessDropDatabaseStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
if (!EnableCreateDatabasePropagation || !ShouldPropagate())
{
return NIL;
}
EnsurePropagationToCoordinator();
DropdbStmt *stmt = (DropdbStmt *) node;
bool isPostProcess = false;
List *addresses = GetObjectAddressListFromParseTree(node, stmt->missing_ok,
isPostProcess);
if (list_length(addresses) != 1)
{
ereport(ERROR, (errmsg("unexpected number of objects found when "
"executing DROP DATABASE command")));
}
ObjectAddress *address = (ObjectAddress *) linitial(addresses);
if (address->objectId == InvalidOid || !IsAnyObjectDistributed(list_make1(address)))
{
return NIL;
}
SerializeDistributedDDLsOnObjectClassObject(OCLASS_DATABASE, stmt->dbname);
char *dropDatabaseCommand = DeparseTreeNode(node);
List *dropDatabaseCommands = list_make3(DISABLE_DDL_PROPAGATION,
(void *) dropDatabaseCommand,
ENABLE_DDL_PROPAGATION);
/*
* Due to same reason stated in PostprocessCreateDatabaseStmt(), we need to
* use NontransactionalNodeDDLTaskList() to send the DROP DATABASE statement
* to the workers.
*/
bool warnForPartialFailure = true;
List *dropDatabaseDDLJobList =
NontransactionalNodeDDLTaskList(REMOTE_NODES, dropDatabaseCommands,
warnForPartialFailure);
return dropDatabaseDDLJobList;
}
/*
* DropDatabaseStmtObjectAddress gets the ObjectAddress of the database that is the
* object of the DropdbStmt.
*/
List *
DropDatabaseStmtObjectAddress(Node *node, bool missingOk, bool isPostprocess)
{
DropdbStmt *stmt = castNode(DropdbStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
return list_make1(dbAddress);
}
/*
* CreateDatabaseStmtObjectAddress gets the ObjectAddress of the database that is the
* object of the CreatedbStmt.
*/
List *
CreateDatabaseStmtObjectAddress(Node *node, bool missingOk, bool isPostprocess)
{
CreatedbStmt *stmt = castNode(CreatedbStmt, node);
ObjectAddress *dbAddress = GetDatabaseAddressFromDatabaseName(stmt->dbname,
missingOk);
return list_make1(dbAddress);
}
/*
* EnsureSupportedCreateDatabaseCommand validates the options provided for the CREATE
* DATABASE command.
*
* Parameters:
* stmt: A CreatedbStmt struct representing a CREATE DATABASE command.
* The options field is a list of DefElem structs, each representing an option.
*
* Currently, this function checks for the following:
* - The "oid" option is not supported.
* - The "template" option is only supported with the value "template1".
* - The "strategy" option is only supported with the value "wal_log".
*/
void
EnsureSupportedCreateDatabaseCommand(CreatedbStmt *stmt)
{
DefElem *option = NULL;
foreach_declared_ptr(option, stmt->options)
{
if (strcmp(option->defname, "oid") == 0)
{
ereport(ERROR,
errmsg("CREATE DATABASE option \"%s\" is not supported",
option->defname));
}
char *optionValue = defGetString(option);
if (strcmp(option->defname, "template") == 0 &&
strcmp(optionValue, "template1") != 0)
{
ereport(ERROR, errmsg("Only template1 is supported as template "
"parameter for CREATE DATABASE"));
}
if (strcmp(option->defname, "strategy") == 0 &&
strcmp(optionValue, "wal_log") != 0)
{
ereport(ERROR, errmsg("Only wal_log is supported as strategy "
"parameter for CREATE DATABASE"));
}
}
}
#endif
/*
@ -763,238 +232,3 @@ GetDatabaseAddressFromDatabaseName(char *databaseName, bool missingOk)
ObjectAddressSet(*dbObjectAddress, DatabaseRelationId, databaseOid);
return dbObjectAddress;
}
/*
* GetTablespaceName gets the tablespace oid and returns the tablespace name.
*/
static char *
GetTablespaceName(Oid tablespaceOid)
{
HeapTuple tuple = SearchSysCache1(TABLESPACEOID, ObjectIdGetDatum(tablespaceOid));
if (!HeapTupleIsValid(tuple))
{
return NULL;
}
Form_pg_tablespace tablespaceForm = (Form_pg_tablespace) GETSTRUCT(tuple);
char *tablespaceName = pstrdup(NameStr(tablespaceForm->spcname));
ReleaseSysCache(tuple);
return tablespaceName;
}
/*
* GetDatabaseMetadataSyncCommands returns a list of sql statements
* for the given database id. The list contains the database ddl command,
* grant commands and comment propagation commands.
*/
List *
GetDatabaseMetadataSyncCommands(Oid dbOid)
{
char *databaseName = get_database_name(dbOid);
char *databaseDDLCommand = CreateDatabaseDDLCommand(dbOid);
List *ddlCommands = list_make1(databaseDDLCommand);
List *grantDDLCommands = GrantOnDatabaseDDLCommands(dbOid);
List *commentDDLCommands = GetCommentPropagationCommands(DatabaseRelationId, dbOid,
databaseName,
OBJECT_DATABASE);
ddlCommands = list_concat(ddlCommands, grantDDLCommands);
ddlCommands = list_concat(ddlCommands, commentDDLCommands);
return ddlCommands;
}
/*
* GetDatabaseCollation gets oid of a database and returns all the collation related information
* We need this method since collation related info in Form_pg_database is not accessible.
*/
static DatabaseCollationInfo
GetDatabaseCollation(Oid dbOid)
{
DatabaseCollationInfo info;
memset(&info, 0, sizeof(DatabaseCollationInfo));
Relation rel = table_open(DatabaseRelationId, AccessShareLock);
HeapTuple tup = get_catalog_object_by_oid(rel, Anum_pg_database_oid, dbOid);
if (!HeapTupleIsValid(tup))
{
elog(ERROR, "cache lookup failed for database %u", dbOid);
}
bool isNull = false;
TupleDesc tupdesc = RelationGetDescr(rel);
Datum collationDatum = heap_getattr(tup, Anum_pg_database_datcollate, tupdesc,
&isNull);
info.datcollate = TextDatumGetCString(collationDatum);
Datum ctypeDatum = heap_getattr(tup, Anum_pg_database_datctype, tupdesc, &isNull);
info.datctype = TextDatumGetCString(ctypeDatum);
Datum icuLocaleDatum = heap_getattr(tup, Anum_pg_database_datlocale, tupdesc,
&isNull);
if (!isNull)
{
info.daticulocale = TextDatumGetCString(icuLocaleDatum);
}
Datum collverDatum = heap_getattr(tup, Anum_pg_database_datcollversion, tupdesc,
&isNull);
if (!isNull)
{
info.datcollversion = TextDatumGetCString(collverDatum);
}
#if PG_VERSION_NUM >= PG_VERSION_16
Datum icurulesDatum = heap_getattr(tup, Anum_pg_database_daticurules, tupdesc,
&isNull);
if (!isNull)
{
info.daticurules = TextDatumGetCString(icurulesDatum);
}
#endif
table_close(rel, AccessShareLock);
heap_freetuple(tup);
return info;
}
/*
* GetLocaleProviderString gets the datlocprovider stored in pg_database
* and returns the string representation of the datlocprovider
*/
static char *
GetLocaleProviderString(char datlocprovider)
{
switch (datlocprovider)
{
case 'c':
{
return "libc";
}
case 'i':
{
return "icu";
}
default:
{
ereport(ERROR, (errmsg("unexpected datlocprovider value: %c",
datlocprovider)));
}
}
}
/*
* GenerateCreateDatabaseStatementFromPgDatabase gets the pg_database tuple and returns the
* CREATE DATABASE statement that can be used to create given database.
*
* Note that this doesn't deparse OID of the database and this is not a
* problem as we anyway don't allow specifying custom OIDs for databases
* when creating them.
*/
static char *
GenerateCreateDatabaseStatementFromPgDatabase(Form_pg_database databaseForm)
{
DatabaseCollationInfo collInfo = GetDatabaseCollation(databaseForm->oid);
StringInfoData str;
initStringInfo(&str);
appendStringInfo(&str, "CREATE DATABASE %s",
quote_identifier(NameStr(databaseForm->datname)));
appendStringInfo(&str, " CONNECTION LIMIT %d", databaseForm->datconnlimit);
appendStringInfo(&str, " ALLOW_CONNECTIONS = %s",
quote_literal_cstr(databaseForm->datallowconn ? "true" : "false"));
appendStringInfo(&str, " IS_TEMPLATE = %s",
quote_literal_cstr(databaseForm->datistemplate ? "true" : "false"));
appendStringInfo(&str, " LC_COLLATE = %s",
quote_literal_cstr(collInfo.datcollate));
appendStringInfo(&str, " LC_CTYPE = %s", quote_literal_cstr(collInfo.datctype));
appendStringInfo(&str, " OWNER = %s",
quote_identifier(GetUserNameFromId(databaseForm->datdba, false)));
appendStringInfo(&str, " TABLESPACE = %s",
quote_identifier(GetTablespaceName(databaseForm->dattablespace)));
appendStringInfo(&str, " ENCODING = %s",
quote_literal_cstr(pg_encoding_to_char(databaseForm->encoding)));
if (collInfo.datcollversion != NULL)
{
appendStringInfo(&str, " COLLATION_VERSION = %s",
quote_identifier(collInfo.datcollversion));
}
if (collInfo.daticulocale != NULL)
{
appendStringInfo(&str, " ICU_LOCALE = %s", quote_identifier(
collInfo.daticulocale));
}
appendStringInfo(&str, " LOCALE_PROVIDER = %s",
quote_identifier(GetLocaleProviderString(
databaseForm->datlocprovider)));
#if PG_VERSION_NUM >= PG_VERSION_16
if (collInfo.daticurules != NULL)
{
appendStringInfo(&str, " ICU_RULES = %s", quote_identifier(
collInfo.daticurules));
}
#endif
return str.data;
}
/*
* CreateDatabaseDDLCommand returns a CREATE DATABASE command to create given
* database
*
* Command is wrapped by citus_internal_database_command() UDF
* to avoid from transaction block restrictions that apply to database commands.
*/
char *
CreateDatabaseDDLCommand(Oid dbId)
{
HeapTuple tuple = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(dbId));
if (!HeapTupleIsValid(tuple))
{
ereport(ERROR, (errcode(ERRCODE_UNDEFINED_DATABASE),
errmsg("database with OID %u does not exist", dbId)));
}
Form_pg_database databaseForm = (Form_pg_database) GETSTRUCT(tuple);
char *createStmt = GenerateCreateDatabaseStatementFromPgDatabase(databaseForm);
StringInfo outerDbStmt = makeStringInfo();
/* Generate the CREATE DATABASE statement */
appendStringInfo(outerDbStmt,
"SELECT citus_internal.database_command(%s)",
quote_literal_cstr(createStmt));
ReleaseSysCache(tuple);
return outerDbStmt->data;
}

View File

@ -31,146 +31,53 @@
#include "distributed/worker_manager.h"
#include "distributed/worker_transaction.h"
typedef enum RequiredObjectSet
{
REQUIRE_ONLY_DEPENDENCIES = 1,
REQUIRE_OBJECT_AND_DEPENDENCIES = 2,
} RequiredObjectSet;
static void EnsureDependenciesCanBeDistributed(const ObjectAddress *relationAddress);
static void ErrorIfCircularDependencyExists(const ObjectAddress *objectAddress);
static int ObjectAddressComparator(const void *a, const void *b);
static void EnsureDependenciesExistOnAllNodes(const ObjectAddress *target);
static void EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
RequiredObjectSet requiredObjectSet);
static List * GetDependencyCreateDDLCommands(const ObjectAddress *dependency);
static bool ShouldPropagateObject(const ObjectAddress *address);
static char * DropTableIfExistsCommand(Oid relationId);
/*
* EnsureObjectAndDependenciesExistOnAllNodes is a wrapper around
* EnsureRequiredObjectSetExistOnAllNodes to ensure the "object itself" (together
* with its dependencies) is available on all nodes.
*
* Different than EnsureDependenciesExistOnAllNodes, we return early if the
* target object is distributed already.
*
* The reason why we don't do the same in EnsureDependenciesExistOnAllNodes
* is that it's is used when altering an object too and hence the target object
* may instantly have a dependency that needs to be propagated now. For example,
* when "GRANT non_dist_role TO dist_role" is executed, we need to propagate
* "non_dist_role" to all nodes before propagating the "GRANT" command itself.
* For this reason, we call EnsureDependenciesExistOnAllNodes for "dist_role"
* and it would automatically discover that "non_dist_role" is a dependency of
* "dist_role" and propagate it beforehand.
*
* However, when we're requested to create the target object itself (and
* implicitly its dependencies), we're sure that we're not altering the target
* object itself, hence we can return early if the target object is already
* distributed. This is the case, for example, when
* "REASSIGN OWNED BY dist_role TO non_dist_role" is executed. In that case,
* "non_dist_role" is not a dependency of "dist_role" but we want to distribute
* "non_dist_role" beforehand and we call this function for "non_dist_role",
* not for "dist_role".
*
* See EnsureRequiredObjectExistOnAllNodes to learn more about how this
* function deals with an object created within the same transaction.
*/
void
EnsureObjectAndDependenciesExistOnAllNodes(const ObjectAddress *target)
{
if (IsAnyObjectDistributed(list_make1((ObjectAddress *) target)))
{
return;
}
EnsureRequiredObjectSetExistOnAllNodes(target, REQUIRE_OBJECT_AND_DEPENDENCIES);
}
/*
* EnsureDependenciesExistOnAllNodes is a wrapper around
* EnsureRequiredObjectSetExistOnAllNodes to ensure "all dependencies" of given
* object --but not the object itself-- are available on all nodes.
*
* See EnsureRequiredObjectSetExistOnAllNodes to learn more about how this
* function deals with an object created within the same transaction.
*/
static void
EnsureDependenciesExistOnAllNodes(const ObjectAddress *target)
{
EnsureRequiredObjectSetExistOnAllNodes(target, REQUIRE_ONLY_DEPENDENCIES);
}
/*
* EnsureRequiredObjectSetExistOnAllNodes finds all the dependencies that we support and makes
* sure these are available on all nodes if required object set is REQUIRE_ONLY_DEPENDENCIES.
* Otherwise, i.e., if required object set is REQUIRE_OBJECT_AND_DEPENDENCIES, then this
* function creates the object itself on all nodes too. This function ensures that each
* of the dependencies are supported by Citus but doesn't check the same for the target
* object itself (when REQUIRE_OBJECT_AND_DEPENDENCIES) is provided because we assume that
* callers don't call this function for an unsupported function at all.
*
* If not available, they will be created on the nodes via a separate session that will be
* committed directly so that the objects are visible to potentially multiple sessions creating
* the shards.
* EnsureDependenciesExistOnAllNodes finds all the dependencies that we support and makes
* sure these are available on all workers. If not available they will be created on the
* workers via a separate session that will be committed directly so that the objects are
* visible to potentially multiple sessions creating the shards.
*
* Note; only the actual objects are created via a separate session, the records to
* pg_dist_object are created in this session. As a side effect the objects could be
* created on the nodes without a catalog entry. Updates to the objects on local node
* are not propagated to the remote nodes until the record is visible on local node.
* created on the workers without a catalog entry. Updates to the objects on the coordinator
* are not propagated to the workers until the record is visible on the coordinator.
*
* This is solved by creating the dependencies in an idempotent manner, either via
* postgres native CREATE IF NOT EXISTS, or citus helper functions.
*/
static void
EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
RequiredObjectSet requiredObjectSet)
EnsureDependenciesExistOnAllNodes(const ObjectAddress *target)
{
Assert(requiredObjectSet == REQUIRE_ONLY_DEPENDENCIES ||
requiredObjectSet == REQUIRE_OBJECT_AND_DEPENDENCIES);
List *objectsWithCommands = NIL;
List *dependenciesWithCommands = NIL;
List *ddlCommands = NULL;
/*
* If there is any unsupported dependency or circular dependency exists, Citus can
* not ensure dependencies will exist on all nodes.
*
* Note that we don't check whether "target" is distributable (in case
* REQUIRE_OBJECT_AND_DEPENDENCIES is provided) because we expect callers
* to not even call this function if Citus doesn't know how to propagate
* "target" object itself.
*/
EnsureDependenciesCanBeDistributed(target);
/* collect all dependencies in creation order and get their ddl commands */
List *objectsToBeCreated = GetDependenciesForObject(target);
/*
* Append the target object to make sure that it's created after its
* dependencies are created, if requested.
*/
if (requiredObjectSet == REQUIRE_OBJECT_AND_DEPENDENCIES)
List *dependencies = GetDependenciesForObject(target);
ObjectAddress *dependency = NULL;
foreach_declared_ptr(dependency, dependencies)
{
ObjectAddress *targetCopy = palloc(sizeof(ObjectAddress));
*targetCopy = *target;
objectsToBeCreated = lappend(objectsToBeCreated, targetCopy);
}
ObjectAddress *object = NULL;
foreach_declared_ptr(object, objectsToBeCreated)
{
List *dependencyCommands = GetDependencyCreateDDLCommands(object);
List *dependencyCommands = GetDependencyCreateDDLCommands(dependency);
ddlCommands = list_concat(ddlCommands, dependencyCommands);
/* create a new list with objects that actually created commands */
/* create a new list with dependencies that actually created commands */
if (list_length(dependencyCommands) > 0)
{
objectsWithCommands = lappend(objectsWithCommands, object);
dependenciesWithCommands = lappend(dependenciesWithCommands, dependency);
}
}
if (list_length(ddlCommands) <= 0)
@ -190,31 +97,29 @@ EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
* either get it now, or get it in citus_add_node after this transaction finishes and
* the pg_dist_object record becomes visible.
*/
List *remoteNodeList = ActivePrimaryRemoteNodeList(RowShareLock);
List *workerNodeList = ActivePrimaryNonCoordinatorNodeList(RowShareLock);
/*
* Lock objects to be created explicitly to make sure same DDL command won't be sent
* Lock dependent objects explicitly to make sure same DDL command won't be sent
* multiple times from parallel sessions.
*
* Sort the objects that will be created on workers to not to have any deadlock
* Sort dependencies that will be created on workers to not to have any deadlock
* issue if different sessions are creating different objects.
*/
List *addressSortedDependencies = SortList(objectsWithCommands,
List *addressSortedDependencies = SortList(dependenciesWithCommands,
ObjectAddressComparator);
foreach_declared_ptr(object, addressSortedDependencies)
foreach_declared_ptr(dependency, addressSortedDependencies)
{
LockDatabaseObject(object->classId, object->objectId,
object->objectSubId, ExclusiveLock);
LockDatabaseObject(dependency->classId, dependency->objectId,
dependency->objectSubId, ExclusiveLock);
}
/*
* We need to propagate objects via the current user's metadata connection if
* any of the objects that we're interested in are created in the current transaction.
* Our assumption is that if we rely on an object created in the current transaction,
* then the current user, most probably, has permissions to create the target object
* as well.
*
* We need to propagate dependencies via the current user's metadata connection if
* any dependency for the target is created in the current transaction. Our assumption
* is that if we rely on a dependency created in the current transaction, then the
* current user, most probably, has permissions to create the target object as well.
* Note that, user still may not be able to create the target due to no permissions
* for any of its dependencies. But this is ok since it should be rare.
*
@ -222,25 +127,14 @@ EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
* have visibility issues since propagated dependencies would be invisible to
* the separate connection until we locally commit.
*/
List *createdObjectList = GetAllSupportedDependenciesForObject(target);
/* consider target as well if we're requested to create it too */
if (requiredObjectSet == REQUIRE_OBJECT_AND_DEPENDENCIES)
if (HasAnyDependencyInPropagatedObjects(target))
{
ObjectAddress *targetCopy = palloc(sizeof(ObjectAddress));
*targetCopy = *target;
createdObjectList = lappend(createdObjectList, targetCopy);
}
if (HasAnyObjectInPropagatedObjects(createdObjectList))
{
SendCommandListToRemoteNodesWithMetadata(ddlCommands);
SendCommandListToWorkersWithMetadata(ddlCommands);
}
else
{
WorkerNode *workerNode = NULL;
foreach_declared_ptr(workerNode, remoteNodeList)
foreach_declared_ptr(workerNode, workerNodeList)
{
const char *nodeName = workerNode->workerName;
uint32 nodePort = workerNode->workerPort;
@ -252,11 +146,11 @@ EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
}
/*
* We do this after creating the objects on remote nodes, we make sure
* that objects have been created on remote nodes before marking them
* We do this after creating the objects on the workers, we make sure
* that objects have been created on worker nodes before marking them
* distributed, so MarkObjectDistributed wouldn't fail.
*/
foreach_declared_ptr(object, objectsWithCommands)
foreach_declared_ptr(dependency, dependenciesWithCommands)
{
/*
* pg_dist_object entries must be propagated with the super user, since
@ -266,7 +160,7 @@ EnsureRequiredObjectSetExistOnAllNodes(const ObjectAddress *target,
* Only dependent object's metadata should be propagated with super user.
* Metadata of the table itself must be propagated with the current user.
*/
MarkObjectDistributedViaSuperUser(object);
MarkObjectDistributedViaSuperUser(dependency);
}
}
@ -565,29 +459,16 @@ GetDependencyCreateDDLCommands(const ObjectAddress *dependency)
case OCLASS_DATABASE:
{
/*
* For the database where Citus is installed, only propagate the ownership of the
* database, only when the feature is on.
*
* This is because this database must exist on all nodes already so we shouldn't
* need to "CREATE" it on other nodes. However, we still need to correctly reflect
* its owner on other nodes too.
*/
if (dependency->objectId == MyDatabaseId && EnableAlterDatabaseOwner)
List *databaseDDLCommands = NIL;
/* only propagate the ownership of the database when the feature is on */
if (EnableAlterDatabaseOwner)
{
return DatabaseOwnerDDLCommands(dependency);
List *ownerDDLCommands = DatabaseOwnerDDLCommands(dependency);
databaseDDLCommands = list_concat(databaseDDLCommands, ownerDDLCommands);
}
/*
* For the other databases, create the database on all nodes, only when the feature
* is on.
*/
if (dependency->objectId != MyDatabaseId && EnableCreateDatabasePropagation)
{
return GetDatabaseMetadataSyncCommands(dependency->objectId);
}
return NIL;
return databaseDDLCommands;
}
case OCLASS_PROC:

View File

@ -16,7 +16,6 @@
#include "distributed/commands.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/comment.h"
#include "distributed/deparser.h"
#include "distributed/version_compat.h"
@ -152,17 +151,6 @@ static DistributeObjectOps Any_AlterRole = {
.address = AlterRoleStmtObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Any_AlterRoleRename = {
.deparse = DeparseRenameRoleStmt,
.qualify = NULL,
.preprocess = PreprocessAlterRoleRenameStmt,
.postprocess = NULL,
.operationType = DIST_OPS_ALTER,
.address = RenameRoleStmtObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Any_AlterRoleSet = {
.deparse = DeparseAlterRoleSetStmt,
.qualify = QualifyAlterRoleSetStmt,
@ -276,17 +264,6 @@ static DistributeObjectOps Any_CreateRole = {
.address = CreateRoleStmtObjectAddress,
.markDistributed = true,
};
static DistributeObjectOps Any_ReassignOwned = {
.deparse = DeparseReassignOwnedStmt,
.qualify = NULL,
.preprocess = NULL,
.postprocess = PostprocessReassignOwnedStmt,
.operationType = DIST_OPS_ALTER,
.address = NULL,
.markDistributed = false,
};
static DistributeObjectOps Any_DropOwned = {
.deparse = DeparseDropOwnedStmt,
.qualify = NULL,
@ -305,17 +282,6 @@ static DistributeObjectOps Any_DropRole = {
.address = NULL,
.markDistributed = false,
};
static DistributeObjectOps Role_Comment = {
.deparse = DeparseCommentStmt,
.qualify = NULL,
.preprocess = PreprocessAlterDistributedObjectStmt,
.postprocess = NULL,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_ALTER,
.address = CommentObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Any_CreateForeignServer = {
.deparse = DeparseCreateForeignServerStmt,
.qualify = NULL,
@ -399,37 +365,10 @@ static DistributeObjectOps Any_Rename = {
.markDistributed = false,
};
static DistributeObjectOps Any_SecLabel = {
.deparse = NULL,
.deparse = DeparseSecLabelStmt,
.qualify = NULL,
.preprocess = NULL,
.postprocess = PostprocessAnySecLabelStmt,
.operationType = DIST_OPS_ALTER,
.address = SecLabelStmtObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Role_SecLabel = {
.deparse = DeparseRoleSecLabelStmt,
.qualify = NULL,
.preprocess = NULL,
.postprocess = PostprocessRoleSecLabelStmt,
.operationType = DIST_OPS_ALTER,
.address = SecLabelStmtObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Table_SecLabel = {
.deparse = DeparseTableSecLabelStmt,
.qualify = NULL,
.preprocess = NULL,
.postprocess = PostprocessTableOrColumnSecLabelStmt,
.operationType = DIST_OPS_ALTER,
.address = SecLabelStmtObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Column_SecLabel = {
.deparse = DeparseColumnSecLabelStmt,
.qualify = NULL,
.preprocess = NULL,
.postprocess = PostprocessTableOrColumnSecLabelStmt,
.postprocess = PostprocessSecLabelStmt,
.operationType = DIST_OPS_ALTER,
.address = SecLabelStmtObjectAddress,
.markDistributed = false,
@ -526,28 +465,7 @@ static DistributeObjectOps Database_Alter = {
.markDistributed = false,
};
static DistributeObjectOps Database_Create = {
.deparse = DeparseCreateDatabaseStmt,
.qualify = NULL,
.preprocess = PreprocessCreateDatabaseStmt,
.postprocess = PostprocessCreateDatabaseStmt,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_CREATE,
.address = CreateDatabaseStmtObjectAddress,
.markDistributed = true,
};
static DistributeObjectOps Database_Drop = {
.deparse = DeparseDropDatabaseStmt,
.qualify = NULL,
.preprocess = PreprocessDropDatabaseStmt,
.postprocess = NULL,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_DROP,
.address = DropDatabaseStmtObjectAddress,
.markDistributed = false,
};
#if PG_VERSION_NUM >= PG_VERSION_15
static DistributeObjectOps Database_RefreshColl = {
.deparse = DeparseAlterDatabaseRefreshCollStmt,
.qualify = NULL,
@ -558,39 +476,7 @@ static DistributeObjectOps Database_RefreshColl = {
.address = NULL,
.markDistributed = false,
};
static DistributeObjectOps Database_Set = {
.deparse = DeparseAlterDatabaseSetStmt,
.qualify = NULL,
.preprocess = PreprocessAlterDatabaseSetStmt,
.postprocess = NULL,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_ALTER,
.address = NULL,
.markDistributed = false,
};
static DistributeObjectOps Database_Comment = {
.deparse = DeparseCommentStmt,
.qualify = NULL,
.preprocess = PreprocessAlterDistributedObjectStmt,
.postprocess = NULL,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_ALTER,
.address = CommentObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps Database_Rename = {
.deparse = DeparseAlterDatabaseRenameStmt,
.qualify = NULL,
.preprocess = PreprocessAlterDatabaseRenameStmt,
.postprocess = PostprocessAlterDatabaseRenameStmt,
.objectType = OBJECT_DATABASE,
.operationType = DIST_OPS_ALTER,
.address = NULL,
.markDistributed = false,
};
#endif
static DistributeObjectOps Domain_Alter = {
.deparse = DeparseAlterDomainStmt,
@ -951,6 +837,7 @@ static DistributeObjectOps Sequence_AlterOwner = {
.address = AlterSequenceOwnerStmtObjectAddress,
.markDistributed = false,
};
#if (PG_VERSION_NUM >= PG_VERSION_15)
static DistributeObjectOps Sequence_AlterPersistence = {
.deparse = DeparseAlterSequencePersistenceStmt,
.qualify = QualifyAlterSequencePersistenceStmt,
@ -960,6 +847,7 @@ static DistributeObjectOps Sequence_AlterPersistence = {
.address = AlterSequencePersistenceStmtObjectAddress,
.markDistributed = false,
};
#endif
static DistributeObjectOps Sequence_Drop = {
.deparse = DeparseDropSequenceStmt,
.qualify = QualifyDropSequenceStmt,
@ -1018,18 +906,13 @@ static DistributeObjectOps TextSearchConfig_AlterOwner = {
.markDistributed = false,
};
static DistributeObjectOps TextSearchConfig_Comment = {
.deparse = DeparseCommentStmt,
/* TODO: When adding new comment types we should create an abstracted
* qualify function, just like we have an abstract deparse
* and adress function
*/
.deparse = DeparseTextSearchConfigurationCommentStmt,
.qualify = QualifyTextSearchConfigurationCommentStmt,
.preprocess = PreprocessAlterDistributedObjectStmt,
.postprocess = NULL,
.objectType = OBJECT_TSCONFIGURATION,
.operationType = DIST_OPS_ALTER,
.address = CommentObjectAddress,
.address = TextSearchConfigurationCommentObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps TextSearchConfig_Define = {
@ -1092,13 +975,13 @@ static DistributeObjectOps TextSearchDict_AlterOwner = {
.markDistributed = false,
};
static DistributeObjectOps TextSearchDict_Comment = {
.deparse = DeparseCommentStmt,
.deparse = DeparseTextSearchDictionaryCommentStmt,
.qualify = QualifyTextSearchDictionaryCommentStmt,
.preprocess = PreprocessAlterDistributedObjectStmt,
.postprocess = NULL,
.objectType = OBJECT_TSDICTIONARY,
.operationType = DIST_OPS_ALTER,
.address = CommentObjectAddress,
.address = TextSearchDictCommentObjectAddress,
.markDistributed = false,
};
static DistributeObjectOps TextSearchDict_Define = {
@ -1416,7 +1299,7 @@ static DistributeObjectOps View_Rename = {
static DistributeObjectOps Trigger_Rename = {
.deparse = NULL,
.qualify = NULL,
.preprocess = NULL,
.preprocess = PreprocessAlterTriggerRenameStmt,
.operationType = DIST_OPS_ALTER,
.postprocess = PostprocessAlterTriggerRenameStmt,
.address = NULL,
@ -1438,27 +1321,13 @@ GetDistributeObjectOps(Node *node)
return &Database_Alter;
}
case T_CreatedbStmt:
{
return &Database_Create;
}
case T_DropdbStmt:
{
return &Database_Drop;
}
#if PG_VERSION_NUM >= PG_VERSION_15
case T_AlterDatabaseRefreshCollStmt:
{
return &Database_RefreshColl;
}
case T_AlterDatabaseSetStmt:
{
return &Database_Set;
}
#endif
case T_AlterDomainStmt:
{
return &Domain_Alter;
@ -1743,6 +1612,7 @@ GetDistributeObjectOps(Node *node)
case OBJECT_SEQUENCE:
{
#if (PG_VERSION_NUM >= PG_VERSION_15)
ListCell *cmdCell = NULL;
foreach(cmdCell, stmt->cmds)
{
@ -1770,6 +1640,7 @@ GetDistributeObjectOps(Node *node)
}
}
}
#endif
/*
* Prior to PG15, the only Alter Table statement
@ -1826,16 +1697,6 @@ GetDistributeObjectOps(Node *node)
return &TextSearchDict_Comment;
}
case OBJECT_DATABASE:
{
return &Database_Comment;
}
case OBJECT_ROLE:
{
return &Role_Comment;
}
default:
{
return &NoDistributeOps;
@ -1945,11 +1806,6 @@ GetDistributeObjectOps(Node *node)
return &Any_DropOwned;
}
case T_ReassignOwnedStmt:
{
return &Any_ReassignOwned;
}
case T_DropStmt:
{
DropStmt *stmt = castNode(DropStmt, node);
@ -2146,27 +2002,7 @@ GetDistributeObjectOps(Node *node)
case T_SecLabelStmt:
{
SecLabelStmt *stmt = castNode(SecLabelStmt, node);
switch (stmt->objtype)
{
case OBJECT_ROLE:
{
return &Role_SecLabel;
}
case OBJECT_TABLE:
{
return &Table_SecLabel;
}
case OBJECT_COLUMN:
{
return &Column_SecLabel;
}
default:
return &Any_SecLabel;
}
return &Any_SecLabel;
}
case T_RenameStmt:
@ -2189,11 +2025,6 @@ GetDistributeObjectOps(Node *node)
return &Collation_Rename;
}
case OBJECT_DATABASE:
{
return &Database_Rename;
}
case OBJECT_DOMAIN:
{
return &Domain_Rename;
@ -2224,11 +2055,6 @@ GetDistributeObjectOps(Node *node)
return &Publication_Rename;
}
case OBJECT_ROLE:
{
return &Any_AlterRoleRename;
}
case OBJECT_ROUTINE:
{
return &Routine_Rename;

View File

@ -467,6 +467,7 @@ ForeignKeyGetDefaultingAttrs(HeapTuple pgConstraintTuple)
}
List *onDeleteSetDefColumnList = NIL;
#if PG_VERSION_NUM >= PG_VERSION_15
Datum onDeleteSetDefColumnsDatum = SysCacheGetAttr(CONSTROID, pgConstraintTuple,
Anum_pg_constraint_confdelsetcols,
&isNull);
@ -481,6 +482,7 @@ ForeignKeyGetDefaultingAttrs(HeapTuple pgConstraintTuple)
onDeleteSetDefColumnList =
IntegerArrayTypeToList(DatumGetArrayTypeP(onDeleteSetDefColumnsDatum));
}
#endif
if (list_length(onDeleteSetDefColumnList) == 0)
{

View File

@ -885,7 +885,6 @@ UpdateFunctionDistributionInfo(const ObjectAddress *distAddress,
char *workerPgDistObjectUpdateCommand =
MarkObjectsDistributedCreateCommand(objectAddressList,
NIL,
distArgumentIndexList,
colocationIdList,
forceDelegationList);
@ -981,6 +980,7 @@ GetAggregateDDLCommand(const RegProcedure funcOid, bool useCreateOrReplace)
char *argmodes = NULL;
int insertorderbyat = -1;
int argsprinted = 0;
int inputargno = 0;
HeapTuple proctup = SearchSysCache1(PROCOID, funcOid);
if (!HeapTupleIsValid(proctup))
@ -1060,6 +1060,7 @@ GetAggregateDDLCommand(const RegProcedure funcOid, bool useCreateOrReplace)
}
}
inputargno++; /* this is a 1-based counter */
if (argsprinted == insertorderbyat)
{
appendStringInfoString(&buf, " ORDER BY ");

View File

@ -17,7 +17,6 @@
#include "distributed/citus_ruleutils.h"
#include "distributed/commands.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/deparser.h"
#include "distributed/metadata/distobject.h"
#include "distributed/metadata_cache.h"
#include "distributed/version_compat.h"
@ -33,6 +32,7 @@ static List * CollectGrantTableIdList(GrantStmt *grantStmt);
* needed during the worker node portion of DDL execution before returning the
* DDLJobs in a List. If no distributed table is involved, this returns NIL.
*
* NB: So far column level privileges are not supported.
*/
List *
PreprocessGrantStmt(Node *node, const char *queryString,
@ -70,12 +70,9 @@ PreprocessGrantStmt(Node *node, const char *queryString,
return NIL;
}
EnsureCoordinator();
/* deparse the privileges */
if (grantStmt->privileges == NIL)
{
/* this is used for table level only */
appendStringInfo(&privsString, "ALL");
}
else
@ -91,44 +88,18 @@ PreprocessGrantStmt(Node *node, const char *queryString,
{
appendStringInfoString(&privsString, ", ");
}
if (priv->priv_name)
{
appendStringInfo(&privsString, "%s", priv->priv_name);
}
/*
* ALL can only be set alone.
* And ALL is not added as a keyword in priv_name by parser, but
* because there are column(s) defined, a grantStmt->privileges is
* defined. So we need to handle this special case here (see if
* condition above).
*/
else if (isFirst)
{
/* this is used for column level only */
appendStringInfo(&privsString, "ALL");
}
/*
* Instead of relying only on the syntax check done by Postgres and
* adding an assert here, add a default ERROR if ALL is not first
* and no priv_name is defined.
*/
else
{
ereport(ERROR, (errcode(ERRCODE_INTERNAL_ERROR),
errmsg("Cannot parse GRANT/REVOKE privileges")));
}
isFirst = false;
if (priv->cols != NIL)
{
StringInfoData colsString;
initStringInfo(&colsString);
AppendColumnNameList(&colsString, priv->cols);
appendStringInfo(&privsString, "%s", colsString.data);
ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("grant/revoke on column list is currently "
"unsupported")));
}
Assert(priv->priv_name != NULL);
appendStringInfo(&privsString, "%s", priv->priv_name);
}
}
@ -182,15 +153,6 @@ PreprocessGrantStmt(Node *node, const char *queryString,
appendStringInfo(&ddlString, "REVOKE %s%s ON %s FROM %s",
grantOption, privsString.data, targetString.data,
granteesString.data);
if (grantStmt->behavior == DROP_CASCADE)
{
appendStringInfoString(&ddlString, " CASCADE");
}
else
{
appendStringInfoString(&ddlString, " RESTRICT");
}
}
DDLJob *ddlJob = palloc0(sizeof(DDLJob));

View File

@ -10,8 +10,6 @@
#include "postgres.h"
#include "miscadmin.h"
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/xact.h"
@ -19,6 +17,13 @@
#include "catalog/index.h"
#include "catalog/namespace.h"
#include "catalog/pg_class.h"
#include "pg_version_constants.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "catalog/pg_namespace.h"
#endif
#include "miscadmin.h"
#include "commands/defrem.h"
#include "commands/tablecmds.h"
#include "lib/stringinfo.h"
@ -31,8 +36,6 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "pg_version_constants.h"
#include "distributed/citus_ruleutils.h"
#include "distributed/commands.h"
#include "distributed/commands/utility_hook.h"
@ -53,10 +56,6 @@
#include "distributed/version_compat.h"
#include "distributed/worker_manager.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "catalog/pg_namespace.h"
#endif
/* Local functions forward declarations for helper functions */
static void ErrorIfCreateIndexHasTooManyColumns(IndexStmt *createIndexStatement);
@ -184,8 +183,6 @@ PreprocessIndexStmt(Node *node, const char *createIndexCommand,
return NIL;
}
EnsureCoordinator();
if (createIndexStatement->idxname == NULL)
{
/*
@ -493,7 +490,6 @@ GenerateCreateIndexDDLJob(IndexStmt *createIndexStatement, const char *createInd
ddlJob->startNewTransaction = createIndexStatement->concurrent;
ddlJob->metadataSyncCommand = createIndexCommand;
ddlJob->taskList = CreateIndexTaskList(createIndexStatement);
ddlJob->warnForPartialFailure = true;
return ddlJob;
}
@ -653,7 +649,6 @@ PreprocessReindexStmt(Node *node, const char *reindexCommand,
"concurrently");
ddlJob->metadataSyncCommand = reindexCommand;
ddlJob->taskList = CreateReindexTaskList(relationId, reindexStatement);
ddlJob->warnForPartialFailure = true;
ddlJobs = list_make1(ddlJob);
}
@ -782,7 +777,6 @@ PreprocessDropIndexStmt(Node *node, const char *dropIndexCommand,
ddlJob->metadataSyncCommand = dropIndexCommand;
ddlJob->taskList = DropIndexTaskList(distributedRelationId, distributedIndexId,
dropIndexStatement);
ddlJob->warnForPartialFailure = true;
ddlJobs = list_make1(ddlJob);
}
@ -947,7 +941,7 @@ CreateIndexTaskList(IndexStmt *indexStmt)
task->dependentTaskList = NULL;
task->anchorShardId = shardId;
task->taskPlacementList = ActiveShardPlacementList(shardId);
task->cannotBeExecutedInTransaction = indexStmt->concurrent;
task->cannotBeExecutedInTransction = indexStmt->concurrent;
taskList = lappend(taskList, task);
@ -992,7 +986,7 @@ CreateReindexTaskList(Oid relationId, ReindexStmt *reindexStmt)
task->dependentTaskList = NULL;
task->anchorShardId = shardId;
task->taskPlacementList = ActiveShardPlacementList(shardId);
task->cannotBeExecutedInTransaction =
task->cannotBeExecutedInTransction =
IsReindexWithParam_compat(reindexStmt, "concurrently");
taskList = lappend(taskList, task);
@ -1330,7 +1324,7 @@ DropIndexTaskList(Oid relationId, Oid indexId, DropStmt *dropStmt)
task->dependentTaskList = NULL;
task->anchorShardId = shardId;
task->taskPlacementList = ActiveShardPlacementList(shardId);
task->cannotBeExecutedInTransaction = dropStmt->concurrent;
task->cannotBeExecutedInTransction = dropStmt->concurrent;
taskList = lappend(taskList, task);

View File

@ -64,6 +64,28 @@
#include "commands/copy.h"
#include "commands/defrem.h"
#include "commands/progress.h"
#include "pg_version_constants.h"
#include "distributed/citus_safe_lib.h"
#include "distributed/commands/multi_copy.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/coordinator_protocol.h"
#include "distributed/intermediate_results.h"
#include "distributed/listutils.h"
#include "distributed/local_executor.h"
#include "distributed/locally_reserved_shared_connections.h"
#include "distributed/log_utils.h"
#include "distributed/metadata_cache.h"
#include "distributed/multi_executor.h"
#include "distributed/multi_partitioning_utils.h"
#include "distributed/multi_physical_planner.h"
#include "distributed/multi_router_planner.h"
#include "distributed/placement_connection.h"
#include "distributed/relation_access_tracking.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "distributed/relation_utils.h"
#endif
#include "executor/executor.h"
#include "foreign/foreign.h"
#include "libpq/libpq.h"
@ -80,41 +102,18 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "pg_version_constants.h"
#include "distributed/citus_safe_lib.h"
#include "distributed/commands/multi_copy.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/coordinator_protocol.h"
#include "distributed/hash_helpers.h"
#include "distributed/intermediate_results.h"
#include "distributed/listutils.h"
#include "distributed/local_executor.h"
#include "distributed/local_multi_copy.h"
#include "distributed/locally_reserved_shared_connections.h"
#include "distributed/log_utils.h"
#include "distributed/metadata_cache.h"
#include "distributed/multi_executor.h"
#include "distributed/multi_partitioning_utils.h"
#include "distributed/multi_physical_planner.h"
#include "distributed/multi_router_planner.h"
#include "distributed/placement_connection.h"
#include "distributed/relation_access_tracking.h"
#include "distributed/remote_commands.h"
#include "distributed/remote_transaction.h"
#include "distributed/replication_origin_session_utils.h"
#include "distributed/resource_lock.h"
#include "distributed/shard_pruning.h"
#include "distributed/shared_connection_stats.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/transmit.h"
#include "distributed/version_compat.h"
#include "distributed/worker_protocol.h"
#if PG_VERSION_NUM >= PG_VERSION_16
#include "distributed/relation_utils.h"
#endif
/* constant used in binary protocol */
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
@ -500,14 +499,10 @@ CopyToExistingShards(CopyStmt *copyStatement, QueryCompletion *completionTag)
/* set up the destination for the COPY */
const bool publishableData = true;
/* we want to track query counters for "COPY (to) distributed-table .." commands */
const bool trackQueryCounters = true;
CitusCopyDestReceiver *copyDest = CreateCitusCopyDestReceiver(tableId, columnNameList,
partitionColumnIndex,
executorState, NULL,
publishableData,
trackQueryCounters);
publishableData);
/* if the user specified an explicit append-to_shard option, write to it */
uint64 appendShardId = ProcessAppendToShardOption(tableId, copyStatement);
@ -1882,15 +1877,11 @@ CopyFlushOutput(CopyOutState cstate, char *start, char *pointer)
* of intermediate results that are co-located with the actual table.
* The names of the intermediate results with be of the form:
* intermediateResultIdPrefix_<shardid>
*
* If trackQueryCounters is true, the COPY will increment the query stat
* counters as needed at the end of the COPY.
*/
CitusCopyDestReceiver *
CreateCitusCopyDestReceiver(Oid tableId, List *columnNameList, int partitionColumnIndex,
EState *executorState,
char *intermediateResultIdPrefix, bool isPublishable,
bool trackQueryCounters)
char *intermediateResultIdPrefix, bool isPublishable)
{
CitusCopyDestReceiver *copyDest = (CitusCopyDestReceiver *) palloc0(
sizeof(CitusCopyDestReceiver));
@ -1910,7 +1901,6 @@ CreateCitusCopyDestReceiver(Oid tableId, List *columnNameList, int partitionColu
copyDest->colocatedIntermediateResultIdPrefix = intermediateResultIdPrefix;
copyDest->memoryContext = CurrentMemoryContext;
copyDest->isPublishable = isPublishable;
copyDest->trackQueryCounters = trackQueryCounters;
return copyDest;
}
@ -2597,9 +2587,8 @@ ShardIdForTuple(CitusCopyDestReceiver *copyDest, Datum *columnValues, bool *colu
/*
* CitusCopyDestReceiverShutdown implements the rShutdown interface of
* CitusCopyDestReceiver. It ends the COPY on all the open connections, closes
* the relation and increments the query stat counters based on the shards
* copied into if requested.
* CitusCopyDestReceiver. It ends the COPY on all the open connections and closes
* the relation.
*/
static void
CitusCopyDestReceiverShutdown(DestReceiver *destReceiver)
@ -2610,26 +2599,6 @@ CitusCopyDestReceiverShutdown(DestReceiver *destReceiver)
ListCell *connectionStateCell = NULL;
Relation distributedRelation = copyDest->distributedRelation;
/*
* Increment the query stat counters based on the shards copied into
* if requested.
*/
if (copyDest->trackQueryCounters)
{
int copiedShardCount =
copyDest->shardStateHash ?
hash_get_num_entries(copyDest->shardStateHash) :
0;
if (copiedShardCount <= 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
}
List *connectionStateList = ConnectionStateList(connectionStateHash);
FinishLocalColocatedIntermediateFiles(copyDest);
@ -2696,6 +2665,7 @@ CreateLocalColocatedIntermediateFile(CitusCopyDestReceiver *copyDest,
CreateIntermediateResultsDirectory();
const int fileFlags = (O_CREAT | O_RDWR | O_TRUNC);
const int fileMode = (S_IRUSR | S_IWUSR);
StringInfo filePath = makeStringInfo();
appendStringInfo(filePath, "%s_%ld", copyDest->colocatedIntermediateResultIdPrefix,
@ -2703,7 +2673,7 @@ CreateLocalColocatedIntermediateFile(CitusCopyDestReceiver *copyDest,
const char *fileName = QueryResultFileName(filePath->data);
shardState->fileDest =
FileCompatFromFileStart(FileOpenForTransmit(fileName, fileFlags));
FileCompatFromFileStart(FileOpenForTransmit(fileName, fileFlags, fileMode));
CopyOutState localFileCopyOutState = shardState->copyOutState;
bool isBinaryCopy = localFileCopyOutState->binary;
@ -3172,15 +3142,6 @@ CitusCopyTo(CopyStmt *copyStatement, QueryCompletion *completionTag)
SendCopyEnd(copyOutState);
if (list_length(shardIntervalList) <= 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
table_close(distributedRelation, AccessShareLock);
if (completionTag != NULL)

View File

@ -1,351 +0,0 @@
/*-------------------------------------------------------------------------
*
* non_main_db_distribute_object_ops.c
*
* Routines to support node-wide object management commands from non-main
* databases.
*
* RunPreprocessNonMainDBCommand and RunPostprocessNonMainDBCommand are
* the entrypoints for this module. These functions are called from
* utility_hook.c to support some of the node-wide object management
* commands from non-main databases.
*
* To add support for a new command type, one needs to define a new
* NonMainDbDistributeObjectOps object within OperationArray. Also, if
* the command requires marking or unmarking some objects as distributed,
* the necessary operations can be implemented in
* RunPreprocessNonMainDBCommand and RunPostprocessNonMainDBCommand.
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/xact.h"
#include "catalog/pg_authid_d.h"
#include "nodes/nodes.h"
#include "nodes/parsenodes.h"
#include "utils/builtins.h"
#include "distributed/commands.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#include "distributed/metadata_cache.h"
#include "distributed/remote_transaction.h"
#define EXECUTE_COMMAND_ON_REMOTE_NODES_AS_USER \
"SELECT citus_internal.execute_command_on_remote_nodes_as_user(%s, %s)"
#define START_MANAGEMENT_TRANSACTION \
"SELECT citus_internal.start_management_transaction('%lu')"
#define MARK_OBJECT_DISTRIBUTED \
"SELECT citus_internal.mark_object_distributed(%d, %s, %d, %s)"
#define UNMARK_OBJECT_DISTRIBUTED \
"SELECT pg_catalog.citus_unmark_object_distributed(%d, %d, %d, %s)"
/*
* NonMainDbDistributeObjectOps contains the necessary callbacks / flags to
* support node-wide object management commands from non-main databases.
*
* cannotBeExecutedInTransaction:
* Indicates whether the statement cannot be executed in a transaction. If
* this is set to true, the statement will be executed directly on the main
* database because there are no transactional visibility issues for such
* commands.
*
* checkSupportedObjectType:
* Callback function that checks whether type of the object referred to by
* given statement is supported. Can be NULL if not applicable for the
* statement type.
*/
typedef struct NonMainDbDistributeObjectOps
{
bool cannotBeExecutedInTransaction;
bool (*checkSupportedObjectType)(Node *parsetree);
} NonMainDbDistributeObjectOps;
/*
* checkSupportedObjectType callbacks for OperationArray.
*/
static bool CreateDbStmtCheckSupportedObjectType(Node *node);
static bool DropDbStmtCheckSupportedObjectType(Node *node);
static bool GrantStmtCheckSupportedObjectType(Node *node);
static bool SecLabelStmtCheckSupportedObjectType(Node *node);
/*
* OperationArray that holds NonMainDbDistributeObjectOps for different command types.
*/
static const NonMainDbDistributeObjectOps *const OperationArray[] = {
[T_CreateRoleStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = NULL
},
[T_DropRoleStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = NULL
},
[T_AlterRoleStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = NULL
},
[T_GrantRoleStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = NULL
},
[T_CreatedbStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = true,
.checkSupportedObjectType = CreateDbStmtCheckSupportedObjectType
},
[T_DropdbStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = true,
.checkSupportedObjectType = DropDbStmtCheckSupportedObjectType
},
[T_GrantStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = GrantStmtCheckSupportedObjectType
},
[T_SecLabelStmt] = &(NonMainDbDistributeObjectOps) {
.cannotBeExecutedInTransaction = false,
.checkSupportedObjectType = SecLabelStmtCheckSupportedObjectType
},
};
/* other static function declarations */
const NonMainDbDistributeObjectOps * GetNonMainDbDistributeObjectOps(Node *parsetree);
static void CreateRoleStmtMarkDistGloballyOnMainDbs(CreateRoleStmt *createRoleStmt);
static void DropRoleStmtUnmarkDistOnLocalMainDb(DropRoleStmt *dropRoleStmt);
static void MarkObjectDistributedGloballyOnMainDbs(Oid catalogRelId, Oid objectId,
char *objectName);
static void UnmarkObjectDistributedOnLocalMainDb(uint16 catalogRelId, Oid objectId);
/*
* RunPreprocessNonMainDBCommand runs the necessary commands for a query, in main
* database before query is run on the local node with PrevProcessUtility.
*
* Returns true if previous utility hook needs to be skipped after completing
* preprocess phase.
*/
bool
RunPreprocessNonMainDBCommand(Node *parsetree)
{
if (IsMainDB)
{
return false;
}
const NonMainDbDistributeObjectOps *ops = GetNonMainDbDistributeObjectOps(parsetree);
if (!ops)
{
return false;
}
char *queryString = DeparseTreeNode(parsetree);
/*
* For the commands that cannot be executed in a transaction, there are no
* transactional visibility issues. We directly route them to main database
* so that we only have to consider one code-path for such commands.
*/
if (ops->cannotBeExecutedInTransaction)
{
IsMainDBCommandInXact = false;
RunCitusMainDBQuery((char *) queryString);
return true;
}
IsMainDBCommandInXact = true;
StringInfo mainDBQuery = makeStringInfo();
appendStringInfo(mainDBQuery,
START_MANAGEMENT_TRANSACTION,
GetCurrentFullTransactionId().value);
RunCitusMainDBQuery(mainDBQuery->data);
mainDBQuery = makeStringInfo();
appendStringInfo(mainDBQuery,
EXECUTE_COMMAND_ON_REMOTE_NODES_AS_USER,
quote_literal_cstr(queryString),
quote_literal_cstr(CurrentUserName()));
RunCitusMainDBQuery(mainDBQuery->data);
if (IsA(parsetree, DropRoleStmt))
{
DropRoleStmtUnmarkDistOnLocalMainDb((DropRoleStmt *) parsetree);
}
return false;
}
/*
* RunPostprocessNonMainDBCommand runs the necessary commands for a query, in main
* database after query is run on the local node with PrevProcessUtility.
*/
void
RunPostprocessNonMainDBCommand(Node *parsetree)
{
if (IsMainDB || !GetNonMainDbDistributeObjectOps(parsetree))
{
return;
}
if (IsA(parsetree, CreateRoleStmt))
{
CreateRoleStmtMarkDistGloballyOnMainDbs((CreateRoleStmt *) parsetree);
}
}
/*
* GetNonMainDbDistributeObjectOps returns the NonMainDbDistributeObjectOps for given
* command if it's node-wide object management command that's supported from non-main
* databases.
*/
const NonMainDbDistributeObjectOps *
GetNonMainDbDistributeObjectOps(Node *parsetree)
{
NodeTag tag = nodeTag(parsetree);
if (tag >= lengthof(OperationArray))
{
return NULL;
}
const NonMainDbDistributeObjectOps *ops = OperationArray[tag];
if (ops == NULL)
{
return NULL;
}
if (!ops->checkSupportedObjectType ||
ops->checkSupportedObjectType(parsetree))
{
return ops;
}
return NULL;
}
/*
* CreateRoleStmtMarkDistGloballyOnMainDbs marks the role as
* distributed on all main databases globally.
*/
static void
CreateRoleStmtMarkDistGloballyOnMainDbs(CreateRoleStmt *createRoleStmt)
{
/* object must exist as we've just created it */
bool missingOk = false;
Oid roleId = get_role_oid(createRoleStmt->role, missingOk);
MarkObjectDistributedGloballyOnMainDbs(AuthIdRelationId, roleId,
createRoleStmt->role);
}
/*
* DropRoleStmtUnmarkDistOnLocalMainDb unmarks the roles as
* distributed on the local main database.
*/
static void
DropRoleStmtUnmarkDistOnLocalMainDb(DropRoleStmt *dropRoleStmt)
{
RoleSpec *roleSpec = NULL;
foreach_declared_ptr(roleSpec, dropRoleStmt->roles)
{
Oid roleOid = get_role_oid(roleSpec->rolename,
dropRoleStmt->missing_ok);
if (roleOid == InvalidOid)
{
continue;
}
UnmarkObjectDistributedOnLocalMainDb(AuthIdRelationId, roleOid);
}
}
/*
* MarkObjectDistributedGloballyOnMainDbs marks an object as
* distributed on all main databases globally.
*/
static void
MarkObjectDistributedGloballyOnMainDbs(Oid catalogRelId, Oid objectId, char *objectName)
{
StringInfo mainDBQuery = makeStringInfo();
appendStringInfo(mainDBQuery,
MARK_OBJECT_DISTRIBUTED,
catalogRelId,
quote_literal_cstr(objectName),
objectId,
quote_literal_cstr(CurrentUserName()));
RunCitusMainDBQuery(mainDBQuery->data);
}
/*
* UnmarkObjectDistributedOnLocalMainDb unmarks an object as
* distributed on the local main database.
*/
static void
UnmarkObjectDistributedOnLocalMainDb(uint16 catalogRelId, Oid objectId)
{
const int subObjectId = 0;
const char *checkObjectExistence = "false";
StringInfo query = makeStringInfo();
appendStringInfo(query,
UNMARK_OBJECT_DISTRIBUTED,
catalogRelId, objectId,
subObjectId, checkObjectExistence);
RunCitusMainDBQuery(query->data);
}
/*
* checkSupportedObjectTypes callbacks for OperationArray lie below.
*/
static bool
CreateDbStmtCheckSupportedObjectType(Node *node)
{
/*
* We don't try to send the query to the main database if the CREATE
* DATABASE command is for the main database itself, this is a very
* rare case but it's exercised by our test suite.
*/
CreatedbStmt *stmt = castNode(CreatedbStmt, node);
return strcmp(stmt->dbname, MainDb) != 0;
}
static bool
DropDbStmtCheckSupportedObjectType(Node *node)
{
/*
* We don't try to send the query to the main database if the DROP
* DATABASE command is for the main database itself, this is a very
* rare case but it's exercised by our test suite.
*/
DropdbStmt *stmt = castNode(DropdbStmt, node);
return strcmp(stmt->dbname, MainDb) != 0;
}
static bool
GrantStmtCheckSupportedObjectType(Node *node)
{
GrantStmt *stmt = castNode(GrantStmt, node);
return stmt->objtype == OBJECT_DATABASE;
}
static bool
SecLabelStmtCheckSupportedObjectType(Node *node)
{
SecLabelStmt *stmt = castNode(SecLabelStmt, node);
return stmt->objtype == OBJECT_ROLE;
}

View File

@ -48,9 +48,6 @@
#include "distributed/version_compat.h"
#include "distributed/worker_transaction.h"
static ObjectAddress * GetNewRoleAddress(ReassignOwnedStmt *stmt);
/*
* PreprocessDropOwnedStmt finds the distributed role out of the ones
* being dropped and unmarks them distributed and creates the drop statements
@ -92,81 +89,3 @@ PreprocessDropOwnedStmt(Node *node, const char *queryString,
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
/*
* PostprocessReassignOwnedStmt takes a Node pointer representing a REASSIGN
* OWNED statement and performs any necessary post-processing after the statement
* has been executed locally.
*
* We filter out local roles in OWNED BY clause before deparsing the command,
* meaning that we skip reassigning what is owned by local roles. However,
* if the role specified in TO clause is local, we automatically distribute
* it before deparsing the command.
*/
List *
PostprocessReassignOwnedStmt(Node *node, const char *queryString)
{
ReassignOwnedStmt *stmt = castNode(ReassignOwnedStmt, node);
List *allReassignRoles = stmt->roles;
List *distributedReassignRoles = FilterDistributedRoles(allReassignRoles);
if (list_length(distributedReassignRoles) <= 0)
{
return NIL;
}
if (!ShouldPropagate())
{
return NIL;
}
EnsureCoordinator();
stmt->roles = distributedReassignRoles;
char *sql = DeparseTreeNode((Node *) stmt);
stmt->roles = allReassignRoles;
ObjectAddress *newRoleAddress = GetNewRoleAddress(stmt);
/*
* We temporarily enable create / alter role propagation to properly
* propagate the role specified in TO clause.
*/
int saveNestLevel = NewGUCNestLevel();
set_config_option("citus.enable_create_role_propagation", "on",
(superuser() ? PGC_SUSET : PGC_USERSET), PGC_S_SESSION,
GUC_ACTION_LOCAL, true, 0, false);
set_config_option("citus.enable_alter_role_propagation", "on",
(superuser() ? PGC_SUSET : PGC_USERSET), PGC_S_SESSION,
GUC_ACTION_LOCAL, true, 0, false);
set_config_option("citus.enable_alter_role_set_propagation", "on",
(superuser() ? PGC_SUSET : PGC_USERSET), PGC_S_SESSION,
GUC_ACTION_LOCAL, true, 0, false);
EnsureObjectAndDependenciesExistOnAllNodes(newRoleAddress);
/* rollback GUCs to the state before this session */
AtEOXact_GUC(true, saveNestLevel);
List *commands = list_make3(DISABLE_DDL_PROPAGATION,
sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
/*
* GetNewRoleAddress returns the ObjectAddress of the new role
*/
static ObjectAddress *
GetNewRoleAddress(ReassignOwnedStmt *stmt)
{
Oid roleOid = get_role_oid(stmt->newrole->rolename, false);
ObjectAddress *address = palloc0(sizeof(ObjectAddress));
ObjectAddressSet(*address, AuthIdRelationId, roleOid);
return address;
}

View File

@ -33,9 +33,11 @@
static CreatePublicationStmt * BuildCreatePublicationStmt(Oid publicationId);
#if (PG_VERSION_NUM >= PG_VERSION_15)
static PublicationObjSpec * BuildPublicationRelationObjSpec(Oid relationId,
Oid publicationId,
bool tableOnly);
#endif
static void AppendPublishOptionList(StringInfo str, List *strings);
static char * AlterPublicationOwnerCommand(Oid publicationId);
static bool ShouldPropagateCreatePublication(CreatePublicationStmt *stmt);
@ -152,6 +154,7 @@ BuildCreatePublicationStmt(Oid publicationId)
ReleaseSysCache(publicationTuple);
#if (PG_VERSION_NUM >= PG_VERSION_15)
List *schemaIds = GetPublicationSchemas(publicationId);
Oid schemaId = InvalidOid;
@ -167,18 +170,21 @@ BuildCreatePublicationStmt(Oid publicationId)
createPubStmt->pubobjects = lappend(createPubStmt->pubobjects, publicationObject);
}
#endif
List *relationIds = GetPublicationRelations(publicationId,
publicationForm->pubviaroot ?
PUBLICATION_PART_ROOT :
PUBLICATION_PART_LEAF);
Oid relationId = InvalidOid;
int citusTableCount PG_USED_FOR_ASSERTS_ONLY = 0;
/* mainly for consistent ordering in test output */
relationIds = SortList(relationIds, CompareOids);
foreach_declared_oid(relationId, relationIds)
{
#if (PG_VERSION_NUM >= PG_VERSION_15)
bool tableOnly = false;
/* since postgres 15, tables can have a column list and filter */
@ -186,6 +192,20 @@ BuildCreatePublicationStmt(Oid publicationId)
BuildPublicationRelationObjSpec(relationId, publicationId, tableOnly);
createPubStmt->pubobjects = lappend(createPubStmt->pubobjects, publicationObject);
#else
/* before postgres 15, only full tables are supported */
char *schemaName = get_namespace_name(get_rel_namespace(relationId));
char *tableName = get_rel_name(relationId);
RangeVar *rangeVar = makeRangeVar(schemaName, tableName, -1);
createPubStmt->tables = lappend(createPubStmt->tables, rangeVar);
#endif
if (IsCitusTable(relationId))
{
citusTableCount++;
}
}
/* WITH (publish_via_partition_root = true) option */
@ -256,6 +276,8 @@ AppendPublishOptionList(StringInfo str, List *options)
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* BuildPublicationRelationObjSpec returns a PublicationObjSpec that
* can be included in a CREATE or ALTER PUBLICATION statement.
@ -335,6 +357,9 @@ BuildPublicationRelationObjSpec(Oid relationId, Oid publicationId,
}
#endif
/*
* PreprocessAlterPublicationStmt handles ALTER PUBLICATION statements
* in a way that is mostly similar to PreprocessAlterDistributedObjectStmt,
@ -433,6 +458,7 @@ GetAlterPublicationTableDDLCommand(Oid publicationId, Oid relationId,
ReleaseSysCache(pubTuple);
#if (PG_VERSION_NUM >= PG_VERSION_15)
bool tableOnly = !isAdd;
/* since postgres 15, tables can have a column list and filter */
@ -441,6 +467,16 @@ GetAlterPublicationTableDDLCommand(Oid publicationId, Oid relationId,
alterPubStmt->pubobjects = lappend(alterPubStmt->pubobjects, publicationObject);
alterPubStmt->action = isAdd ? AP_AddObjects : AP_DropObjects;
#else
/* before postgres 15, only full tables are supported */
char *schemaName = get_namespace_name(get_rel_namespace(relationId));
char *tableName = get_rel_name(relationId);
RangeVar *rangeVar = makeRangeVar(schemaName, tableName, -1);
alterPubStmt->tables = lappend(alterPubStmt->tables, rangeVar);
alterPubStmt->tableAction = isAdd ? DEFELEM_ADD : DEFELEM_DROP;
#endif
/* we take the WHERE clause from the catalog where it is already transformed */
bool whereClauseNeedsTransform = false;

View File

@ -45,7 +45,6 @@
#include "distributed/citus_safe_lib.h"
#include "distributed/commands.h"
#include "distributed/commands/utility_hook.h"
#include "distributed/comment.h"
#include "distributed/coordinator_protocol.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
@ -83,6 +82,7 @@ static const char * WrapQueryInAlterRoleIfExistsCall(const char *query, RoleSpec
static VariableSetStmt * MakeVariableSetStmt(const char *config);
static int ConfigGenericNameCompare(const void *lhs, const void *rhs);
static List * RoleSpecToObjectAddress(RoleSpec *role, bool missing_ok);
static bool IsGrantRoleWithInheritOrSetOption(GrantRoleStmt *stmt);
/* controlled via GUC */
bool EnableCreateRolePropagation = true;
@ -160,7 +160,7 @@ PostprocessAlterRoleStmt(Node *node, const char *queryString)
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
AlterRoleStmt *stmt = castNode(AlterRoleStmt, node);
@ -189,7 +189,7 @@ PostprocessAlterRoleStmt(Node *node, const char *queryString)
(void *) CreateAlterRoleIfExistsCommand(stmt),
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commands);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
@ -235,7 +235,7 @@ PreprocessAlterRoleSetStmt(Node *node, const char *queryString,
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
QualifyTreeNode((Node *) stmt);
const char *sql = DeparseTreeNode((Node *) stmt);
@ -244,7 +244,7 @@ PreprocessAlterRoleSetStmt(Node *node, const char *queryString,
(void *) sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commandList);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commandList);
}
@ -584,17 +584,6 @@ GenerateCreateOrAlterRoleCommand(Oid roleOid)
{
completeRoleList = lappend(completeRoleList, DeparseTreeNode(stmt));
}
/*
* append COMMENT ON ROLE commands for this specific user
* When we propagate user creation, we also want to make sure that we propagate
* all the comments it has been given. For this, we check pg_shdescription
* for the ROLE entry corresponding to roleOid, and generate the relevant
* Comment stmts to be run in the new node.
*/
List *commentStmts = GetCommentPropagationCommands(AuthIdRelationId, roleOid,
rolename, OBJECT_ROLE);
completeRoleList = list_concat(completeRoleList, commentStmts);
}
return completeRoleList;
@ -887,14 +876,6 @@ GenerateGrantRoleStmtsOfRole(Oid roleid)
{
Form_pg_auth_members membership = (Form_pg_auth_members) GETSTRUCT(tuple);
ObjectAddress *roleAddress = palloc0(sizeof(ObjectAddress));
ObjectAddressSet(*roleAddress, AuthIdRelationId, membership->grantor);
if (!IsAnyObjectDistributed(list_make1(roleAddress)))
{
/* we only need to propagate the grant if the grantor is distributed */
continue;
}
GrantRoleStmt *grantRoleStmt = makeNode(GrantRoleStmt);
grantRoleStmt->is_grant = true;
@ -910,38 +891,13 @@ GenerateGrantRoleStmtsOfRole(Oid roleid)
granteeRole->rolename = GetUserNameFromId(membership->member, true);
grantRoleStmt->grantee_roles = list_make1(granteeRole);
RoleSpec *grantorRole = makeNode(RoleSpec);
grantorRole->roletype = ROLESPEC_CSTRING;
grantorRole->location = -1;
grantorRole->rolename = GetUserNameFromId(membership->grantor, false);
grantRoleStmt->grantor = grantorRole;
grantRoleStmt->grantor = NULL;
#if PG_VERSION_NUM >= PG_VERSION_16
/* inherit option is always included */
DefElem *inherit_opt;
if (membership->inherit_option)
{
inherit_opt = makeDefElem("inherit", (Node *) makeBoolean(true), -1);
}
else
{
inherit_opt = makeDefElem("inherit", (Node *) makeBoolean(false), -1);
}
grantRoleStmt->opt = list_make1(inherit_opt);
/* admin option is false by default, only include true case */
if (membership->admin_option)
{
DefElem *admin_opt = makeDefElem("admin", (Node *) makeBoolean(true), -1);
grantRoleStmt->opt = lappend(grantRoleStmt->opt, admin_opt);
}
/* set option is true by default, only include false case */
if (!membership->set_option)
{
DefElem *set_opt = makeDefElem("set", (Node *) makeBoolean(false), -1);
grantRoleStmt->opt = lappend(grantRoleStmt->opt, set_opt);
DefElem *opt = makeDefElem("admin", (Node *) makeBoolean(true), -1);
grantRoleStmt->opt = list_make1(opt);
}
#else
grantRoleStmt->admin_opt = membership->admin_option;
@ -1020,8 +976,7 @@ PreprocessCreateRoleStmt(Node *node, const char *queryString,
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
EnsureSequentialModeForRoleDDL();
LockRelationOid(DistNodeRelationId(), RowShareLock);
@ -1056,7 +1011,7 @@ PreprocessCreateRoleStmt(Node *node, const char *queryString,
commands = lappend(commands, ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commands);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
@ -1072,8 +1027,13 @@ makeStringConst(char *str, int location)
{
A_Const *n = makeNode(A_Const);
#if PG_VERSION_NUM >= PG_VERSION_15
n->val.sval.type = T_String;
n->val.sval.sval = str;
#else
n->val.type = T_String;
n->val.val.str = str;
#endif
n->location = location;
return (Node *) n;
@ -1093,8 +1053,13 @@ makeIntConst(int val, int location)
{
A_Const *n = makeNode(A_Const);
#if PG_VERSION_NUM >= PG_VERSION_15
n->val.ival.type = T_Integer;
n->val.ival.ival = val;
#else
n->val.type = T_Integer;
n->val.val.ival = val;
#endif
n->location = location;
return (Node *) n;
@ -1111,8 +1076,13 @@ makeFloatConst(char *str, int location)
{
A_Const *n = makeNode(A_Const);
#if PG_VERSION_NUM >= PG_VERSION_15
n->val.fval.type = T_Float;
n->val.fval.fval = str;
#else
n->val.type = T_Float;
n->val.val.str = str;
#endif
n->location = location;
return (Node *) n;
@ -1142,8 +1112,7 @@ PreprocessDropRoleStmt(Node *node, const char *queryString,
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
EnsureSequentialModeForRoleDDL();
@ -1155,7 +1124,7 @@ PreprocessDropRoleStmt(Node *node, const char *queryString,
sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commands);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
@ -1232,7 +1201,7 @@ PreprocessGrantRoleStmt(Node *node, const char *queryString,
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
GrantRoleStmt *stmt = castNode(GrantRoleStmt, node);
List *allGranteeRoles = stmt->grantee_roles;
@ -1244,6 +1213,25 @@ PreprocessGrantRoleStmt(Node *node, const char *queryString,
return NIL;
}
if (IsGrantRoleWithInheritOrSetOption(stmt))
{
if (EnableUnsupportedFeatureMessages)
{
ereport(NOTICE, (errmsg("not propagating GRANT/REVOKE commands with specified"
" INHERIT/SET options to worker nodes"),
errhint(
"Connect to worker nodes directly to manually run the same"
" GRANT/REVOKE command after disabling DDL propagation.")));
}
return NIL;
}
/*
* Postgres don't seem to use the grantor. Even dropping the grantor doesn't
* seem to affect the membership. If this changes, we might need to add grantors
* to the dependency resolution too. For now we just don't propagate it.
*/
stmt->grantor = NULL;
stmt->grantee_roles = distributedGranteeRoles;
char *sql = DeparseTreeNode((Node *) stmt);
stmt->grantee_roles = allGranteeRoles;
@ -1253,7 +1241,7 @@ PreprocessGrantRoleStmt(Node *node, const char *queryString,
sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commands);
return NodeDDLTaskList(NON_COORDINATOR_NODES, commands);
}
@ -1264,13 +1252,11 @@ PreprocessGrantRoleStmt(Node *node, const char *queryString,
List *
PostprocessGrantRoleStmt(Node *node, const char *queryString)
{
if (!EnableCreateRolePropagation || !ShouldPropagate())
if (!EnableCreateRolePropagation || !IsCoordinator() || !ShouldPropagate())
{
return NIL;
}
EnsurePropagationToCoordinator();
GrantRoleStmt *stmt = castNode(GrantRoleStmt, node);
RoleSpec *role = NULL;
@ -1289,6 +1275,27 @@ PostprocessGrantRoleStmt(Node *node, const char *queryString)
}
/*
* IsGrantRoleWithInheritOrSetOption returns true if the given
* GrantRoleStmt has inherit or set option specified in its options
*/
static bool
IsGrantRoleWithInheritOrSetOption(GrantRoleStmt *stmt)
{
#if PG_VERSION_NUM >= PG_VERSION_16
DefElem *opt = NULL;
foreach_declared_ptr(opt, stmt->opt)
{
if (strcmp(opt->defname, "inherit") == 0 || strcmp(opt->defname, "set") == 0)
{
return true;
}
}
#endif
return false;
}
/*
* ConfigGenericNameCompare compares two config_generic structs based on their
* name fields. If the name fields contain the same strings two structs are
@ -1370,54 +1377,3 @@ EnsureSequentialModeForRoleDDL(void)
"use only one connection for all future commands")));
SetLocalMultiShardModifyModeToSequential();
}
/*
* PreprocessAlterDatabaseSetStmt is executed before the statement is applied to the local
* postgres instance.
*
* In this stage we can prepare the commands that need to be run on all workers to grant
* on databases.
*/
List *
PreprocessAlterRoleRenameStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
if (!ShouldPropagate())
{
return NIL;
}
if (!EnableAlterRolePropagation)
{
return NIL;
}
RenameStmt *stmt = castNode(RenameStmt, node);
Assert(stmt->renameType == OBJECT_ROLE);
EnsurePropagationToCoordinator();
char *sql = DeparseTreeNode((Node *) stmt);
List *commands = list_make3(DISABLE_DDL_PROPAGATION,
(void *) sql,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commands);
}
List *
RenameRoleStmtObjectAddress(Node *node, bool missing_ok, bool isPostprocess)
{
RenameStmt *stmt = castNode(RenameStmt, node);
Assert(stmt->renameType == OBJECT_ROLE);
Oid roleOid = get_role_oid(stmt->subname, missing_ok);
ObjectAddress *address = palloc0(sizeof(ObjectAddress));
ObjectAddressSet(*address, AuthIdRelationId, roleOid);
return list_make1(address);
}

View File

@ -15,20 +15,21 @@
#include "distributed/commands/utility_hook.h"
#include "distributed/coordinator_protocol.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#include "distributed/log_utils.h"
#include "distributed/metadata/distobject.h"
#include "distributed/metadata_sync.h"
/*
* PostprocessRoleSecLabelStmt prepares the commands that need to be run on all workers to assign
* security labels on distributed roles. It also ensures that all object dependencies exist on all
* nodes for the role in the SecLabelStmt.
* PostprocessSecLabelStmt prepares the commands that need to be run on all workers to assign
* security labels on distributed objects, currently supporting just Role objects.
* It also ensures that all object dependencies exist on all
* nodes for the object in the SecLabelStmt.
*/
List *
PostprocessRoleSecLabelStmt(Node *node, const char *queryString)
PostprocessSecLabelStmt(Node *node, const char *queryString)
{
if (!EnableAlterRolePropagation || !ShouldPropagate())
if (!ShouldPropagate())
{
return NIL;
}
@ -41,91 +42,38 @@ PostprocessRoleSecLabelStmt(Node *node, const char *queryString)
return NIL;
}
EnsurePropagationToCoordinator();
EnsureAllObjectDependenciesExistOnAllNodes(objectAddresses);
if (secLabelStmt->objtype != OBJECT_ROLE)
{
/*
* If we are not in the coordinator, we don't want to interrupt the security
* label command with notices, the user expects that from the worker node
* the command will not be propagated
*/
if (EnableUnsupportedFeatureMessages && IsCoordinator())
{
ereport(NOTICE, (errmsg("not propagating SECURITY LABEL commands whose "
"object type is not role"),
errhint("Connect to worker nodes directly to manually "
"run the same SECURITY LABEL command.")));
}
return NIL;
}
const char *secLabelCommands = DeparseTreeNode((Node *) secLabelStmt);
List *commandList = list_make3(DISABLE_DDL_PROPAGATION,
(void *) secLabelCommands,
ENABLE_DDL_PROPAGATION);
return NodeDDLTaskList(REMOTE_NODES, commandList);
}
/*
* PostprocessTableOrColumnSecLabelStmt prepares the commands that need to be run on all
* workers to assign security labels on distributed tables or the columns of a distributed
* table. It also ensures that all object dependencies exist on all nodes for the table in
* the SecLabelStmt.
*/
List *
PostprocessTableOrColumnSecLabelStmt(Node *node, const char *queryString)
{
if (!EnableAlterRolePropagation || !ShouldPropagate())
if (!EnableCreateRolePropagation)
{
return NIL;
}
SecLabelStmt *secLabelStmt = castNode(SecLabelStmt, node);
List *objectAddresses = GetObjectAddressListFromParseTree(node, false, true);
if (!IsAnyParentObjectDistributed(objectAddresses))
{
return NIL;
}
EnsurePropagationToCoordinator();
EnsureCoordinator();
EnsureAllObjectDependenciesExistOnAllNodes(objectAddresses);
const char *secLabelCommands = DeparseTreeNode((Node *) secLabelStmt);
const char *sql = DeparseTreeNode((Node *) secLabelStmt);
List *commandList = list_make3(DISABLE_DDL_PROPAGATION,
(void *) secLabelCommands,
(void *) sql,
ENABLE_DDL_PROPAGATION);
List *DDLJobs = NodeDDLTaskList(REMOTE_NODES, commandList);
ListCell *lc = NULL;
/*
* The label is for a table or a column, so we need to set the targetObjectAddress
* of the DDLJob to the relationId of the table. This is needed to ensure that
* the search path is correctly set for the remote security label command; it
* needs to be able to resolve the table that the label is being defined on.
*/
Assert(list_length(objectAddresses) == 1);
ObjectAddress *target = linitial(objectAddresses);
Oid relationId = target->objectId;
Assert(relationId != InvalidOid);
foreach(lc, DDLJobs)
{
DDLJob *ddlJob = (DDLJob *) lfirst(lc);
ObjectAddressSet(ddlJob->targetObjectAddress, RelationRelationId, relationId);
}
return DDLJobs;
}
/*
* PostprocessAnySecLabelStmt is used for any other object types
* that are not supported by Citus. It issues a notice to the client
* if appropriate. Is effectively a nop.
*/
List *
PostprocessAnySecLabelStmt(Node *node, const char *queryString)
{
/*
* If we are not in the coordinator, we don't want to interrupt the security
* label command with notices, the user expects that from the worker node
* the command will not be propagated
*/
if (EnableUnsupportedFeatureMessages && IsCoordinator())
{
ereport(NOTICE, (errmsg("not propagating SECURITY LABEL commands whose "
"object type is not role or table or column"),
errhint("Connect to worker nodes directly to manually "
"run the same SECURITY LABEL command.")));
}
return NIL;
return NodeDDLTaskList(NON_COORDINATOR_NODES, commandList);
}

View File

@ -735,6 +735,8 @@ PostprocessAlterSequenceOwnerStmt(Node *node, const char *queryString)
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* PreprocessAlterSequencePersistenceStmt is called for change of persistence
* of sequences before the persistence is changed on the local instance.
@ -845,6 +847,9 @@ PreprocessSequenceAlterTableStmt(Node *node, const char *queryString,
}
#endif
/*
* PreprocessGrantOnSequenceStmt is executed before the statement is applied to the local
* postgres instance.

View File

@ -1,275 +0,0 @@
/*-------------------------------------------------------------------------
*
* serialize_distributed_ddls.c
*
* This file contains functions for serializing distributed DDLs.
*
* If you're adding support for serializing a new DDL, you should
* extend the following functions to support the new object class:
* AcquireCitusAdvisoryObjectClassLockGetOid()
* AcquireCitusAdvisoryObjectClassLockCheckPrivileges()
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "miscadmin.h"
#include "catalog/dependency.h"
#include "catalog/pg_database_d.h"
#include "commands/dbcommands.h"
#include "storage/lock.h"
#include "utils/builtins.h"
#include "pg_version_compat.h"
#include "distributed/adaptive_executor.h"
#include "distributed/argutils.h"
#include "distributed/commands/serialize_distributed_ddls.h"
#include "distributed/deparse_shard_query.h"
#include "distributed/resource_lock.h"
PG_FUNCTION_INFO_V1(citus_internal_acquire_citus_advisory_object_class_lock);
static void SerializeDistributedDDLsOnObjectClassInternal(ObjectClass objectClass,
char *qualifiedObjectName);
static char * AcquireCitusAdvisoryObjectClassLockCommand(ObjectClass objectClass,
char *qualifiedObjectName);
static void AcquireCitusAdvisoryObjectClassLock(ObjectClass objectClass,
char *qualifiedObjectName);
static Oid AcquireCitusAdvisoryObjectClassLockGetOid(ObjectClass objectClass,
char *qualifiedObjectName);
static void AcquireCitusAdvisoryObjectClassLockCheckPrivileges(ObjectClass objectClass,
Oid oid);
/*
* citus_internal_acquire_citus_advisory_object_class_lock is an internal UDF
* to call AcquireCitusAdvisoryObjectClassLock().
*/
Datum
citus_internal_acquire_citus_advisory_object_class_lock(PG_FUNCTION_ARGS)
{
CheckCitusVersion(ERROR);
PG_ENSURE_ARGNOTNULL(0, "object_class");
ObjectClass objectClass = PG_GETARG_INT32(0);
char *qualifiedObjectName = PG_ARGISNULL(1) ? NULL : PG_GETARG_CSTRING(1);
AcquireCitusAdvisoryObjectClassLock(objectClass, qualifiedObjectName);
PG_RETURN_VOID();
}
/*
* SerializeDistributedDDLsOnObjectClass is a wrapper around
* SerializeDistributedDDLsOnObjectClassInternal to acquire the lock on given
* object class itself, see the comment in header file for more details about
* the difference between this function and
* SerializeDistributedDDLsOnObjectClassObject().
*/
void
SerializeDistributedDDLsOnObjectClass(ObjectClass objectClass)
{
SerializeDistributedDDLsOnObjectClassInternal(objectClass, NULL);
}
/*
* SerializeDistributedDDLsOnObjectClassObject is a wrapper around
* SerializeDistributedDDLsOnObjectClassInternal to acquire the lock on given
* object that belongs to given object class, see the comment in header file
* for more details about the difference between this function and
* SerializeDistributedDDLsOnObjectClass().
*/
void
SerializeDistributedDDLsOnObjectClassObject(ObjectClass objectClass,
char *qualifiedObjectName)
{
if (qualifiedObjectName == NULL)
{
elog(ERROR, "qualified object name cannot be NULL");
}
SerializeDistributedDDLsOnObjectClassInternal(objectClass, qualifiedObjectName);
}
/*
* SerializeDistributedDDLsOnObjectClassInternal serializes distributed DDLs
* that target given object class by acquiring a Citus specific advisory lock
* on the first primary worker node if there are any workers in the cluster.
*
* The lock is acquired via a coordinated transaction. For this reason,
* it automatically gets released when (maybe implicit) transaction on
* current server commits or rolls back.
*
* If qualifiedObjectName is provided to be non-null, then the oid of the
* object is first resolved on the first primary worker node and then the
* lock is acquired on that oid. If qualifiedObjectName is null, then the
* lock is acquired on the object class itself.
*
* Note that those two lock types don't conflict with each other and are
* acquired for different purposes. The lock on the object class
* (qualifiedObjectName = NULL) is used to serialize DDLs that target the
* object class itself, e.g., when creating a new object of that class, and
* the latter is used to serialize DDLs that target a specific object of
* that class, e.g., when altering an object.
*
* In some cases, we may want to acquire both locks at the same time. For
* example, when renaming a database, we want to acquire both lock types
* because while the object class lock is used to ensure that another session
* doesn't create a new database with the same name, the object lock is used
* to ensure that another session doesn't alter the same database.
*/
static void
SerializeDistributedDDLsOnObjectClassInternal(ObjectClass objectClass,
char *qualifiedObjectName)
{
WorkerNode *firstWorkerNode = GetFirstPrimaryWorkerNode();
if (firstWorkerNode == NULL)
{
/*
* If there are no worker nodes in the cluster, then we don't need
* to acquire the lock at all; and we cannot indeed.
*/
return;
}
/*
* Indeed we would already ensure permission checks in remote node
* --via AcquireCitusAdvisoryObjectClassLock()-- but we first do so on
* the local node to avoid from reporting confusing error messages.
*/
Oid oid = AcquireCitusAdvisoryObjectClassLockGetOid(objectClass, qualifiedObjectName);
AcquireCitusAdvisoryObjectClassLockCheckPrivileges(objectClass, oid);
Task *task = CitusMakeNode(Task);
task->taskType = DDL_TASK;
char *command = AcquireCitusAdvisoryObjectClassLockCommand(objectClass,
qualifiedObjectName);
SetTaskQueryString(task, command);
ShardPlacement *targetPlacement = CitusMakeNode(ShardPlacement);
SetPlacementNodeMetadata(targetPlacement, firstWorkerNode);
task->taskPlacementList = list_make1(targetPlacement);
/* need to be in a transaction to acquire a lock that's bound to transactions */
UseCoordinatedTransaction();
bool localExecutionSupported = true;
ExecuteUtilityTaskList(list_make1(task), localExecutionSupported);
}
/*
* AcquireCitusAdvisoryObjectClassLockCommand returns a command to call
* citus_internal.acquire_citus_advisory_object_class_lock().
*/
static char *
AcquireCitusAdvisoryObjectClassLockCommand(ObjectClass objectClass,
char *qualifiedObjectName)
{
/* safe to cast to int as it's an enum */
int objectClassInt = (int) objectClass;
char *quotedObjectName =
!qualifiedObjectName ? "NULL" :
quote_literal_cstr(qualifiedObjectName);
StringInfo command = makeStringInfo();
appendStringInfo(command,
"SELECT citus_internal.acquire_citus_advisory_object_class_lock(%d, %s)",
objectClassInt, quotedObjectName);
return command->data;
}
/*
* AcquireCitusAdvisoryObjectClassLock acquires a Citus specific advisory
* ExclusiveLock based on given object class.
*/
static void
AcquireCitusAdvisoryObjectClassLock(ObjectClass objectClass, char *qualifiedObjectName)
{
Oid oid = AcquireCitusAdvisoryObjectClassLockGetOid(objectClass, qualifiedObjectName);
AcquireCitusAdvisoryObjectClassLockCheckPrivileges(objectClass, oid);
LOCKTAG locktag;
SET_LOCKTAG_GLOBAL_DDL_SERIALIZATION(locktag, objectClass, oid);
LOCKMODE lockmode = ExclusiveLock;
bool sessionLock = false;
bool dontWait = false;
LockAcquire(&locktag, lockmode, sessionLock, dontWait);
}
/*
* AcquireCitusAdvisoryObjectClassLockGetOid returns the oid of given object
* that belongs to given object class. If qualifiedObjectName is NULL, then
* it returns InvalidOid.
*/
static Oid
AcquireCitusAdvisoryObjectClassLockGetOid(ObjectClass objectClass,
char *qualifiedObjectName)
{
if (qualifiedObjectName == NULL)
{
return InvalidOid;
}
bool missingOk = false;
switch (objectClass)
{
case OCLASS_DATABASE:
{
return get_database_oid(qualifiedObjectName, missingOk);
}
default:
elog(ERROR, "unsupported object class: %d", objectClass);
}
}
/*
* AcquireCitusAdvisoryObjectClassLockCheckPrivileges is used to perform privilege checks
* before acquiring the Citus specific advisory lock on given object class and oid.
*/
static void
AcquireCitusAdvisoryObjectClassLockCheckPrivileges(ObjectClass objectClass, Oid oid)
{
switch (objectClass)
{
case OCLASS_DATABASE:
{
if (OidIsValid(oid) && !object_ownercheck(DatabaseRelationId, oid,
GetUserId()))
{
aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_DATABASE,
get_database_name(oid));
}
else if (!OidIsValid(oid) && !have_createdb_privilege())
{
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied to create / rename database")));
}
break;
}
default:
elog(ERROR, "unsupported object class: %d", objectClass);
}
}

View File

@ -1153,6 +1153,7 @@ PreprocessAlterTableStmt(Node *node, const char *alterTableCommand,
{
AlterTableStmt *stmtCopy = copyObject(alterTableStatement);
stmtCopy->objtype = OBJECT_SEQUENCE;
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* it must be ALTER TABLE .. OWNER TO ..
@ -1162,6 +1163,16 @@ PreprocessAlterTableStmt(Node *node, const char *alterTableCommand,
*/
return PreprocessSequenceAlterTableStmt((Node *) stmtCopy, alterTableCommand,
processUtilityContext);
#else
/*
* it must be ALTER TABLE .. OWNER TO .. command
* since this is the only ALTER command of a sequence that
* passes through an AlterTableStmt
*/
return PreprocessAlterSequenceOwnerStmt((Node *) stmtCopy, alterTableCommand,
processUtilityContext);
#endif
}
else if (relKind == RELKIND_VIEW)
{
@ -3662,8 +3673,9 @@ ErrorIfUnsupportedAlterTableStmt(AlterTableStmt *alterTableStatement)
"are currently unsupported.")));
break;
}
#endif
#endif
#if PG_VERSION_NUM >= PG_VERSION_15
case AT_SetAccessMethod:
{
/*
@ -3683,6 +3695,7 @@ ErrorIfUnsupportedAlterTableStmt(AlterTableStmt *alterTableStatement)
break;
}
#endif
case AT_SetNotNull:
case AT_ReplicaIdentity:
case AT_ChangeOwner:

View File

@ -790,6 +790,45 @@ AlterTextSearchDictionarySchemaStmtObjectAddress(Node *node, bool missing_ok, bo
}
/*
* TextSearchConfigurationCommentObjectAddress resolves the ObjectAddress for the TEXT
* SEARCH CONFIGURATION on which the comment is placed. Optionally errors if the
* configuration does not exist based on the missing_ok flag passed in by the caller.
*/
List *
TextSearchConfigurationCommentObjectAddress(Node *node, bool missing_ok, bool
isPostprocess)
{
CommentStmt *stmt = castNode(CommentStmt, node);
Assert(stmt->objtype == OBJECT_TSCONFIGURATION);
Oid objid = get_ts_config_oid(castNode(List, stmt->object), missing_ok);
ObjectAddress *address = palloc0(sizeof(ObjectAddress));
ObjectAddressSet(*address, TSConfigRelationId, objid);
return list_make1(address);
}
/*
* TextSearchDictCommentObjectAddress resolves the ObjectAddress for the TEXT SEARCH
* DICTIONARY on which the comment is placed. Optionally errors if the dictionary does not
* exist based on the missing_ok flag passed in by the caller.
*/
List *
TextSearchDictCommentObjectAddress(Node *node, bool missing_ok, bool isPostprocess)
{
CommentStmt *stmt = castNode(CommentStmt, node);
Assert(stmt->objtype == OBJECT_TSDICTIONARY);
Oid objid = get_ts_dict_oid(castNode(List, stmt->object), missing_ok);
ObjectAddress *address = palloc0(sizeof(ObjectAddress));
ObjectAddressSet(*address, TSDictionaryRelationId, objid);
return list_make1(address);
}
/*
* AlterTextSearchConfigurationOwnerObjectAddress resolves the ObjectAddress for the TEXT
* SEARCH CONFIGURATION for which the owner is changed. Optionally errors if the

View File

@ -57,6 +57,9 @@ static void ExtractDropStmtTriggerAndRelationName(DropStmt *dropTriggerStmt,
static void ErrorIfDropStmtDropsMultipleTriggers(DropStmt *dropTriggerStmt);
static char * GetTriggerNameById(Oid triggerId);
static int16 GetTriggerTypeById(Oid triggerId);
#if (PG_VERSION_NUM < PG_VERSION_15)
static void ErrorOutIfCloneTrigger(Oid tgrelid, const char *tgname);
#endif
/* GUC that overrides trigger checks for distributed tables and reference tables */
@ -401,6 +404,40 @@ CreateTriggerEventExtendNames(CreateTrigStmt *createTriggerStmt, char *schemaNam
}
/*
* PreprocessAlterTriggerRenameStmt is called before a ALTER TRIGGER RENAME
* command has been executed by standard process utility. This function errors
* out if we are trying to rename a child trigger on a partition of a distributed
* table. In PG15, this is not allowed anyway.
*/
List *
PreprocessAlterTriggerRenameStmt(Node *node, const char *queryString,
ProcessUtilityContext processUtilityContext)
{
#if (PG_VERSION_NUM < PG_VERSION_15)
RenameStmt *renameTriggerStmt = castNode(RenameStmt, node);
Assert(renameTriggerStmt->renameType == OBJECT_TRIGGER);
RangeVar *relation = renameTriggerStmt->relation;
bool missingOk = false;
Oid relationId = RangeVarGetRelid(relation, ALTER_TRIGGER_LOCK_MODE, missingOk);
if (!IsCitusTable(relationId))
{
return NIL;
}
EnsureCoordinator();
ErrorOutForTriggerIfNotSupported(relationId);
ErrorOutIfCloneTrigger(relationId, renameTriggerStmt->subname);
#endif
return NIL;
}
/*
* PostprocessAlterTriggerRenameStmt is called after a ALTER TRIGGER RENAME
* command has been executed by standard process utility. This function errors
@ -722,6 +759,64 @@ ErrorIfRelationHasUnsupportedTrigger(Oid relationId)
}
#if (PG_VERSION_NUM < PG_VERSION_15)
/*
* ErrorOutIfCloneTrigger is a helper function to error
* out if we are trying to rename a child trigger on a
* partition of a distributed table.
* A lot of this code is borrowed from PG15 because
* renaming clone triggers isn't allowed in PG15 anymore.
*/
static void
ErrorOutIfCloneTrigger(Oid tgrelid, const char *tgname)
{
HeapTuple tuple;
ScanKeyData key[2];
Relation tgrel = table_open(TriggerRelationId, RowExclusiveLock);
/*
* Search for the trigger to modify.
*/
ScanKeyInit(&key[0],
Anum_pg_trigger_tgrelid,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(tgrelid));
ScanKeyInit(&key[1],
Anum_pg_trigger_tgname,
BTEqualStrategyNumber, F_NAMEEQ,
CStringGetDatum(tgname));
SysScanDesc tgscan = systable_beginscan(tgrel, TriggerRelidNameIndexId, true,
NULL, 2, key);
if (HeapTupleIsValid(tuple = systable_getnext(tgscan)))
{
Form_pg_trigger trigform = (Form_pg_trigger) GETSTRUCT(tuple);
/*
* If the trigger descends from a trigger on a parent partitioned
* table, reject the rename.
* Appended shard ids to find the trigger on the partition's shards
* are not correct. Hence we would fail to find the trigger on the
* partition's shard.
*/
if (OidIsValid(trigform->tgparentid))
{
ereport(ERROR, (
errmsg(
"cannot rename child triggers on distributed partitions")));
}
}
systable_endscan(tgscan);
table_close(tgrel, RowExclusiveLock);
}
#endif
/*
* GetDropTriggerStmtRelation takes a DropStmt for a trigger object and returns
* RangeVar for the relation that owns the trigger.

View File

@ -34,8 +34,6 @@
#include "access/htup_details.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_database.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/extension.h"
@ -45,7 +43,6 @@
#include "nodes/makefuncs.h"
#include "nodes/parsenodes.h"
#include "nodes/pg_list.h"
#include "postmaster/postmaster.h"
#include "tcop/utility.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
@ -79,7 +76,6 @@
#include "distributed/multi_partitioning_utils.h"
#include "distributed/multi_physical_planner.h"
#include "distributed/reference_table_utils.h"
#include "distributed/remote_commands.h"
#include "distributed/resource_lock.h"
#include "distributed/string_utils.h"
#include "distributed/transaction_management.h"
@ -87,6 +83,7 @@
#include "distributed/worker_shard_visibility.h"
#include "distributed/worker_transaction.h"
bool EnableDDLPropagation = true; /* ddl propagation is enabled */
int CreateObjectPropagationMode = CREATE_OBJECT_PROPAGATION_IMMEDIATE;
PropSetCmdBehavior PropagateSetCommands = PROPSETCMD_NONE; /* SET prop off */
@ -100,13 +97,13 @@ int UtilityHookLevel = 0;
/* Local functions forward declarations for helper functions */
static void citus_ProcessUtilityInternal(PlannedStmt *pstmt,
const char *queryString,
ProcessUtilityContext context,
ParamListInfo params,
struct QueryEnvironment *queryEnv,
DestReceiver *dest,
QueryCompletion *completionTag);
static void ProcessUtilityInternal(PlannedStmt *pstmt,
const char *queryString,
ProcessUtilityContext context,
ParamListInfo params,
struct QueryEnvironment *queryEnv,
DestReceiver *dest,
QueryCompletion *completionTag);
static void set_indexsafe_procflags(void);
static char * CurrentSearchPath(void);
static void IncrementUtilityHookCountersIfNecessary(Node *parsetree);
@ -115,7 +112,6 @@ static void DecrementUtilityHookCountersIfNecessary(Node *parsetree);
static bool IsDropSchemaOrDB(Node *parsetree);
static bool ShouldCheckUndistributeCitusLocalTables(void);
/*
* ProcessUtilityParseTree is a convenience method to create a PlannedStmt out of
* pieces of a utility statement before invoking ProcessUtility.
@ -136,7 +132,7 @@ ProcessUtilityParseTree(Node *node, const char *queryString, ProcessUtilityConte
/*
* citus_ProcessUtility is the main entry hook for implementing Citus-specific
* multi_ProcessUtility is the main entry hook for implementing Citus-specific
* utility behavior. Its primary responsibilities are intercepting COPY and DDL
* commands and augmenting the coordinator's command with corresponding tasks
* to be run on worker nodes, after suitably ensuring said commands' options
@ -145,7 +141,7 @@ ProcessUtilityParseTree(Node *node, const char *queryString, ProcessUtilityConte
* TRUNCATE and VACUUM are also supported.
*/
void
citus_ProcessUtility(PlannedStmt *pstmt,
multi_ProcessUtility(PlannedStmt *pstmt,
const char *queryString,
bool readOnlyTree,
ProcessUtilityContext context,
@ -247,25 +243,11 @@ citus_ProcessUtility(PlannedStmt *pstmt,
if (!CitusHasBeenLoaded())
{
/*
* Process the command via RunPreprocessNonMainDBCommand and
* RunPostprocessNonMainDBCommand hooks if we're in a non-main database
* and if the command is a node-wide object management command that we
* support from non-main databases.
* Ensure that utility commands do not behave any differently until CREATE
* EXTENSION is invoked.
*/
bool shouldSkipPrevUtilityHook = RunPreprocessNonMainDBCommand(parsetree);
if (!shouldSkipPrevUtilityHook)
{
/*
* Ensure that utility commands do not behave any differently until CREATE
* EXTENSION is invoked.
*/
PrevProcessUtility(pstmt, queryString, false, context,
params, queryEnv, dest, completionTag);
}
RunPostprocessNonMainDBCommand(parsetree);
PrevProcessUtility(pstmt, queryString, false, context,
params, queryEnv, dest, completionTag);
return;
}
@ -349,8 +331,8 @@ citus_ProcessUtility(PlannedStmt *pstmt,
PG_TRY();
{
citus_ProcessUtilityInternal(pstmt, queryString, context, params, queryEnv, dest,
completionTag);
ProcessUtilityInternal(pstmt, queryString, context, params, queryEnv, dest,
completionTag);
if (UtilityHookLevel == 1)
{
@ -424,7 +406,7 @@ citus_ProcessUtility(PlannedStmt *pstmt,
/*
* citus_ProcessUtilityInternal is a helper function for citus_ProcessUtility where majority
* ProcessUtilityInternal is a helper function for multi_ProcessUtility where majority
* of the Citus specific utility statements are handled here. The distinction between
* both functions is that Citus_ProcessUtility does not handle CALL and DO statements.
* The reason for the distinction is implemented to be able to find the "top-level" DDL
@ -432,13 +414,13 @@ citus_ProcessUtility(PlannedStmt *pstmt,
* this goal.
*/
static void
citus_ProcessUtilityInternal(PlannedStmt *pstmt,
const char *queryString,
ProcessUtilityContext context,
ParamListInfo params,
struct QueryEnvironment *queryEnv,
DestReceiver *dest,
QueryCompletion *completionTag)
ProcessUtilityInternal(PlannedStmt *pstmt,
const char *queryString,
ProcessUtilityContext context,
ParamListInfo params,
struct QueryEnvironment *queryEnv,
DestReceiver *dest,
QueryCompletion *completionTag)
{
Node *parsetree = pstmt->utilityStmt;
List *ddlJobs = NIL;
@ -714,32 +696,25 @@ citus_ProcessUtilityInternal(PlannedStmt *pstmt,
}
/* inform the user about potential caveats */
if (IsA(parsetree, CreatedbStmt) && !EnableCreateDatabasePropagation)
if (IsA(parsetree, CreatedbStmt))
{
if (EnableUnsupportedFeatureMessages)
{
ereport(NOTICE, (errmsg("Citus partially supports CREATE DATABASE for "
"distributed databases"),
errdetail("Citus does not propagate CREATE DATABASE "
"command to other nodes"),
"command to workers"),
errhint("You can manually create a database and its "
"extensions on other nodes.")));
"extensions on workers.")));
}
}
else if (IsA(parsetree, CreateRoleStmt) && !EnableCreateRolePropagation)
{
ereport(NOTICE, (errmsg("not propagating CREATE ROLE/USER commands to other"
ereport(NOTICE, (errmsg("not propagating CREATE ROLE/USER commands to worker"
" nodes"),
errhint("Connect to other nodes directly to manually create all"
errhint("Connect to worker nodes directly to manually create all"
" necessary users and roles.")));
}
else if (IsA(parsetree, SecLabelStmt) && !EnableAlterRolePropagation)
{
ereport(NOTICE, (errmsg("not propagating SECURITY LABEL commands to other"
" nodes"),
errhint("Connect to other nodes directly to manually assign"
" necessary labels.")));
}
/*
* Make sure that on DROP EXTENSION we terminate the background daemon
@ -751,13 +726,22 @@ citus_ProcessUtilityInternal(PlannedStmt *pstmt,
}
/*
* Make sure that dropping node-wide objects deletes the pg_dist_object
* entries. There is a separate logic for node-wide objects (such as role
* and databases), since they are not included as dropped objects in the
* drop event trigger. To handle it both on worker and coordinator nodes,
* it is not implemented as a part of process functions but here.
* Make sure that dropping the role deletes the pg_dist_object entries. There is a
* separate logic for roles, since roles are not included as dropped objects in the
* drop event trigger. To handle it both on worker and coordinator nodes, it is not
* implemented as a part of process functions but here.
*/
UnmarkNodeWideObjectsDistributed(parsetree);
if (IsA(parsetree, DropRoleStmt))
{
DropRoleStmt *stmt = castNode(DropRoleStmt, parsetree);
List *allDropRoles = stmt->roles;
List *distributedDropRoles = FilterDistributedRoles(allDropRoles);
if (list_length(distributedDropRoles) > 0)
{
UnmarkRolesDistributed(distributedDropRoles);
}
}
pstmt->utilityStmt = parsetree;
@ -835,6 +819,19 @@ citus_ProcessUtilityInternal(PlannedStmt *pstmt,
ddlJobs = processJobs;
}
}
if (IsA(parsetree, RenameStmt) && ((RenameStmt *) parsetree)->renameType ==
OBJECT_ROLE && EnableAlterRolePropagation)
{
if (EnableUnsupportedFeatureMessages)
{
ereport(NOTICE, (errmsg(
"not propagating ALTER ROLE ... RENAME TO commands "
"to worker nodes"),
errhint("Connect to worker nodes directly to manually "
"rename the role")));
}
}
}
if (IsA(parsetree, CreateStmt))
@ -1124,17 +1121,16 @@ IsDropSchemaOrDB(Node *parsetree)
* each shard placement and COMMIT/ROLLBACK is handled by
* CoordinatedTransactionCallback function.
*
* The function errors out if the DDL is on a partitioned table which has replication
* factor > 1, or if the the coordinator is not added into metadata and we're on a
* worker node because we want to make sure that distributed DDL jobs are executed
* on the coordinator node too. See EnsurePropagationToCoordinator() for more details.
* The function errors out if the node is not the coordinator or if the DDL is on
* a partitioned table which has replication factor > 1.
*
*/
void
ExecuteDistributedDDLJob(DDLJob *ddlJob)
{
bool shouldSyncMetadata = false;
EnsurePropagationToCoordinator();
EnsureCoordinator();
ObjectAddress targetObjectAddress = ddlJob->targetObjectAddress;
@ -1158,24 +1154,23 @@ ExecuteDistributedDDLJob(DDLJob *ddlJob)
{
if (shouldSyncMetadata)
{
SendCommandToRemoteNodesWithMetadata(DISABLE_DDL_PROPAGATION);
SendCommandToWorkersWithMetadata(DISABLE_DDL_PROPAGATION);
char *currentSearchPath = CurrentSearchPath();
/*
* Given that we're relaying the query to the remote nodes directly,
* Given that we're relaying the query to the worker nodes directly,
* we should set the search path exactly the same when necessary.
*/
if (currentSearchPath != NULL)
{
SendCommandToRemoteNodesWithMetadata(
SendCommandToWorkersWithMetadata(
psprintf("SET LOCAL search_path TO %s;", currentSearchPath));
}
if (ddlJob->metadataSyncCommand != NULL)
{
SendCommandToRemoteNodesWithMetadata(
(char *) ddlJob->metadataSyncCommand);
SendCommandToWorkersWithMetadata((char *) ddlJob->metadataSyncCommand);
}
}
@ -1254,7 +1249,7 @@ ExecuteDistributedDDLJob(DDLJob *ddlJob)
char *currentSearchPath = CurrentSearchPath();
/*
* Given that we're relaying the query to the remote nodes directly,
* Given that we're relaying the query to the worker nodes directly,
* we should set the search path exactly the same when necessary.
*/
if (currentSearchPath != NULL)
@ -1266,7 +1261,7 @@ ExecuteDistributedDDLJob(DDLJob *ddlJob)
commandList = lappend(commandList, (char *) ddlJob->metadataSyncCommand);
SendBareCommandListToRemoteMetadataNodes(commandList);
SendBareCommandListToMetadataWorkers(commandList);
}
}
PG_CATCH();
@ -1289,18 +1284,15 @@ ExecuteDistributedDDLJob(DDLJob *ddlJob)
errhint("Use DROP INDEX CONCURRENTLY IF EXISTS to remove the "
"invalid index, then retry the original command.")));
}
else if (ddlJob->warnForPartialFailure)
else
{
ereport(WARNING,
(errmsg(
"Commands that are not transaction-safe may result in "
"partial failure, potentially leading to an inconsistent "
"state.\nIf the problematic command is a CREATE operation, "
"consider using the 'IF EXISTS' syntax to drop the object,"
"\nif applicable, and then re-attempt the original command.")));
"CONCURRENTLY-enabled index commands can fail partially, "
"leaving behind an INVALID index.\n Use DROP INDEX "
"CONCURRENTLY IF EXISTS to remove the invalid index.")));
PG_RE_THROW();
}
PG_RE_THROW();
}
PG_END_TRY();
}
@ -1409,7 +1401,7 @@ PostStandardProcessUtility(Node *parsetree)
* on the local table first. However, in order to decide whether the
* command leads to an invalidation, we need to check before the command
* is being executed since we read pg_constraint table. Thus, we maintain a
* local flag and do the invalidation after citus_ProcessUtility,
* local flag and do the invalidation after multi_ProcessUtility,
* before ExecuteDistributedDDLJob().
*/
InvalidateForeignKeyGraphForDDL();
@ -1512,33 +1504,6 @@ DDLTaskList(Oid relationId, const char *commandString)
}
/*
* NontransactionalNodeDDLTaskList builds a list of tasks to execute a DDL command on a
* given target set of nodes with cannotBeExecutedInTransaction is set to make sure
* that task list is executed outside a transaction block.
*
* Also sets warnForPartialFailure for the returned DDLJobs.
*/
List *
NontransactionalNodeDDLTaskList(TargetWorkerSet targets, List *commands,
bool warnForPartialFailure)
{
List *ddlJobs = NodeDDLTaskList(targets, commands);
DDLJob *ddlJob = NULL;
foreach_declared_ptr(ddlJob, ddlJobs)
{
Task *task = NULL;
foreach_declared_ptr(task, ddlJob->taskList)
{
task->cannotBeExecutedInTransaction = true;
}
ddlJob->warnForPartialFailure = warnForPartialFailure;
}
return ddlJobs;
}
/*
* NodeDDLTaskList builds a list of tasks to execute a DDL command on a
* given target set of nodes.

View File

@ -185,6 +185,7 @@ ExecuteVacuumOnDistributedTables(VacuumStmt *vacuumStmt, List *relationIdList,
CitusVacuumParams vacuumParams)
{
int relationIndex = 0;
int executedVacuumCount = 0;
Oid relationId = InvalidOid;
foreach_declared_oid(relationId, relationIdList)
@ -197,6 +198,7 @@ ExecuteVacuumOnDistributedTables(VacuumStmt *vacuumStmt, List *relationIdList,
/* local execution is not implemented for VACUUM commands */
bool localExecutionSupported = false;
ExecuteUtilityTaskList(taskList, localExecutionSupported);
executedVacuumCount++;
}
relationIndex++;
}
@ -278,7 +280,7 @@ VacuumTaskList(Oid relationId, CitusVacuumParams vacuumParams, List *vacuumColum
task->replicationModel = REPLICATION_MODEL_INVALID;
task->anchorShardId = shardId;
task->taskPlacementList = ActiveShardPlacementList(shardId);
task->cannotBeExecutedInTransaction = ((vacuumParams.options) & VACOPT_VACUUM);
task->cannotBeExecutedInTransction = ((vacuumParams.options) & VACOPT_VACUUM);
taskList = lappend(taskList, task);
}
@ -718,7 +720,7 @@ ExecuteUnqualifiedVacuumTasks(VacuumStmt *vacuumStmt, CitusVacuumParams vacuumPa
SetTaskQueryStringList(task, unqualifiedVacuumCommands);
task->dependentTaskList = NULL;
task->replicationModel = REPLICATION_MODEL_INVALID;
task->cannotBeExecutedInTransaction = ((vacuumParams.options) & VACOPT_VACUUM);
task->cannotBeExecutedInTransction = ((vacuumParams.options) & VACOPT_VACUUM);
bool hasPeerWorker = false;

View File

@ -444,13 +444,11 @@ GetConnParam(const char *keyword)
/*
* GetEffectiveConnKey checks whether there is any pooler configuration for the
* provided key (host/port combination). If a corresponding row is found in the
* poolinfo table, a modified (effective) key is returned with the node, port,
* and dbname overridden, as applicable, otherwise, the original key is returned
* unmodified.
*
* In the case of Citus non-main databases we just return the key, since we
* would not have access to tables with worker information.
* provided key (host/port combination). The one case where this logic is not
* applied is for loopback connections originating within the task tracker. If
* a corresponding row is found in the poolinfo table, a modified (effective)
* key is returned with the node, port, and dbname overridden, as applicable,
* otherwise, the original key is returned unmodified.
*/
ConnectionHashKey *
GetEffectiveConnKey(ConnectionHashKey *key)
@ -460,22 +458,12 @@ GetEffectiveConnKey(ConnectionHashKey *key)
if (!IsTransactionState())
{
/* we're in the task tracker, so should only see loopback */
Assert(strncmp(LocalHostName, key->hostname, MAX_NODE_LENGTH) == 0 &&
Assert(strncmp(LOCAL_HOST_NAME, key->hostname, MAX_NODE_LENGTH) == 0 &&
PostPortNumber == key->port);
return key;
}
if (!CitusHasBeenLoaded())
{
/*
* This happens when we connect to main database over localhost
* from some non Citus database.
*/
return key;
}
WorkerNode *worker = FindWorkerNode(key->hostname, key->port);
if (worker == NULL)
{
/* this can be hit when the key references an unknown node */
@ -536,23 +524,9 @@ char *
GetAuthinfo(char *hostname, int32 port, char *user)
{
char *authinfo = NULL;
bool isLoopback = (strncmp(LocalHostName, hostname, MAX_NODE_LENGTH) == 0 &&
bool isLoopback = (strncmp(LOCAL_HOST_NAME, hostname, MAX_NODE_LENGTH) == 0 &&
PostPortNumber == port);
/*
* Citus will not be loaded when we run a global DDL command from a
* Citus non-main database.
*/
if (!CitusHasBeenLoaded())
{
/*
* We don't expect non-main databases to connect to a node other than
* the local one.
*/
Assert(isLoopback);
return "";
}
if (IsTransactionState())
{
int64 nodeId = WILDCARD_NODE_ID;

View File

@ -39,7 +39,6 @@
#include "distributed/remote_commands.h"
#include "distributed/run_from_same_connection.h"
#include "distributed/shared_connection_stats.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/time_constants.h"
#include "distributed/version_compat.h"
#include "distributed/worker_log_messages.h"
@ -355,18 +354,6 @@ StartNodeUserDatabaseConnection(uint32 flags, const char *hostname, int32 port,
MultiConnection *connection = FindAvailableConnection(entry->connections, flags);
if (connection)
{
/*
* Increment the connection stat counter for the connections that are
* reused only if the connection is in a good state. Here we don't
* bother shutting down the connection or such if it is not in a good
* state but we mostly want to avoid incrementing the connection stat
* counter for a connection that the caller cannot really use.
*/
if (PQstatus(connection->pgConn) == CONNECTION_OK)
{
IncrementStatCounterForMyDb(STAT_CONNECTION_REUSED);
}
return connection;
}
}
@ -408,12 +395,6 @@ StartNodeUserDatabaseConnection(uint32 flags, const char *hostname, int32 port,
dlist_delete(&connection->connectionNode);
pfree(connection);
/*
* Here we don't increment the connection stat counter for the optional
* connections that we gave up establishing due to connection throttling
* because the callers who request optional connections know how to
* survive without them.
*/
return NULL;
}
}
@ -1001,14 +982,6 @@ FinishConnectionListEstablishment(List *multiConnectionList)
{
waitCount++;
}
else if (connectionState->phase == MULTI_CONNECTION_PHASE_ERROR)
{
/*
* Here we count the connections establishments that failed and that
* we won't wait anymore.
*/
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
}
}
/* prepare space for socket events */
@ -1053,11 +1026,6 @@ FinishConnectionListEstablishment(List *multiConnectionList)
if (event->events & WL_POSTMASTER_DEATH)
{
/*
* Here we don't increment the connection stat counter for the
* optional failed connections because this is not a connection
* failure, but a postmaster death in the local node.
*/
ereport(ERROR, (errmsg("postmaster was shut down, exiting")));
}
@ -1074,12 +1042,6 @@ FinishConnectionListEstablishment(List *multiConnectionList)
* reset the memory context
*/
MemoryContextDelete(MemoryContextSwitchTo(oldContext));
/*
* Similarly, we don't increment the connection stat counter for the
* failed connections here because this is not a connection failure
* but a cancellation request is received.
*/
return;
}
@ -1110,7 +1072,6 @@ FinishConnectionListEstablishment(List *multiConnectionList)
eventMask, NULL);
if (!success)
{
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
ereport(ERROR, (errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("connection establishment for node %s:%d "
"failed", connection->hostname,
@ -1127,15 +1088,7 @@ FinishConnectionListEstablishment(List *multiConnectionList)
*/
if (connectionState->phase == MULTI_CONNECTION_PHASE_CONNECTED)
{
/*
* Since WaitEventSetFromMultiConnectionStates() only adds the
* connections that we haven't completed the connection
* establishment yet, here we always have a new connection.
* In other words, at this point, we surely know that we're
* not dealing with a cached connection.
*/
bool newConnection = true;
MarkConnectionConnected(connectionState->connection, newConnection);
MarkConnectionConnected(connectionState->connection);
}
}
}
@ -1219,8 +1172,6 @@ CloseNotReadyMultiConnectionStates(List *connectionStates)
/* close connection, otherwise we take up resource on the other side */
CitusPQFinish(connection);
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
}
}
@ -1633,7 +1584,7 @@ RemoteTransactionIdle(MultiConnection *connection)
* establishment time when necessary.
*/
void
MarkConnectionConnected(MultiConnection *connection, bool newConnection)
MarkConnectionConnected(MultiConnection *connection)
{
connection->connectionState = MULTI_CONNECTION_CONNECTED;
@ -1641,11 +1592,6 @@ MarkConnectionConnected(MultiConnection *connection, bool newConnection)
{
INSTR_TIME_SET_CURRENT(connection->connectionEstablishmentEnd);
}
if (newConnection)
{
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_SUCCEEDED);
}
}

View File

@ -303,8 +303,8 @@ EnsureConnectionPossibilityForRemotePrimaryNodes(void)
* seem to cause any problems as none of the placements that we are
* going to access would be on the new node.
*/
List *remoteNodeList = ActivePrimaryRemoteNodeList(NoLock);
EnsureConnectionPossibilityForNodeList(remoteNodeList);
List *primaryNodeList = ActivePrimaryRemoteNodeList(NoLock);
EnsureConnectionPossibilityForNodeList(primaryNodeList);
}

View File

@ -614,6 +614,16 @@ WaitForSharedConnection(void)
void
InitializeSharedConnectionStats(void)
{
/* on PG 15, we use shmem_request_hook_type */
#if PG_VERSION_NUM < PG_VERSION_15
/* allocate shared memory */
if (!IsUnderPostmaster)
{
RequestAddinShmemSpace(SharedConnectionStatsShmemSize());
}
#endif
prev_shmem_startup_hook = shmem_startup_hook;
shmem_startup_hook = SharedConnectionStatsShmemInit;
}

View File

@ -1,90 +0,0 @@
/*-------------------------------------------------------------------------
*
* citus_deparseutils.c
*
* This file contains common functions used for deparsing PostgreSQL
* statements to their equivalent SQL representation.
*
* Copyright (c) Citus Data, Inc.
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "commands/defrem.h"
#include "utils/builtins.h"
#include "utils/elog.h"
#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/syscache.h"
#include "utils/typcache.h"
#include "pg_version_constants.h"
#include "distributed/deparser.h"
/**
* DefElemOptionToStatement converts a DefElem option to a SQL statement and
* appends it to the given StringInfo buffer.
*
* @param buf The StringInfo buffer to append the SQL statement to.
* @param option The DefElem option to convert to a SQL statement.
* @param optionFormats The option format specification to use for the conversion.
* @param optionFormatsLen The number of option formats in the opt_formats array.
*/
void
DefElemOptionToStatement(StringInfo buf, DefElem *option,
const DefElemOptionFormat *optionFormats,
int optionFormatsLen)
{
const char *name = option->defname;
int i;
for (i = 0; i < optionFormatsLen; i++)
{
if (strcmp(name, optionFormats[i].name) == 0)
{
switch (optionFormats[i].type)
{
case OPTION_FORMAT_STRING:
{
char *value = defGetString(option);
appendStringInfo(buf, optionFormats[i].format, quote_identifier(
value));
break;
}
case OPTION_FORMAT_INTEGER:
{
int32 value = defGetInt32(option);
appendStringInfo(buf, optionFormats[i].format, value);
break;
}
case OPTION_FORMAT_BOOLEAN:
{
bool value = defGetBoolean(option);
appendStringInfo(buf, optionFormats[i].format, value ? "true" :
"false");
break;
}
case OPTION_FORMAT_LITERAL_CSTR:
{
char *value = defGetString(option);
appendStringInfo(buf, optionFormats[i].format, quote_literal_cstr(
value));
break;
}
default:
{
elog(ERROR, "unrecognized option type: %d", optionFormats[i].type);
break;
}
}
}
}
}

View File

@ -74,7 +74,7 @@ AppendGrantRestrictAndCascade(StringInfo buf, GrantStmt *stmt)
void
AppendGrantedByInGrantForRoleSpec(StringInfo buf, RoleSpec *grantor, bool isGrant)
{
if (grantor)
if (isGrant && grantor)
{
appendStringInfo(buf, " GRANTED BY %s", RoleSpecString(grantor, true));
}

View File

@ -83,8 +83,6 @@ static void AppendStorageParametersToString(StringInfo stringBuffer,
static const char * convert_aclright_to_string(int aclright);
static void simple_quote_literal(StringInfo buf, const char *val);
static void AddVacuumParams(ReindexStmt *reindexStmt, StringInfo buffer);
static void process_acl_items(Acl *acl, const char *relationName,
const char *attributeName, List **defs);
/*
@ -260,8 +258,10 @@ pg_get_sequencedef_string(Oid sequenceRelationId)
char *typeName = format_type_be(pgSequenceForm->seqtypid);
char *sequenceDef = psprintf(CREATE_SEQUENCE_COMMAND,
#if (PG_VERSION_NUM >= PG_VERSION_15)
get_rel_persistence(sequenceRelationId) ==
RELPERSISTENCE_UNLOGGED ? "UNLOGGED " : "",
#endif
qualifiedSequenceName,
typeName,
pgSequenceForm->seqincrement, pgSequenceForm->seqmin,
@ -857,10 +857,12 @@ deparse_shard_index_statement(IndexStmt *origStmt, Oid distrelid, int64 shardid,
appendStringInfoString(buffer, ") ");
}
#if PG_VERSION_NUM >= PG_VERSION_15
if (indexStmt->nulls_not_distinct)
{
appendStringInfoString(buffer, "NULLS NOT DISTINCT ");
}
#endif /* PG_VERSION_15 */
if (indexStmt->options != NIL)
{
@ -1112,8 +1114,9 @@ pg_get_indexclusterdef_string(Oid indexRelationId)
/*
* pg_get_table_grants returns a list of sql statements which recreate the
* permissions for a specific table, including attributes privileges.
* permissions for a specific table.
*
* This function is modeled after aclexplode(), don't change too heavily.
*/
List *
pg_get_table_grants(Oid relationId)
@ -1137,8 +1140,6 @@ pg_get_table_grants(Oid relationId)
errmsg("relation with OID %u does not exist",
relationId)));
}
Form_pg_class classForm = (Form_pg_class) GETSTRUCT(classTuple);
AttrNumber nattrs = classForm->relnatts;
Datum aclDatum = SysCacheGetAttr(RELOID, classTuple, Anum_pg_class_relacl,
&isNull);
@ -1166,132 +1167,80 @@ pg_get_table_grants(Oid relationId)
/* iterate through the acl datastructure, emit GRANTs */
Acl *acl = DatumGetAclP(aclDatum);
AclItem *aidat = ACL_DAT(acl);
process_acl_items(acl, relationName, NULL, &defs);
/* if we have a detoasted copy, free it */
if ((Pointer) acl != DatumGetPointer(aclDatum))
pfree(acl);
}
resetStringInfo(&buffer);
/* lookup all attribute level grants */
for (AttrNumber attNum = 1; attNum <= nattrs; attNum++)
{
HeapTuple attTuple = SearchSysCache2(ATTNUM, ObjectIdGetDatum(relationId),
Int16GetDatum(attNum));
if (!HeapTupleIsValid(attTuple))
int offtype = -1;
int i = 0;
while (i < ACL_NUM(acl))
{
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("attribute with OID %u does not exist",
attNum)));
}
AclItem *aidata = NULL;
AclMode priv_bit = 0;
Form_pg_attribute thisAttribute = (Form_pg_attribute) GETSTRUCT(attTuple);
offtype++;
/* ignore dropped columns */
if (thisAttribute->attisdropped)
{
ReleaseSysCache(attTuple);
continue;
}
Datum aclAttDatum = SysCacheGetAttr(ATTNUM, attTuple, Anum_pg_attribute_attacl,
&isNull);
if (!isNull)
{
/* iterate through the acl datastructure, emit GRANTs */
Acl *acl = DatumGetAclP(aclAttDatum);
process_acl_items(acl, relationName, NameStr(thisAttribute->attname), &defs);
/* if we have a detoasted copy, free it */
if ((Pointer) acl != DatumGetPointer(aclAttDatum))
pfree(acl);
}
ReleaseSysCache(attTuple);
}
relation_close(relation, NoLock);
return defs;
}
/*
* Helper function to process ACL items.
* If attributeName is NULL, the function emits table-level GRANT commands;
* otherwise it emits column-level GRANT commands.
* This function was modeled after aclexplode(), previously in pg_get_table_grants().
*/
static void
process_acl_items(Acl *acl, const char *relationName, const char *attributeName,
List **defs)
{
AclItem *aidat = ACL_DAT(acl);
int i = 0;
int offtype = -1;
StringInfoData buffer;
initStringInfo(&buffer);
while (i < ACL_NUM(acl))
{
offtype++;
if (offtype == N_ACL_RIGHTS)
{
offtype = 0;
i++;
if (i >= ACL_NUM(acl)) /* done */
if (offtype == N_ACL_RIGHTS)
{
break;
}
}
AclItem *aidata = &aidat[i];
AclMode priv_bit = 1 << offtype;
if (ACLITEM_GET_PRIVS(*aidata) & priv_bit)
{
const char *roleName = NULL;
const char *withGrant = "";
if (aidata->ai_grantee != 0)
{
roleName = quote_identifier(GetUserNameFromId(aidata->ai_grantee, false));
}
else
{
roleName = "PUBLIC";
offtype = 0;
i++;
if (i >= ACL_NUM(acl)) /* done */
{
break;
}
}
if ((ACLITEM_GET_GOPTIONS(*aidata) & priv_bit) != 0)
{
withGrant = " WITH GRANT OPTION";
}
aidata = &aidat[i];
priv_bit = 1 << offtype;
if (attributeName)
{
appendStringInfo(&buffer, "GRANT %s(%s) ON %s TO %s%s",
convert_aclright_to_string(priv_bit),
quote_identifier(attributeName),
relationName,
roleName,
withGrant);
}
else
if (ACLITEM_GET_PRIVS(*aidata) & priv_bit)
{
const char *roleName = NULL;
const char *withGrant = "";
if (aidata->ai_grantee != 0)
{
HeapTuple htup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(aidata->ai_grantee));
if (HeapTupleIsValid(htup))
{
Form_pg_authid authForm = ((Form_pg_authid) GETSTRUCT(htup));
roleName = quote_identifier(NameStr(authForm->rolname));
ReleaseSysCache(htup);
}
else
{
elog(ERROR, "cache lookup failed for role %u", aidata->ai_grantee);
}
}
else
{
roleName = "PUBLIC";
}
if ((ACLITEM_GET_GOPTIONS(*aidata) & priv_bit) != 0)
{
withGrant = " WITH GRANT OPTION";
}
appendStringInfo(&buffer, "GRANT %s ON %s TO %s%s",
convert_aclright_to_string(priv_bit),
relationName,
roleName,
withGrant);
defs = lappend(defs, pstrdup(buffer.data));
resetStringInfo(&buffer);
}
*defs = lappend(*defs, pstrdup(buffer.data));
resetStringInfo(&buffer);
}
}
resetStringInfo(&buffer);
relation_close(relation, NoLock);
return defs;
/* *INDENT-ON* */
}

View File

@ -1,174 +0,0 @@
#include "postgres.h"
#include "catalog/namespace.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
#include "nodes/print.h"
#include "parser/parse_type.h"
#include "utils/builtins.h"
#include "pg_version_compat.h"
#include "distributed/citus_ruleutils.h"
#include "distributed/deparser.h"
#include "distributed/log_utils.h"
void AppendVarSetValue(StringInfo buf, VariableSetStmt *setStmt);
/*
* AppendVarSetValueDb deparses a VariableSetStmt with VAR_SET_VALUE kind.
* It takes from flatten_set_variable_args in postgres's utils/misc/guc.c,
* however flatten_set_variable_args does not apply correct quoting.
*/
void
AppendVarSetValue(StringInfo buf, VariableSetStmt *setStmt)
{
ListCell *varArgCell = NULL;
ListCell *firstCell = list_head(setStmt->args);
Assert(setStmt->kind == VAR_SET_VALUE);
foreach(varArgCell, setStmt->args)
{
Node *varArgNode = lfirst(varArgCell);
A_Const *varArgConst = NULL;
TypeName *typeName = NULL;
if (IsA(varArgNode, A_Const))
{
varArgConst = (A_Const *) varArgNode;
}
else if (IsA(varArgNode, TypeCast))
{
TypeCast *varArgTypeCast = (TypeCast *) varArgNode;
varArgConst = castNode(A_Const, varArgTypeCast->arg);
typeName = varArgTypeCast->typeName;
}
else
{
elog(ERROR, "unrecognized node type: %d", varArgNode->type);
}
/* don't know how to start SET until we inspect first arg */
if (varArgCell != firstCell)
{
appendStringInfoChar(buf, ',');
}
else if (typeName != NULL)
{
appendStringInfoString(buf, " SET TIME ZONE");
}
else
{
appendStringInfo(buf, " SET %s =", quote_identifier(setStmt->name));
}
Node *value = (Node *) &varArgConst->val;
switch (value->type)
{
case T_Integer:
{
appendStringInfo(buf, " %d", intVal(value));
break;
}
case T_Float:
{
appendStringInfo(buf, " %s", nodeToString(value));
break;
}
case T_String:
{
if (typeName != NULL)
{
/*
* Must be a ConstInterval argument for TIME ZONE. Coerce
* to interval and back to normalize the value and account
* for any typmod.
*/
Oid typoid = InvalidOid;
int32 typmod = -1;
typenameTypeIdAndMod(NULL, typeName, &typoid, &typmod);
Assert(typoid == INTERVALOID);
Datum interval =
DirectFunctionCall3(interval_in,
CStringGetDatum(strVal(value)),
ObjectIdGetDatum(InvalidOid),
Int32GetDatum(typmod));
char *intervalout =
DatumGetCString(DirectFunctionCall1(interval_out,
interval));
appendStringInfo(buf, " INTERVAL '%s'", intervalout);
}
else
{
appendStringInfo(buf, " %s", quote_literal_cstr(strVal(value)));
}
break;
}
default:
{
elog(ERROR, "Unexpected Value type in VAR_SET_VALUE arguments.");
break;
}
}
}
}
/*
* AppendVariableSetDb appends a string representing the VariableSetStmt to a buffer
*/
void
AppendVariableSet(StringInfo buf, VariableSetStmt *setStmt)
{
switch (setStmt->kind)
{
case VAR_SET_VALUE:
{
AppendVarSetValue(buf, setStmt);
break;
}
case VAR_SET_CURRENT:
{
appendStringInfo(buf, " SET %s FROM CURRENT", quote_identifier(
setStmt->name));
break;
}
case VAR_SET_DEFAULT:
{
appendStringInfo(buf, " SET %s TO DEFAULT", quote_identifier(setStmt->name));
break;
}
case VAR_RESET:
{
appendStringInfo(buf, " RESET %s", quote_identifier(setStmt->name));
break;
}
case VAR_RESET_ALL:
{
appendStringInfoString(buf, " RESET ALL");
break;
}
/* VAR_SET_MULTI is a special case for SET TRANSACTION that should not occur here */
case VAR_SET_MULTI:
default:
{
ereport(ERROR, (errmsg("Unable to deparse SET statement")));
break;
}
}
}

View File

@ -1,77 +0,0 @@
/*-------------------------------------------------------------------------
*
* deparse_coment_stmts.c
*
* All routines to deparse comment statements.
*
* Copyright (c), Citus Data, Inc.
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "catalog/namespace.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
#include "parser/parse_type.h"
#include "utils/builtins.h"
#include "utils/elog.h"
#include "pg_version_compat.h"
#include "distributed/citus_ruleutils.h"
#include "distributed/commands.h"
#include "distributed/comment.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#include "distributed/log_utils.h"
const char *ObjectTypeNames[] =
{
[OBJECT_DATABASE] = "DATABASE",
[OBJECT_ROLE] = "ROLE",
[OBJECT_TSCONFIGURATION] = "TEXT SEARCH CONFIGURATION",
[OBJECT_TSDICTIONARY] = "TEXT SEARCH DICTIONARY",
/* When support for propagating comments to new objects is introduced, an entry for each
* statement type should be added to this list. The first element in each entry is the 'object_type' keyword
* that will be included in the 'COMMENT ON <object_type> ..' statement (i.e. DATABASE,). The second element is the type of
* stmt->object, which represents the name of the propagated object.
*/
};
char *
DeparseCommentStmt(Node *node)
{
CommentStmt *stmt = castNode(CommentStmt, node);
StringInfoData str = { 0 };
initStringInfo(&str);
const char *objectName = NULL;
if (IsA(stmt->object, String))
{
objectName = quote_identifier(strVal(stmt->object));
}
else if (IsA(stmt->object, List))
{
objectName = NameListToQuotedString(castNode(List, stmt->object));
}
else
{
ereport(ERROR,
(errcode(ERRCODE_INTERNAL_ERROR),
errmsg("unknown object type")));
}
const char *objectType = ObjectTypeNames[stmt->objtype];
char *comment = stmt->comment != NULL ? quote_literal_cstr(stmt->comment) : "NULL";
appendStringInfo(&str, "COMMENT ON %s %s IS %s;", objectType, objectName, comment);
return str.data;
}

View File

@ -15,53 +15,16 @@
#include "commands/defrem.h"
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
#include "parser/parse_type.h"
#include "utils/builtins.h"
#include "pg_version_compat.h"
#include "distributed/citus_ruleutils.h"
#include "distributed/commands.h"
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#include "distributed/log_utils.h"
static void AppendAlterDatabaseOwnerStmt(StringInfo buf, AlterOwnerStmt *stmt);
static void AppendAlterDatabaseSetStmt(StringInfo buf, AlterDatabaseSetStmt *stmt);
static void AppendAlterDatabaseStmt(StringInfo buf, AlterDatabaseStmt *stmt);
static void AppendCreateDatabaseStmt(StringInfo buf, CreatedbStmt *stmt);
static void AppendDropDatabaseStmt(StringInfo buf, DropdbStmt *stmt);
static void AppendGrantOnDatabaseStmt(StringInfo buf, GrantStmt *stmt);
static void AppendBasicAlterDatabaseOptions(StringInfo buf, AlterDatabaseStmt *stmt);
static void AppendGrantDatabases(StringInfo buf, GrantStmt *stmt);
static void AppendAlterDatabaseSetTablespace(StringInfo buf, DefElem *def, char *dbname);
const DefElemOptionFormat createDatabaseOptionFormats[] = {
{ "owner", " OWNER %s", OPTION_FORMAT_STRING },
{ "template", " TEMPLATE %s", OPTION_FORMAT_STRING },
{ "encoding", " ENCODING %s", OPTION_FORMAT_LITERAL_CSTR },
{ "strategy", " STRATEGY %s", OPTION_FORMAT_LITERAL_CSTR },
{ "locale", " LOCALE %s", OPTION_FORMAT_LITERAL_CSTR },
{ "lc_collate", " LC_COLLATE %s", OPTION_FORMAT_LITERAL_CSTR },
{ "lc_ctype", " LC_CTYPE %s", OPTION_FORMAT_LITERAL_CSTR },
{ "icu_locale", " ICU_LOCALE %s", OPTION_FORMAT_LITERAL_CSTR },
{ "icu_rules", " ICU_RULES %s", OPTION_FORMAT_LITERAL_CSTR },
{ "locale_provider", " LOCALE_PROVIDER %s", OPTION_FORMAT_LITERAL_CSTR },
{ "collation_version", " COLLATION_VERSION %s", OPTION_FORMAT_LITERAL_CSTR },
{ "tablespace", " TABLESPACE %s", OPTION_FORMAT_STRING },
{ "allow_connections", " ALLOW_CONNECTIONS %s", OPTION_FORMAT_BOOLEAN },
{ "connection_limit", " CONNECTION LIMIT %d", OPTION_FORMAT_INTEGER },
{ "is_template", " IS_TEMPLATE %s", OPTION_FORMAT_BOOLEAN }
};
const DefElemOptionFormat alterDatabaseOptionFormats[] = {
{ "is_template", " IS_TEMPLATE %s", OPTION_FORMAT_BOOLEAN },
{ "allow_connections", " ALLOW_CONNECTIONS %s", OPTION_FORMAT_BOOLEAN },
{ "connection_limit", " CONNECTION LIMIT %d", OPTION_FORMAT_INTEGER },
};
char *
DeparseAlterDatabaseOwnerStmt(Node *node)
@ -121,67 +84,52 @@ AppendGrantOnDatabaseStmt(StringInfo buf, GrantStmt *stmt)
}
static void
AppendDefElemConnLimit(StringInfo buf, DefElem *def)
{
appendStringInfo(buf, " CONNECTION LIMIT %ld", (long int) defGetNumeric(def));
}
static void
AppendAlterDatabaseStmt(StringInfo buf, AlterDatabaseStmt *stmt)
{
if (list_length(stmt->options) == 0)
{
elog(ERROR, "got unexpected number of options for ALTER DATABASE");
}
appendStringInfo(buf, "ALTER DATABASE %s ", quote_identifier(stmt->dbname));
if (stmt->options)
{
DefElem *firstOption = linitial(stmt->options);
if (strcmp(firstOption->defname, "tablespace") == 0)
ListCell *cell = NULL;
appendStringInfo(buf, "WITH ");
foreach(cell, stmt->options)
{
AppendAlterDatabaseSetTablespace(buf, firstOption, stmt->dbname);
/* SET tablespace cannot be combined with other options */
return;
DefElem *def = castNode(DefElem, lfirst(cell));
if (strcmp(def->defname, "is_template") == 0)
{
appendStringInfo(buf, "IS_TEMPLATE %s",
quote_literal_cstr(strVal(def->arg)));
}
else if (strcmp(def->defname, "connection_limit") == 0)
{
AppendDefElemConnLimit(buf, def);
}
else if (strcmp(def->defname, "allow_connections") == 0)
{
ereport(ERROR,
errmsg("ALLOW_CONNECTIONS is not supported"));
}
else
{
ereport(ERROR,
errmsg("unrecognized ALTER DATABASE option: %s",
def->defname));
}
}
appendStringInfo(buf, "ALTER DATABASE %s WITH",
quote_identifier(stmt->dbname));
AppendBasicAlterDatabaseOptions(buf, stmt);
}
appendStringInfo(buf, ";");
}
static void
AppendAlterDatabaseSetTablespace(StringInfo buf, DefElem *def, char *dbname)
{
appendStringInfo(buf,
"ALTER DATABASE %s SET TABLESPACE %s",
quote_identifier(dbname), quote_identifier(defGetString(def)));
}
/*
* AppendBasicAlterDatabaseOptions appends basic ALTER DATABASE options to a string buffer.
* Basic options are those that can be appended to the ALTER DATABASE statement
* after the "WITH" keyword.(i.e. ALLOW_CONNECTIONS, CONNECTION LIMIT, IS_TEMPLATE)
* For example, the tablespace option is not a basic option since it is defined via SET keyword.
*
* This function takes a string buffer and an AlterDatabaseStmt as input.
* It appends the basic options to the string buffer.
*
*/
static void
AppendBasicAlterDatabaseOptions(StringInfo buf, AlterDatabaseStmt *stmt)
{
DefElem *def = NULL;
foreach_declared_ptr(def, stmt->options)
{
DefElemOptionToStatement(buf, def, alterDatabaseOptionFormats, lengthof(
alterDatabaseOptionFormats));
}
}
char *
DeparseGrantOnDatabaseStmt(Node *node)
{
@ -211,6 +159,7 @@ DeparseAlterDatabaseStmt(Node *node)
}
#if PG_VERSION_NUM >= PG_VERSION_15
char *
DeparseAlterDatabaseRefreshCollStmt(Node *node)
{
@ -227,131 +176,4 @@ DeparseAlterDatabaseRefreshCollStmt(Node *node)
}
static void
AppendAlterDatabaseSetStmt(StringInfo buf, AlterDatabaseSetStmt *stmt)
{
appendStringInfo(buf, "ALTER DATABASE %s", quote_identifier(stmt->dbname));
VariableSetStmt *varSetStmt = castNode(VariableSetStmt, stmt->setstmt);
AppendVariableSet(buf, varSetStmt);
}
char *
DeparseAlterDatabaseRenameStmt(Node *node)
{
RenameStmt *stmt = (RenameStmt *) node;
StringInfoData str;
initStringInfo(&str);
appendStringInfo(&str, "ALTER DATABASE %s RENAME TO %s",
quote_identifier(stmt->subname),
quote_identifier(stmt->newname));
return str.data;
}
char *
DeparseAlterDatabaseSetStmt(Node *node)
{
AlterDatabaseSetStmt *stmt = castNode(AlterDatabaseSetStmt, node);
StringInfoData str = { 0 };
initStringInfo(&str);
AppendAlterDatabaseSetStmt(&str, stmt);
return str.data;
}
static void
AppendCreateDatabaseStmt(StringInfo buf, CreatedbStmt *stmt)
{
/*
* Make sure that we don't try to deparse something that this
* function doesn't expect.
*
* This is also useful to throw an error for unsupported CREATE
* DATABASE options when the command is issued from non-main dbs
* because we use the same function to deparse CREATE DATABASE
* commands there too.
*/
EnsureSupportedCreateDatabaseCommand(stmt);
appendStringInfo(buf,
"CREATE DATABASE %s",
quote_identifier(stmt->dbname));
DefElem *option = NULL;
foreach_declared_ptr(option, stmt->options)
{
DefElemOptionToStatement(buf, option, createDatabaseOptionFormats,
lengthof(createDatabaseOptionFormats));
}
}
char *
DeparseCreateDatabaseStmt(Node *node)
{
CreatedbStmt *stmt = castNode(CreatedbStmt, node);
StringInfoData str = { 0 };
initStringInfo(&str);
AppendCreateDatabaseStmt(&str, stmt);
return str.data;
}
static void
AppendDropDatabaseStmt(StringInfo buf, DropdbStmt *stmt)
{
char *ifExistsStatement = stmt->missing_ok ? "IF EXISTS" : "";
appendStringInfo(buf,
"DROP DATABASE %s %s",
ifExistsStatement,
quote_identifier(stmt->dbname));
if (list_length(stmt->options) > 1)
{
/* FORCE is the only option that can be provided for this command */
elog(ERROR, "got unexpected number of options for DROP DATABASE");
}
else if (list_length(stmt->options) == 1)
{
DefElem *option = linitial(stmt->options);
appendStringInfo(buf, " WITH ( ");
if (strcmp(option->defname, "force") == 0)
{
appendStringInfo(buf, "FORCE");
}
else
{
/* FORCE is the only option that can be provided for this command */
ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR),
errmsg("unrecognized DROP DATABASE option \"%s\"",
option->defname)));
}
appendStringInfo(buf, " )");
}
}
char *
DeparseDropDatabaseStmt(Node *node)
{
DropdbStmt *stmt = castNode(DropdbStmt, node);
StringInfoData str = { 0 };
initStringInfo(&str);
AppendDropDatabaseStmt(&str, stmt);
return str.data;
}
#endif

View File

@ -62,6 +62,7 @@ static void AppendDefElemRows(StringInfo buf, DefElem *def);
static void AppendDefElemSet(StringInfo buf, DefElem *def);
static void AppendDefElemSupport(StringInfo buf, DefElem *def);
static void AppendVarSetValue(StringInfo buf, VariableSetStmt *setStmt);
static void AppendRenameFunctionStmt(StringInfo buf, RenameStmt *stmt);
static void AppendAlterFunctionSchemaStmt(StringInfo buf, AlterObjectSchemaStmt *stmt);
static void AppendAlterFunctionOwnerStmt(StringInfo buf, AlterOwnerStmt *stmt);
@ -300,6 +301,164 @@ AppendDefElemSupport(StringInfo buf, DefElem *def)
}
/*
* AppendVariableSet appends a string representing the VariableSetStmt to a buffer
*/
void
AppendVariableSet(StringInfo buf, VariableSetStmt *setStmt)
{
switch (setStmt->kind)
{
case VAR_SET_VALUE:
{
AppendVarSetValue(buf, setStmt);
break;
}
case VAR_SET_CURRENT:
{
appendStringInfo(buf, " SET %s FROM CURRENT", quote_identifier(
setStmt->name));
break;
}
case VAR_SET_DEFAULT:
{
appendStringInfo(buf, " SET %s TO DEFAULT", quote_identifier(setStmt->name));
break;
}
case VAR_RESET:
{
appendStringInfo(buf, " RESET %s", quote_identifier(setStmt->name));
break;
}
case VAR_RESET_ALL:
{
appendStringInfoString(buf, " RESET ALL");
break;
}
/* VAR_SET_MULTI is a special case for SET TRANSACTION that should not occur here */
case VAR_SET_MULTI:
default:
{
ereport(ERROR, (errmsg("Unable to deparse SET statement")));
break;
}
}
}
/*
* AppendVarSetValue deparses a VariableSetStmt with VAR_SET_VALUE kind.
* It takes from flatten_set_variable_args in postgres's utils/misc/guc.c,
* however flatten_set_variable_args does not apply correct quoting.
*/
static void
AppendVarSetValue(StringInfo buf, VariableSetStmt *setStmt)
{
ListCell *varArgCell = NULL;
ListCell *firstCell = list_head(setStmt->args);
Assert(setStmt->kind == VAR_SET_VALUE);
foreach(varArgCell, setStmt->args)
{
Node *varArgNode = lfirst(varArgCell);
A_Const *varArgConst = NULL;
TypeName *typeName = NULL;
if (IsA(varArgNode, A_Const))
{
varArgConst = (A_Const *) varArgNode;
}
else if (IsA(varArgNode, TypeCast))
{
TypeCast *varArgTypeCast = (TypeCast *) varArgNode;
varArgConst = castNode(A_Const, varArgTypeCast->arg);
typeName = varArgTypeCast->typeName;
}
else
{
elog(ERROR, "unrecognized node type: %d", varArgNode->type);
}
/* don't know how to start SET until we inspect first arg */
if (varArgCell != firstCell)
{
appendStringInfoChar(buf, ',');
}
else if (typeName != NULL)
{
appendStringInfoString(buf, " SET TIME ZONE");
}
else
{
appendStringInfo(buf, " SET %s =", quote_identifier(setStmt->name));
}
Node *value = (Node *) &varArgConst->val;
switch (value->type)
{
case T_Integer:
{
appendStringInfo(buf, " %d", intVal(value));
break;
}
case T_Float:
{
appendStringInfo(buf, " %s", strVal(value));
break;
}
case T_String:
{
if (typeName != NULL)
{
/*
* Must be a ConstInterval argument for TIME ZONE. Coerce
* to interval and back to normalize the value and account
* for any typmod.
*/
Oid typoid = InvalidOid;
int32 typmod = -1;
typenameTypeIdAndMod(NULL, typeName, &typoid, &typmod);
Assert(typoid == INTERVALOID);
Datum interval =
DirectFunctionCall3(interval_in,
CStringGetDatum(strVal(value)),
ObjectIdGetDatum(InvalidOid),
Int32GetDatum(typmod));
char *intervalout =
DatumGetCString(DirectFunctionCall1(interval_out,
interval));
appendStringInfo(buf, " INTERVAL '%s'", intervalout);
}
else
{
appendStringInfo(buf, " %s", quote_literal_cstr(strVal(
value)));
}
break;
}
default:
{
elog(ERROR, "Unexpected Value type in VAR_SET_VALUE arguments.");
break;
}
}
}
}
/*
* DeparseRenameFunctionStmt builds and returns a string representing the RenameStmt
*/

View File

@ -71,7 +71,7 @@ AppendRoleList(StringInfo buf, List *roleList)
{
Node *roleNode = (Node *) lfirst(cell);
Assert(IsA(roleNode, RoleSpec) || IsA(roleNode, AccessPriv));
const char *rolename = NULL;
char const *rolename = NULL;
if (IsA(roleNode, RoleSpec))
{
rolename = RoleSpecString((RoleSpec *) roleNode, true);
@ -83,27 +83,3 @@ AppendRoleList(StringInfo buf, List *roleList)
}
}
}
static void
AppendReassignOwnedStmt(StringInfo buf, ReassignOwnedStmt *stmt)
{
appendStringInfo(buf, "REASSIGN OWNED BY ");
AppendRoleList(buf, stmt->roles);
const char *newRoleName = RoleSpecString(stmt->newrole, true);
appendStringInfo(buf, " TO %s", newRoleName);
}
char *
DeparseReassignOwnedStmt(Node *node)
{
ReassignOwnedStmt *stmt = castNode(ReassignOwnedStmt, node);
StringInfoData buf = { 0 };
initStringInfo(&buf);
AppendReassignOwnedStmt(&buf, stmt);
return buf.data;
}

View File

@ -32,6 +32,7 @@
static void AppendCreatePublicationStmt(StringInfo buf, CreatePublicationStmt *stmt,
bool whereClauseNeedsTransform,
bool includeLocalTables);
#if (PG_VERSION_NUM >= PG_VERSION_15)
static bool AppendPublicationObjects(StringInfo buf, List *publicationObjects,
bool whereClauseNeedsTransform,
bool includeLocalTables);
@ -39,6 +40,10 @@ static void AppendWhereClauseExpression(StringInfo buf, RangeVar *tableName,
Node *whereClause,
bool whereClauseNeedsTransform);
static void AppendAlterPublicationAction(StringInfo buf, AlterPublicationAction action);
#else
static bool AppendTables(StringInfo buf, List *tables, bool includeLocalTables);
static void AppendDefElemAction(StringInfo buf, DefElemAction action);
#endif
static bool AppendAlterPublicationStmt(StringInfo buf, AlterPublicationStmt *stmt,
bool whereClauseNeedsTransform,
bool includeLocalTables);
@ -103,6 +108,7 @@ AppendCreatePublicationStmt(StringInfo buf, CreatePublicationStmt *stmt,
{
appendStringInfoString(buf, " FOR ALL TABLES");
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
else if (stmt->pubobjects != NIL)
{
bool hasObjects = false;
@ -140,6 +146,32 @@ AppendCreatePublicationStmt(StringInfo buf, CreatePublicationStmt *stmt,
includeLocalTables);
}
}
#else
else if (stmt->tables != NIL)
{
bool hasTables = false;
RangeVar *rangeVar = NULL;
/*
* Check whether there are tables to propagate, mainly to know whether
* we should include "FOR".
*/
foreach_declared_ptr(rangeVar, stmt->tables)
{
if (includeLocalTables || IsCitusTableRangeVar(rangeVar, NoLock, false))
{
hasTables = true;
break;
}
}
if (hasTables)
{
appendStringInfoString(buf, " FOR");
AppendTables(buf, stmt->tables, includeLocalTables);
}
}
#endif
if (stmt->options != NIL)
{
@ -150,6 +182,8 @@ AppendCreatePublicationStmt(StringInfo buf, CreatePublicationStmt *stmt,
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* AppendPublicationObjects appends a string representing a list of publication
* objects to a buffer.
@ -286,6 +320,57 @@ AppendWhereClauseExpression(StringInfo buf, RangeVar *tableName,
}
#else
/*
* AppendPublicationObjects appends a string representing a list of publication
* objects to a buffer.
*
* For instance: TABLE users, departments
*/
static bool
AppendTables(StringInfo buf, List *tables, bool includeLocalTables)
{
RangeVar *rangeVar = NULL;
bool appendedObject = false;
foreach_declared_ptr(rangeVar, tables)
{
if (!includeLocalTables &&
!IsCitusTableRangeVar(rangeVar, NoLock, false))
{
/* do not propagate local tables */
continue;
}
char *schemaName = rangeVar->schemaname;
char *tableName = rangeVar->relname;
if (schemaName != NULL)
{
/* qualified table name */
appendStringInfo(buf, "%s %s",
appendedObject ? "," : " TABLE",
quote_qualified_identifier(schemaName, tableName));
}
else
{
/* unqualified table name */
appendStringInfo(buf, "%s %s",
appendedObject ? "," : " TABLE",
quote_identifier(tableName));
}
appendedObject = true;
}
return appendedObject;
}
#endif
/*
* DeparseAlterPublicationSchemaStmt builds and returns a string representing
* an AlterPublicationStmt.
@ -354,12 +439,19 @@ AppendAlterPublicationStmt(StringInfo buf, AlterPublicationStmt *stmt,
return true;
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
AppendAlterPublicationAction(buf, stmt->action);
return AppendPublicationObjects(buf, stmt->pubobjects, whereClauseNeedsTransform,
includeLocalTables);
#else
AppendDefElemAction(buf, stmt->tableAction);
return AppendTables(buf, stmt->tables, includeLocalTables);
#endif
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* AppendAlterPublicationAction appends a string representing an AlterPublicationAction
* to a buffer.
@ -395,6 +487,46 @@ AppendAlterPublicationAction(StringInfo buf, AlterPublicationAction action)
}
#else
/*
* AppendDefElemAction appends a string representing a DefElemAction
* to a buffer.
*/
static void
AppendDefElemAction(StringInfo buf, DefElemAction action)
{
switch (action)
{
case DEFELEM_ADD:
{
appendStringInfoString(buf, " ADD");
break;
}
case DEFELEM_DROP:
{
appendStringInfoString(buf, " DROP");
break;
}
case DEFELEM_SET:
{
appendStringInfoString(buf, " SET");
break;
}
default:
{
ereport(ERROR, (errmsg("unrecognized publication action: %d", action)));
}
}
}
#endif
/*
* DeparseDropPublicationStmt builds and returns a string representing the DropStmt
*/
@ -519,7 +651,11 @@ AppendPublicationOptions(StringInfo stringBuffer, List *optionList)
appendStringInfo(stringBuffer, "%s = ",
quote_identifier(optionName));
#if (PG_VERSION_NUM >= PG_VERSION_15)
if (valueType == T_Integer || valueType == T_Float || valueType == T_Boolean)
#else
if (valueType == T_Integer || valueType == T_Float)
#endif
{
/* string escaping is unnecessary for numeric types and can cause issues */
appendStringInfo(stringBuffer, "%s", optionValue);

View File

@ -202,51 +202,17 @@ DeparseCreateRoleStmt(Node *node)
}
static void
AppendSysIdStatement(StringInfo buf, ListCell *optionCell)
{
DefElem *option = (DefElem *) lfirst(optionCell);
if (strcmp(option->defname, "sysid") == 0)
{
appendStringInfo(buf, " SYSID %d", intVal(option->arg));
}
}
/*
* AppendInlinePriviliges generates the string representation for the inline
* privileges of the role in create statement and appends it to the buffer.
* AppendCreateRoleStmt generates the string representation of the
* CreateRoleStmt and appends it to the buffer.
*/
static void
AppendInlinePriviliges(StringInfo buf, ListCell *optionCell)
AppendCreateRoleStmt(StringInfo buf, CreateRoleStmt *stmt)
{
DefElem *option = (DefElem *) lfirst(optionCell);
ListCell *optionCell = NULL;
if (strcmp(option->defname, "adminmembers") == 0)
{
appendStringInfo(buf, " ADMIN ");
AppendRoleList(buf, (List *) option->arg);
}
else if (strcmp(option->defname, "rolemembers") == 0)
{
appendStringInfo(buf, " ROLE ");
AppendRoleList(buf, (List *) option->arg);
}
else if (strcmp(option->defname, "addroleto") == 0)
{
appendStringInfo(buf, " IN ROLE ");
AppendRoleList(buf, (List *) option->arg);
}
}
appendStringInfo(buf, "CREATE ");
/*
* AppendStatementType generates the string representation for the statement
* type (role, user or group) in alter/create statement and appends it to the buffer.
*/
static void
AppendStatementType(StringInfo buf, CreateRoleStmt *stmt)
{
switch (stmt->stmt_type)
{
case ROLESTMT_ROLE:
@ -267,29 +233,34 @@ AppendStatementType(StringInfo buf, CreateRoleStmt *stmt)
break;
}
}
}
/*
* AppendCreateRoleStmt generates the string representation of the
* CreateRoleStmt and appends it to the buffer.
*/
static void
AppendCreateRoleStmt(StringInfo buf, CreateRoleStmt *stmt)
{
ListCell *optionCell = NULL;
appendStringInfo(buf, "CREATE ");
AppendStatementType(buf, stmt);
appendStringInfo(buf, "%s", quote_identifier(stmt->role));
foreach(optionCell, stmt->options)
{
AppendRoleOption(buf, optionCell);
AppendInlinePriviliges(buf, optionCell);
AppendSysIdStatement(buf, optionCell);
DefElem *option = (DefElem *) lfirst(optionCell);
if (strcmp(option->defname, "sysid") == 0)
{
appendStringInfo(buf, " SYSID %d", intVal(option->arg));
}
else if (strcmp(option->defname, "adminmembers") == 0)
{
appendStringInfo(buf, " ADMIN ");
AppendRoleList(buf, (List *) option->arg);
}
else if (strcmp(option->defname, "rolemembers") == 0)
{
appendStringInfo(buf, " ROLE ");
AppendRoleList(buf, (List *) option->arg);
}
else if (strcmp(option->defname, "addroleto") == 0)
{
appendStringInfo(buf, " IN ROLE ");
AppendRoleList(buf, (List *) option->arg);
}
}
}
@ -356,22 +327,6 @@ AppendRoleList(StringInfo buf, List *roleList)
}
char *
DeparseRenameRoleStmt(Node *node)
{
RenameStmt *stmt = castNode(RenameStmt, node);
StringInfoData str = { 0 };
initStringInfo(&str);
Assert(stmt->renameType == OBJECT_ROLE);
appendStringInfo(&str, "ALTER ROLE %s RENAME TO %s;",
quote_identifier(stmt->subname), quote_identifier(stmt->newname));
return str.data;
}
/*
* DeparseGrantRoleStmt builds and returns a string representing of the
* GrantRoleStmt for application on a remote server.
@ -411,16 +366,6 @@ AppendRevokeAdminOptionFor(StringInfo buf, GrantRoleStmt *stmt)
appendStringInfo(buf, "ADMIN OPTION FOR ");
break;
}
else if (strcmp(opt->defname, "inherit") == 0)
{
appendStringInfo(buf, "INHERIT OPTION FOR ");
break;
}
else if (strcmp(opt->defname, "set") == 0)
{
appendStringInfo(buf, "SET OPTION FOR ");
break;
}
}
}
#else
@ -438,29 +383,16 @@ AppendGrantWithAdminOption(StringInfo buf, GrantRoleStmt *stmt)
if (stmt->is_grant)
{
#if PG_VERSION_NUM >= PG_VERSION_16
int opt_count = 0;
DefElem *opt = NULL;
foreach_declared_ptr(opt, stmt->opt)
{
bool admin_option = false;
char *optval = defGetString(opt);
bool option_value = false;
if (parse_bool(optval, &option_value))
if (strcmp(opt->defname, "admin") == 0 &&
parse_bool(optval, &admin_option) && admin_option)
{
opt_count++;
char *prefix = opt_count > 1 ? "," : " WITH";
if (strcmp(opt->defname, "inherit") == 0)
{
appendStringInfo(buf, "%s INHERIT %s", prefix, option_value ? "TRUE" :
"FALSE");
}
else if (strcmp(opt->defname, "admin") == 0 && option_value)
{
appendStringInfo(buf, "%s ADMIN OPTION", prefix);
}
else if (strcmp(opt->defname, "set") == 0 && !option_value)
{
appendStringInfo(buf, "%s SET FALSE", prefix);
}
appendStringInfo(buf, " WITH ADMIN OPTION");
break;
}
}
#else
@ -488,6 +420,7 @@ AppendGrantRoleStmt(StringInfo buf, GrantRoleStmt *stmt)
AppendGrantWithAdminOption(buf, stmt);
AppendGrantedByInGrantForRoleSpec(buf, stmt->grantor, stmt->is_grant);
AppendGrantRestrictAndCascadeForRoleSpec(buf, stmt->behavior, stmt->is_grant);
AppendGrantedByInGrantForRoleSpec(buf, stmt->grantor, stmt->is_grant);
appendStringInfo(buf, ";");
}

View File

@ -10,16 +10,37 @@
#include "postgres.h"
#include "catalog/namespace.h"
#include "nodes/parsenodes.h"
#include "utils/builtins.h"
#include "distributed/deparser.h"
static void
BeginSecLabel(StringInfo buf, SecLabelStmt *stmt)
static void AppendSecLabelStmt(StringInfo buf, SecLabelStmt *stmt);
/*
* DeparseSecLabelStmt builds and returns a string representing of the
* SecLabelStmt for application on a remote server.
*/
char *
DeparseSecLabelStmt(Node *node)
{
SecLabelStmt *secLabelStmt = castNode(SecLabelStmt, node);
StringInfoData buf = { 0 };
initStringInfo(&buf);
AppendSecLabelStmt(&buf, secLabelStmt);
return buf.data;
}
/*
* AppendSecLabelStmt generates the string representation of the
* SecLabelStmt and appends it to the buffer.
*/
static void
AppendSecLabelStmt(StringInfo buf, SecLabelStmt *stmt)
{
initStringInfo(buf);
appendStringInfoString(buf, "SECURITY LABEL ");
if (stmt->provider != NULL)
@ -28,84 +49,31 @@ BeginSecLabel(StringInfo buf, SecLabelStmt *stmt)
}
appendStringInfoString(buf, "ON ");
}
static void
EndSecLabel(StringInfo buf, SecLabelStmt *stmt)
{
appendStringInfo(buf, "IS %s", (stmt->label != NULL) ?
quote_literal_cstr(stmt->label) : "NULL");
}
/*
* DeparseRoleSecLabelStmt builds and returns a string representation of the
* SecLabelStmt for application on a remote server. The SecLabelStmt is for
* a role object.
*/
char *
DeparseRoleSecLabelStmt(Node *node)
{
SecLabelStmt *secLabelStmt = castNode(SecLabelStmt, node);
char *role_name = strVal(secLabelStmt->object);
StringInfoData buf = { 0 };
BeginSecLabel(&buf, secLabelStmt);
appendStringInfo(&buf, "ROLE %s ", quote_identifier(role_name));
EndSecLabel(&buf, secLabelStmt);
return buf.data;
}
/*
* DeparseTableSecLabelStmt builds and returns a string representation of the
* SecLabelStmt for application on a remote server. The SecLabelStmt is for a
* table.
*/
char *
DeparseTableSecLabelStmt(Node *node)
{
SecLabelStmt *secLabelStmt = castNode(SecLabelStmt, node);
List *names = (List *) secLabelStmt->object;
StringInfoData buf = { 0 };
BeginSecLabel(&buf, secLabelStmt);
appendStringInfo(&buf, "TABLE %s", quote_identifier(strVal(linitial(names))));
if (list_length(names) > 1)
switch (stmt->objtype)
{
appendStringInfo(&buf, ".%s", quote_identifier(strVal(lsecond(names))));
case OBJECT_ROLE:
{
appendStringInfo(buf, "ROLE %s ", quote_identifier(strVal(stmt->object)));
break;
}
/* normally, we shouldn't reach this */
default:
{
ereport(ERROR, (errmsg("unsupported security label statement for"
" deparsing")));
}
}
appendStringInfoString(&buf, " ");
EndSecLabel(&buf, secLabelStmt);
return buf.data;
}
appendStringInfoString(buf, "IS ");
/*
* DeparseColumnSecLabelStmt builds and returns a string representation of the
* SecLabelStmt for application on a remote server. The SecLabelStmt is for a
* column of a distributed table.
*/
char *
DeparseColumnSecLabelStmt(Node *node)
{
SecLabelStmt *secLabelStmt = castNode(SecLabelStmt, node);
List *names = (List *) secLabelStmt->object;
StringInfoData buf = { 0 };
BeginSecLabel(&buf, secLabelStmt);
appendStringInfo(&buf, "COLUMN %s.%s",
quote_identifier(strVal(linitial(names))),
quote_identifier(strVal(lsecond(names))));
if (list_length(names) > 2)
if (stmt->label != NULL)
{
appendStringInfo(&buf, ".%s", quote_identifier(strVal(lthird(names))));
appendStringInfo(buf, "%s", quote_literal_cstr(stmt->label));
}
else
{
appendStringInfoString(buf, "NULL");
}
appendStringInfoString(&buf, " ");
EndSecLabel(&buf, secLabelStmt);
return buf.data;
}

View File

@ -28,7 +28,9 @@ static void AppendSequenceNameList(StringInfo buf, List *objects, ObjectType obj
static void AppendRenameSequenceStmt(StringInfo buf, RenameStmt *stmt);
static void AppendAlterSequenceSchemaStmt(StringInfo buf, AlterObjectSchemaStmt *stmt);
static void AppendAlterSequenceOwnerStmt(StringInfo buf, AlterTableStmt *stmt);
#if (PG_VERSION_NUM >= PG_VERSION_15)
static void AppendAlterSequencePersistenceStmt(StringInfo buf, AlterTableStmt *stmt);
#endif
static void AppendGrantOnSequenceStmt(StringInfo buf, GrantStmt *stmt);
static void AppendGrantOnSequenceSequences(StringInfo buf, GrantStmt *stmt);
@ -260,6 +262,8 @@ AppendAlterSequenceOwnerStmt(StringInfo buf, AlterTableStmt *stmt)
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* DeparseAlterSequencePersistenceStmt builds and returns a string representing
* the AlterTableStmt consisting of changing the persistence of a sequence
@ -345,6 +349,9 @@ AppendAlterSequencePersistenceStmt(StringInfo buf, AlterTableStmt *stmt)
}
#endif
/*
* DeparseGrantOnSequenceStmt builds and returns a string representing the GrantOnSequenceStmt
*/

View File

@ -193,10 +193,12 @@ AppendAlterTableCmdConstraint(StringInfo buf, Constraint *constraint,
{
appendStringInfoString(buf, " UNIQUE");
#if (PG_VERSION_NUM >= PG_VERSION_15)
if (constraint->nulls_not_distinct == true)
{
appendStringInfoString(buf, " NULLS NOT DISTINCT");
}
#endif
}
if (subtype == AT_AddConstraint)

View File

@ -395,6 +395,68 @@ DeparseAlterTextSearchDictionarySchemaStmt(Node *node)
}
/*
* DeparseTextSearchConfigurationCommentStmt returns the sql statement representing
* COMMENT ON TEXT SEARCH CONFIGURATION ... IS ...
*/
char *
DeparseTextSearchConfigurationCommentStmt(Node *node)
{
CommentStmt *stmt = castNode(CommentStmt, node);
Assert(stmt->objtype == OBJECT_TSCONFIGURATION);
StringInfoData buf = { 0 };
initStringInfo(&buf);
appendStringInfo(&buf, "COMMENT ON TEXT SEARCH CONFIGURATION %s IS ",
NameListToQuotedString(castNode(List, stmt->object)));
if (stmt->comment == NULL)
{
appendStringInfoString(&buf, "NULL");
}
else
{
appendStringInfoString(&buf, quote_literal_cstr(stmt->comment));
}
appendStringInfoString(&buf, ";");
return buf.data;
}
/*
* DeparseTextSearchDictionaryCommentStmt returns the sql statement representing
* COMMENT ON TEXT SEARCH DICTIONARY ... IS ...
*/
char *
DeparseTextSearchDictionaryCommentStmt(Node *node)
{
CommentStmt *stmt = castNode(CommentStmt, node);
Assert(stmt->objtype == OBJECT_TSDICTIONARY);
StringInfoData buf = { 0 };
initStringInfo(&buf);
appendStringInfo(&buf, "COMMENT ON TEXT SEARCH DICTIONARY %s IS ",
NameListToQuotedString(castNode(List, stmt->object)));
if (stmt->comment == NULL)
{
appendStringInfoString(&buf, "NULL");
}
else
{
appendStringInfoString(&buf, quote_literal_cstr(stmt->comment));
}
appendStringInfoString(&buf, ";");
return buf.data;
}
/*
* AppendStringInfoTokentypeList specializes in adding a comma separated list of
* token_tyoe's to TEXT SEARCH CONFIGURATION commands

View File

@ -19,7 +19,11 @@
#include "distributed/deparser.h"
#include "distributed/listutils.h"
#if (PG_VERSION_NUM >= PG_VERSION_15)
static void QualifyPublicationObjects(List *publicationObjects);
#else
static void QualifyTables(List *tables);
#endif
static void QualifyPublicationRangeVar(RangeVar *publication);
@ -32,10 +36,16 @@ QualifyCreatePublicationStmt(Node *node)
{
CreatePublicationStmt *stmt = castNode(CreatePublicationStmt, node);
#if (PG_VERSION_NUM >= PG_VERSION_15)
QualifyPublicationObjects(stmt->pubobjects);
#else
QualifyTables(stmt->tables);
#endif
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* QualifyPublicationObjects ensures all table names in a list of
* publication objects are fully qualified.
@ -58,6 +68,26 @@ QualifyPublicationObjects(List *publicationObjects)
}
#else
/*
* QualifyTables ensures all table names in a list are fully qualified.
*/
static void
QualifyTables(List *tables)
{
RangeVar *rangeVar = NULL;
foreach_declared_ptr(rangeVar, tables)
{
QualifyPublicationRangeVar(rangeVar);
}
}
#endif
/*
* QualifyPublicationObjects ensures all table names in a list of
* publication objects are fully qualified.
@ -67,7 +97,11 @@ QualifyAlterPublicationStmt(Node *node)
{
AlterPublicationStmt *stmt = castNode(AlterPublicationStmt, node);
#if (PG_VERSION_NUM >= PG_VERSION_15)
QualifyPublicationObjects(stmt->pubobjects);
#else
QualifyTables(stmt->tables);
#endif
}

View File

@ -52,6 +52,8 @@ QualifyAlterSequenceOwnerStmt(Node *node)
}
#if (PG_VERSION_NUM >= PG_VERSION_15)
/*
* QualifyAlterSequencePersistenceStmt transforms a
* ALTER SEQUENCE .. SET LOGGED/UNLOGGED
@ -78,6 +80,9 @@ QualifyAlterSequencePersistenceStmt(Node *node)
}
#endif
/*
* QualifyAlterSequenceSchemaStmt transforms a
* ALTER SEQUENCE .. SET SCHEMA ..

File diff suppressed because it is too large Load Diff

View File

@ -67,6 +67,7 @@
#include "parser/parse_node.h"
#include "parser/parse_agg.h"
#include "parser/parse_func.h"
#include "parser/parse_node.h"
#include "parser/parse_oper.h"
#include "parser/parse_relation.h"
#include "parser/parser.h"
@ -122,18 +123,16 @@ typedef struct
{
StringInfo buf; /* output buffer to append to */
List *namespaces; /* List of deparse_namespace nodes */
TupleDesc resultDesc; /* if top level of a view, the view's tupdesc */
List *targetList; /* Current query level's SELECT targetlist */
List *windowClause; /* Current query level's WINDOW clause */
List *windowTList; /* targetlist for resolving WINDOW clause */
int prettyFlags; /* enabling of pretty-print functions */
int wrapColumn; /* max line length, or -1 for no limit */
int indentLevel; /* current indent level for prettyprint */
bool varprefix; /* true to print prefixes on Vars */
Oid distrelid; /* the distributed table being modified, if valid */
int64 shardid; /* a distributed table's shardid, if positive */
bool colNamesVisible; /* do we care about output column names? */
bool inGroupBy; /* deparsing GROUP BY clause? */
bool varInOrderBy; /* deparsing simple Var in ORDER BY? */
ParseExprKind special_exprkind; /* set only for exprkinds needing special
* handling */
Bitmapset *appendparents; /* if not null, map child Vars of these relids
* back to the parent rel */
} deparse_context;
@ -365,19 +364,27 @@ static void get_query_def_extended(Query *query, StringInfo buf,
int startIndent);
static void get_values_def(List *values_lists, deparse_context *context);
static void get_with_clause(Query *query, deparse_context *context);
static void get_select_query_def(Query *query, deparse_context *context);
static void get_insert_query_def(Query *query, deparse_context *context);
static void get_update_query_def(Query *query, deparse_context *context);
static void get_select_query_def(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_insert_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_update_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_merge_query_def(Query *query, deparse_context *context);
static void get_update_query_targetlist_def(Query *query, List *targetList,
deparse_context *context,
RangeTblEntry *rte);
static void get_delete_query_def(Query *query, deparse_context *context);
static void get_merge_query_def(Query *query, deparse_context *context);
static void get_delete_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_target_list(List *targetList, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
bool force_colno,
deparse_context *context);
@ -455,7 +462,7 @@ static char *generate_fragment_name(char *schemaName, char *tableName);
static char *generate_function_name(Oid funcid, int nargs,
List *argnames, Oid *argtypes,
bool has_variadic, bool *use_variadic_p,
bool inGroupBy);
ParseExprKind special_exprkind);
static List *get_insert_column_names_list(List *targetList, StringInfo buf, deparse_context *context, RangeTblEntry *rte);
#define only_marker(rte) ((rte)->inh ? "" : "ONLY ")
@ -629,16 +636,13 @@ pg_get_rule_expr(Node *expression)
context.buf = buffer;
context.namespaces = NIL;
context.resultDesc = NULL;
context.targetList = NIL;
context.windowClause = NIL;
context.windowTList = NIL;
context.varprefix = false;
context.prettyFlags = 0;
context.wrapColumn = WRAP_COLUMN_DEFAULT;
context.indentLevel = 0;
context.colNamesVisible = true;
context.inGroupBy = false;
context.varInOrderBy = false;
context.special_exprkind = EXPR_KIND_NONE;
context.distrelid = InvalidOid;
context.shardid = INVALID_SHARD_ID;
@ -2062,17 +2066,14 @@ get_query_def_extended(Query *query, StringInfo buf, List *parentnamespace,
context.buf = buf;
context.namespaces = lcons(&dpns, list_copy(parentnamespace));
context.resultDesc = NULL;
context.targetList = NIL;
context.windowClause = NIL;
context.windowTList = NIL;
context.varprefix = (parentnamespace != NIL ||
list_length(query->rtable) != 1);
context.prettyFlags = prettyFlags;
context.wrapColumn = wrapColumn;
context.indentLevel = startIndent;
context.colNamesVisible = true;
context.inGroupBy = false;
context.varInOrderBy = false;
context.special_exprkind = EXPR_KIND_NONE;
context.appendparents = NULL;
context.distrelid = distrelid;
context.shardid = shardid;
@ -2082,21 +2083,19 @@ get_query_def_extended(Query *query, StringInfo buf, List *parentnamespace,
switch (query->commandType)
{
case CMD_SELECT:
/* We set context.resultDesc only if it's a SELECT */
context.resultDesc = resultDesc;
get_select_query_def(query, &context);
get_select_query_def(query, &context, resultDesc, colNamesVisible);
break;
case CMD_UPDATE:
get_update_query_def(query, &context);
get_update_query_def(query, &context, colNamesVisible);
break;
case CMD_INSERT:
get_insert_query_def(query, &context);
get_insert_query_def(query, &context, colNamesVisible);
break;
case CMD_DELETE:
get_delete_query_def(query, &context);
get_delete_query_def(query, &context, colNamesVisible);
break;
case CMD_MERGE:
@ -2308,18 +2307,23 @@ get_with_clause(Query *query, deparse_context *context)
* ----------
*/
static void
get_select_query_def(Query *query, deparse_context *context)
get_select_query_def(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
List *save_windowclause;
List *save_windowtlist;
bool force_colno;
ListCell *l;
/* Insert the WITH clause if given */
get_with_clause(query, context);
/* Subroutines may need to consult the SELECT targetlist and windowClause */
context->targetList = query->targetList;
/* Set up context for possible window functions */
save_windowclause = context->windowClause;
context->windowClause = query->windowClause;
save_windowtlist = context->windowTList;
context->windowTList = query->targetList;
/*
* If the Query node has a setOperations tree, then it's the top level of
@ -2328,13 +2332,14 @@ get_select_query_def(Query *query, deparse_context *context)
*/
if (query->setOperations)
{
get_setop_query(query->setOperations, query, context);
get_setop_query(query->setOperations, query, context, resultDesc,
colNamesVisible);
/* ORDER BY clauses must be simple in this case */
force_colno = true;
}
else
{
get_basic_select_query(query, context);
get_basic_select_query(query, context, resultDesc, colNamesVisible);
force_colno = false;
}
@ -2424,6 +2429,9 @@ get_select_query_def(Query *query, deparse_context *context)
appendStringInfoString(buf, " SKIP LOCKED");
}
}
context->windowClause = save_windowclause;
context->windowTList = save_windowtlist;
}
/*
@ -2498,7 +2506,8 @@ get_simple_values_rte(Query *query, TupleDesc resultDesc)
}
static void
get_basic_select_query(Query *query, deparse_context *context)
get_basic_select_query(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *values_rte;
@ -2516,7 +2525,7 @@ get_basic_select_query(Query *query, deparse_context *context)
* VALUES part. This reverses what transformValuesClause() did at parse
* time.
*/
values_rte = get_simple_values_rte(query, context->resultDesc);
values_rte = get_simple_values_rte(query, resultDesc);
if (values_rte)
{
get_values_def(values_rte->values_lists, context);
@ -2554,7 +2563,7 @@ get_basic_select_query(Query *query, deparse_context *context)
}
/* Then we tell what to select (the targetlist) */
get_target_list(query->targetList, context);
get_target_list(query->targetList, context, resultDesc, colNamesVisible);
/* Add the FROM clause if needed */
get_from_clause(query, " FROM ", context);
@ -2570,15 +2579,15 @@ get_basic_select_query(Query *query, deparse_context *context)
/* Add the GROUP BY clause if given */
if (query->groupClause != NULL || query->groupingSets != NULL)
{
bool save_ingroupby;
ParseExprKind save_exprkind;
appendContextKeyword(context, " GROUP BY ",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
if (query->groupDistinct)
appendStringInfoString(buf, "DISTINCT ");
save_ingroupby = context->inGroupBy;
context->inGroupBy = true;
save_exprkind = context->special_exprkind;
context->special_exprkind = EXPR_KIND_GROUP_BY;
if (query->groupingSets == NIL)
{
@ -2606,7 +2615,7 @@ get_basic_select_query(Query *query, deparse_context *context)
}
}
context->inGroupBy = save_ingroupby;
context->special_exprkind = save_exprkind;
}
/* Add the HAVING clause if given */
@ -2625,11 +2634,14 @@ get_basic_select_query(Query *query, deparse_context *context)
/* ----------
* get_target_list - Parse back a SELECT target list
*
* This is also used for RETURNING lists in INSERT/UPDATE/DELETE/MERGE.
* This is also used for RETURNING lists in INSERT/UPDATE/DELETE.
*
* resultDesc and colNamesVisible are as for get_query_def()
* ----------
*/
static void
get_target_list(List *targetList, deparse_context *context)
get_target_list(List *targetList, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
StringInfoData targetbuf;
@ -2686,7 +2698,7 @@ get_target_list(List *targetList, deparse_context *context)
* assigned column name explicitly. Otherwise, show it only if
* it's not FigureColname's fallback.
*/
attname = context->colNamesVisible ? NULL : "?column?";
attname = colNamesVisible ? NULL : "?column?";
}
/*
@ -2695,9 +2707,8 @@ get_target_list(List *targetList, deparse_context *context)
* effects of any column RENAME that's been done on the view).
* Otherwise, just use what we can find in the TLE.
*/
if (context->resultDesc && colno <= context->resultDesc->natts)
colname = NameStr(TupleDescAttr(context->resultDesc,
colno - 1)->attname);
if (resultDesc && colno <= resultDesc->natts)
colname = NameStr(TupleDescAttr(resultDesc, colno - 1)->attname);
else
colname = tle->resname;
@ -2765,7 +2776,8 @@ get_target_list(List *targetList, deparse_context *context)
}
static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
get_setop_query(Node *setOp, Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
bool need_paren;
@ -2790,8 +2802,8 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
subquery->limitCount);
if (need_paren)
appendStringInfoChar(buf, '(');
get_query_def(subquery, buf, context->namespaces,
context->resultDesc, context->colNamesVisible,
get_query_def(subquery, buf, context->namespaces, resultDesc,
colNamesVisible,
context->prettyFlags, context->wrapColumn,
context->indentLevel);
if (need_paren)
@ -2801,7 +2813,6 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
SetOperationStmt *op = (SetOperationStmt *) setOp;
int subindent;
bool save_colnamesvisible;
/*
* We force parens when nesting two SetOperationStmts, except when the
@ -2835,7 +2846,7 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
else
subindent = 0;
get_setop_query(op->larg, query, context);
get_setop_query(op->larg, query, context, resultDesc, colNamesVisible);
if (need_paren)
appendContextKeyword(context, ") ", -subindent, 0, 0);
@ -2879,13 +2890,7 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
subindent = 0;
appendContextKeyword(context, "", subindent, 0, 0);
/*
* The output column names of the RHS sub-select don't matter.
*/
save_colnamesvisible = context->colNamesVisible;
context->colNamesVisible = false;
get_setop_query(op->rarg, query, context);
context->colNamesVisible = save_colnamesvisible;
get_setop_query(op->rarg, query, context, resultDesc, false);
if (PRETTY_INDENT(context))
context->indentLevel -= subindent;
@ -2919,31 +2924,20 @@ get_rule_sortgroupclause(Index ref, List *tlist, bool force_colno,
* Use column-number form if requested by caller. Otherwise, if
* expression is a constant, force it to be dumped with an explicit cast
* as decoration --- this is because a simple integer constant is
* ambiguous (and will be misinterpreted by findTargetlistEntrySQL92()) if
* we dump it without any decoration. Similarly, if it's just a Var,
* there is risk of misinterpretation if the column name is reassigned in
* the SELECT list, so we may need to force table qualification. And, if
* it's anything more complex than a simple Var, then force extra parens
* around it, to ensure it can't be misinterpreted as a cube() or rollup()
* construct.
* ambiguous (and will be misinterpreted by findTargetlistEntry()) if we
* dump it without any decoration. If it's anything more complex than a
* simple Var, then force extra parens around it, to ensure it can't be
* misinterpreted as a cube() or rollup() construct.
*/
if (force_colno)
{
Assert(!tle->resjunk);
appendStringInfo(buf, "%d", tle->resno);
}
else if (!expr)
/* do nothing, probably can't happen */ ;
else if (IsA(expr, Const))
else if (expr && IsA(expr, Const))
get_const_expr((Const *) expr, context, 1);
else if (IsA(expr, Var))
{
/* Tell get_variable to check for name conflict */
bool save_varinorderby = context->varInOrderBy;
context->varInOrderBy = true;
(void) get_variable((Var *) expr, 0, false, context);
context->varInOrderBy = save_varinorderby;
}
else if (!expr || IsA(expr, Var))
get_rule_expr(expr, context, true);
else
{
/*
@ -3231,7 +3225,8 @@ get_rule_windowspec(WindowClause *wc, List *targetList,
* ----------
*/
static void
get_insert_query_def(Query *query, deparse_context *context)
get_insert_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *select_rte = NULL;
@ -3410,7 +3405,7 @@ get_insert_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3419,7 +3414,8 @@ get_insert_query_def(Query *query, deparse_context *context)
* ----------
*/
static void
get_update_query_def(Query *query, deparse_context *context)
get_update_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *rte;
@ -3489,7 +3485,7 @@ get_update_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3649,7 +3645,8 @@ get_update_query_targetlist_def(Query *query, List *targetList,
* ----------
*/
static void
get_delete_query_def(Query *query, deparse_context *context)
get_delete_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *rte;
@ -3714,7 +3711,7 @@ get_delete_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3966,7 +3963,6 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
deparse_columns *colinfo;
char *refname;
char *attname;
bool need_prefix;
/* Find appropriate nesting depth */
netlevelsup = var->varlevelsup + levelsup;
@ -4167,42 +4163,7 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
attname = get_rte_attribute_name(rte, attnum);
}
need_prefix = (context->varprefix || attname == NULL);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
* clause, we may need to add a table-name prefix to prevent
* findTargetlistEntrySQL92 from misinterpreting the name as an
* output-column name. To avoid cluttering the output with unnecessary
* prefixes, do so only if there is a name match to a SELECT tlist item
* that is different from the Var.
*/
if (context->varInOrderBy && !context->inGroupBy && !need_prefix)
{
int colno = 0;
ListCell *l;
foreach(l, context->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(l);
char *colname;
if (tle->resjunk)
continue; /* ignore junk entries */
colno++;
/* This must match colname-choosing logic in get_target_list() */
if (context->resultDesc && colno <= context->resultDesc->natts)
colname = NameStr(TupleDescAttr(context->resultDesc,
colno - 1)->attname);
else
colname = tle->resname;
if (colname && strcmp(colname, attname) == 0 &&
!equal(var, tle->expr))
{
need_prefix = true;
break;
}
}
}
if (refname && need_prefix)
if (refname && (context->varprefix || attname == NULL))
{
appendStringInfoString(buf, quote_identifier(refname));
appendStringInfoChar(buf, '.');
@ -6766,7 +6727,7 @@ get_func_expr(FuncExpr *expr, deparse_context *context,
argnames, argtypes,
expr->funcvariadic,
&use_variadic,
context->inGroupBy));
context->special_exprkind));
nargs = 0;
foreach(l, expr->args)
{
@ -6809,7 +6770,7 @@ get_proc_expr(CallStmt *stmt, deparse_context *context,
namedArgList, argumentTypes,
stmt->funcexpr->funcvariadic,
&use_variadic,
context->inGroupBy));
context->special_exprkind));
int argNumber = 0;
foreach(argumentCell, finalArgumentList)
{
@ -6871,7 +6832,7 @@ get_agg_expr(Aggref *aggref, deparse_context *context,
NIL, argtypes,
aggref->aggvariadic,
&use_variadic,
context->inGroupBy),
context->special_exprkind),
(aggref->aggdistinct != NIL) ? "DISTINCT " : "");
if (AGGKIND_IS_ORDERED_SET(aggref->aggkind))
@ -6980,7 +6941,7 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
generate_function_name(wfunc->winfnoid, nargs,
argnames, argtypes,
false, NULL,
context->inGroupBy));
context->special_exprkind));
/* winstar can be set only in zero-argument aggregates */
if (wfunc->winstar)
@ -7005,7 +6966,7 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
if (wc->name)
appendStringInfoString(buf, quote_identifier(wc->name));
else
get_rule_windowspec(wc, context->targetList, context);
get_rule_windowspec(wc, context->windowTList, context);
break;
}
}
@ -8310,7 +8271,7 @@ get_tablesample_def(TableSampleClause *tablesample, deparse_context *context)
appendStringInfo(buf, " TABLESAMPLE %s (",
generate_function_name(tablesample->tsmhandler, 1,
NIL, argtypes,
false, NULL, false));
false, NULL, EXPR_KIND_NONE));
nargs = 0;
foreach(l, tablesample->args)
@ -8657,14 +8618,12 @@ generate_fragment_name(char *schemaName, char *tableName)
* the output. For non-FuncExpr cases, has_variadic should be false and
* use_variadic_p can be NULL.
*
* inGroupBy must be true if we're deparsing a GROUP BY clause.
*
* The result includes all necessary quoting and schema-prefixing.
*/
static char *
generate_function_name(Oid funcid, int nargs, List *argnames, Oid *argtypes,
bool has_variadic, bool *use_variadic_p,
bool inGroupBy)
ParseExprKind special_exprkind)
{
char *result;
HeapTuple proctup;
@ -8689,9 +8648,9 @@ generate_function_name(Oid funcid, int nargs, List *argnames, Oid *argtypes,
/*
* Due to parser hacks to avoid needing to reserve CUBE, we need to force
* qualification of some function names within GROUP BY.
* qualification in some special cases.
*/
if (inGroupBy)
if (special_exprkind == EXPR_KIND_GROUP_BY)
{
if (strcmp(proname, "cube") == 0 || strcmp(proname, "rollup") == 0)
force_qualify = true;

View File

@ -67,6 +67,7 @@
#include "parser/parse_node.h"
#include "parser/parse_agg.h"
#include "parser/parse_func.h"
#include "parser/parse_node.h"
#include "parser/parse_oper.h"
#include "parser/parse_relation.h"
#include "parser/parser.h"
@ -122,18 +123,16 @@ typedef struct
{
StringInfo buf; /* output buffer to append to */
List *namespaces; /* List of deparse_namespace nodes */
TupleDesc resultDesc; /* if top level of a view, the view's tupdesc */
List *targetList; /* Current query level's SELECT targetlist */
List *windowClause; /* Current query level's WINDOW clause */
List *windowTList; /* targetlist for resolving WINDOW clause */
int prettyFlags; /* enabling of pretty-print functions */
int wrapColumn; /* max line length, or -1 for no limit */
int indentLevel; /* current indent level for prettyprint */
bool varprefix; /* true to print prefixes on Vars */
Oid distrelid; /* the distributed table being modified, if valid */
int64 shardid; /* a distributed table's shardid, if positive */
bool colNamesVisible; /* do we care about output column names? */
bool inGroupBy; /* deparsing GROUP BY clause? */
bool varInOrderBy; /* deparsing simple Var in ORDER BY? */
ParseExprKind special_exprkind; /* set only for exprkinds needing special
* handling */
Bitmapset *appendparents; /* if not null, map child Vars of these relids
* back to the parent rel */
} deparse_context;
@ -365,19 +364,27 @@ static void get_query_def_extended(Query *query, StringInfo buf,
int startIndent);
static void get_values_def(List *values_lists, deparse_context *context);
static void get_with_clause(Query *query, deparse_context *context);
static void get_select_query_def(Query *query, deparse_context *context);
static void get_insert_query_def(Query *query, deparse_context *context);
static void get_update_query_def(Query *query, deparse_context *context);
static void get_select_query_def(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_insert_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_update_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_update_query_targetlist_def(Query *query, List *targetList,
deparse_context *context,
RangeTblEntry *rte);
static void get_delete_query_def(Query *query, deparse_context *context);
static void get_merge_query_def(Query *query, deparse_context *context);
static void get_delete_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_merge_query_def(Query *query, deparse_context *context,
bool colNamesVisible);
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_target_list(List *targetList, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
bool force_colno,
deparse_context *context);
@ -472,7 +479,7 @@ static char *generate_fragment_name(char *schemaName, char *tableName);
static char *generate_function_name(Oid funcid, int nargs,
List *argnames, Oid *argtypes,
bool has_variadic, bool *use_variadic_p,
bool inGroupBy);
ParseExprKind special_exprkind);
static List *get_insert_column_names_list(List *targetList, StringInfo buf, deparse_context *context, RangeTblEntry *rte);
#define only_marker(rte) ((rte)->inh ? "" : "ONLY ")
@ -646,16 +653,13 @@ pg_get_rule_expr(Node *expression)
context.buf = buffer;
context.namespaces = NIL;
context.resultDesc = NULL;
context.targetList = NIL;
context.windowClause = NIL;
context.windowTList = NIL;
context.varprefix = false;
context.prettyFlags = 0;
context.wrapColumn = WRAP_COLUMN_DEFAULT;
context.indentLevel = 0;
context.colNamesVisible = true;
context.inGroupBy = false;
context.varInOrderBy = false;
context.special_exprkind = EXPR_KIND_NONE;
context.distrelid = InvalidOid;
context.shardid = INVALID_SHARD_ID;
@ -2076,17 +2080,14 @@ get_query_def_extended(Query *query, StringInfo buf, List *parentnamespace,
context.buf = buf;
context.namespaces = lcons(&dpns, list_copy(parentnamespace));
context.resultDesc = NULL;
context.targetList = NIL;
context.windowClause = NIL;
context.windowTList = NIL;
context.varprefix = (parentnamespace != NIL ||
list_length(query->rtable) != 1);
context.prettyFlags = prettyFlags;
context.wrapColumn = wrapColumn;
context.indentLevel = startIndent;
context.colNamesVisible = true;
context.inGroupBy = false;
context.varInOrderBy = false;
context.special_exprkind = EXPR_KIND_NONE;
context.appendparents = NULL;
context.distrelid = distrelid;
context.shardid = shardid;
@ -2096,25 +2097,23 @@ get_query_def_extended(Query *query, StringInfo buf, List *parentnamespace,
switch (query->commandType)
{
case CMD_SELECT:
/* We set context.resultDesc only if it's a SELECT */
context.resultDesc = resultDesc;
get_select_query_def(query, &context);
get_select_query_def(query, &context, resultDesc, colNamesVisible);
break;
case CMD_UPDATE:
get_update_query_def(query, &context);
get_update_query_def(query, &context, colNamesVisible);
break;
case CMD_INSERT:
get_insert_query_def(query, &context);
get_insert_query_def(query, &context, colNamesVisible);
break;
case CMD_DELETE:
get_delete_query_def(query, &context);
get_delete_query_def(query, &context, colNamesVisible);
break;
case CMD_MERGE:
get_merge_query_def(query, &context);
get_merge_query_def(query, &context, colNamesVisible);
break;
case CMD_NOTHING:
@ -2322,18 +2321,23 @@ get_with_clause(Query *query, deparse_context *context)
* ----------
*/
static void
get_select_query_def(Query *query, deparse_context *context)
get_select_query_def(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
List *save_windowclause;
List *save_windowtlist;
bool force_colno;
ListCell *l;
/* Insert the WITH clause if given */
get_with_clause(query, context);
/* Subroutines may need to consult the SELECT targetlist and windowClause */
context->targetList = query->targetList;
/* Set up context for possible window functions */
save_windowclause = context->windowClause;
context->windowClause = query->windowClause;
save_windowtlist = context->windowTList;
context->windowTList = query->targetList;
/*
* If the Query node has a setOperations tree, then it's the top level of
@ -2342,13 +2346,14 @@ get_select_query_def(Query *query, deparse_context *context)
*/
if (query->setOperations)
{
get_setop_query(query->setOperations, query, context);
get_setop_query(query->setOperations, query, context, resultDesc,
colNamesVisible);
/* ORDER BY clauses must be simple in this case */
force_colno = true;
}
else
{
get_basic_select_query(query, context);
get_basic_select_query(query, context, resultDesc, colNamesVisible);
force_colno = false;
}
@ -2438,6 +2443,9 @@ get_select_query_def(Query *query, deparse_context *context)
appendStringInfoString(buf, " SKIP LOCKED");
}
}
context->windowClause = save_windowclause;
context->windowTList = save_windowtlist;
}
/*
@ -2512,7 +2520,8 @@ get_simple_values_rte(Query *query, TupleDesc resultDesc)
}
static void
get_basic_select_query(Query *query, deparse_context *context)
get_basic_select_query(Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *values_rte;
@ -2530,7 +2539,7 @@ get_basic_select_query(Query *query, deparse_context *context)
* VALUES part. This reverses what transformValuesClause() did at parse
* time.
*/
values_rte = get_simple_values_rte(query, context->resultDesc);
values_rte = get_simple_values_rte(query, resultDesc);
if (values_rte)
{
get_values_def(values_rte->values_lists, context);
@ -2568,7 +2577,7 @@ get_basic_select_query(Query *query, deparse_context *context)
}
/* Then we tell what to select (the targetlist) */
get_target_list(query->targetList, context);
get_target_list(query->targetList, context, resultDesc, colNamesVisible);
/* Add the FROM clause if needed */
get_from_clause(query, " FROM ", context);
@ -2584,15 +2593,15 @@ get_basic_select_query(Query *query, deparse_context *context)
/* Add the GROUP BY clause if given */
if (query->groupClause != NULL || query->groupingSets != NULL)
{
bool save_ingroupby;
ParseExprKind save_exprkind;
appendContextKeyword(context, " GROUP BY ",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
if (query->groupDistinct)
appendStringInfoString(buf, "DISTINCT ");
save_ingroupby = context->inGroupBy;
context->inGroupBy = true;
save_exprkind = context->special_exprkind;
context->special_exprkind = EXPR_KIND_GROUP_BY;
if (query->groupingSets == NIL)
{
@ -2620,7 +2629,7 @@ get_basic_select_query(Query *query, deparse_context *context)
}
}
context->inGroupBy = save_ingroupby;
context->special_exprkind = save_exprkind;
}
/* Add the HAVING clause if given */
@ -2639,11 +2648,14 @@ get_basic_select_query(Query *query, deparse_context *context)
/* ----------
* get_target_list - Parse back a SELECT target list
*
* This is also used for RETURNING lists in INSERT/UPDATE/DELETE/MERGE.
* This is also used for RETURNING lists in INSERT/UPDATE/DELETE.
*
* resultDesc and colNamesVisible are as for get_query_def()
* ----------
*/
static void
get_target_list(List *targetList, deparse_context *context)
get_target_list(List *targetList, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
StringInfoData targetbuf;
@ -2700,7 +2712,7 @@ get_target_list(List *targetList, deparse_context *context)
* assigned column name explicitly. Otherwise, show it only if
* it's not FigureColname's fallback.
*/
attname = context->colNamesVisible ? NULL : "?column?";
attname = colNamesVisible ? NULL : "?column?";
}
/*
@ -2709,9 +2721,8 @@ get_target_list(List *targetList, deparse_context *context)
* effects of any column RENAME that's been done on the view).
* Otherwise, just use what we can find in the TLE.
*/
if (context->resultDesc && colno <= context->resultDesc->natts)
colname = NameStr(TupleDescAttr(context->resultDesc,
colno - 1)->attname);
if (resultDesc && colno <= resultDesc->natts)
colname = NameStr(TupleDescAttr(resultDesc, colno - 1)->attname);
else
colname = tle->resname;
@ -2779,7 +2790,8 @@ get_target_list(List *targetList, deparse_context *context)
}
static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
get_setop_query(Node *setOp, Query *query, deparse_context *context,
TupleDesc resultDesc, bool colNamesVisible)
{
StringInfo buf = context->buf;
bool need_paren;
@ -2804,8 +2816,8 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
subquery->limitCount);
if (need_paren)
appendStringInfoChar(buf, '(');
get_query_def(subquery, buf, context->namespaces,
context->resultDesc, context->colNamesVisible,
get_query_def(subquery, buf, context->namespaces, resultDesc,
colNamesVisible,
context->prettyFlags, context->wrapColumn,
context->indentLevel);
if (need_paren)
@ -2815,7 +2827,6 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
SetOperationStmt *op = (SetOperationStmt *) setOp;
int subindent;
bool save_colnamesvisible;
/*
* We force parens when nesting two SetOperationStmts, except when the
@ -2849,7 +2860,7 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
else
subindent = 0;
get_setop_query(op->larg, query, context);
get_setop_query(op->larg, query, context, resultDesc, colNamesVisible);
if (need_paren)
appendContextKeyword(context, ") ", -subindent, 0, 0);
@ -2893,13 +2904,7 @@ get_setop_query(Node *setOp, Query *query, deparse_context *context)
subindent = 0;
appendContextKeyword(context, "", subindent, 0, 0);
/*
* The output column names of the RHS sub-select don't matter.
*/
save_colnamesvisible = context->colNamesVisible;
context->colNamesVisible = false;
get_setop_query(op->rarg, query, context);
context->colNamesVisible = save_colnamesvisible;
get_setop_query(op->rarg, query, context, resultDesc, false);
if (PRETTY_INDENT(context))
context->indentLevel -= subindent;
@ -2933,31 +2938,20 @@ get_rule_sortgroupclause(Index ref, List *tlist, bool force_colno,
* Use column-number form if requested by caller. Otherwise, if
* expression is a constant, force it to be dumped with an explicit cast
* as decoration --- this is because a simple integer constant is
* ambiguous (and will be misinterpreted by findTargetlistEntrySQL92()) if
* we dump it without any decoration. Similarly, if it's just a Var,
* there is risk of misinterpretation if the column name is reassigned in
* the SELECT list, so we may need to force table qualification. And, if
* it's anything more complex than a simple Var, then force extra parens
* around it, to ensure it can't be misinterpreted as a cube() or rollup()
* construct.
* ambiguous (and will be misinterpreted by findTargetlistEntry()) if we
* dump it without any decoration. If it's anything more complex than a
* simple Var, then force extra parens around it, to ensure it can't be
* misinterpreted as a cube() or rollup() construct.
*/
if (force_colno)
{
Assert(!tle->resjunk);
appendStringInfo(buf, "%d", tle->resno);
}
else if (!expr)
/* do nothing, probably can't happen */ ;
else if (IsA(expr, Const))
else if (expr && IsA(expr, Const))
get_const_expr((Const *) expr, context, 1);
else if (IsA(expr, Var))
{
/* Tell get_variable to check for name conflict */
bool save_varinorderby = context->varInOrderBy;
context->varInOrderBy = true;
(void) get_variable((Var *) expr, 0, false, context);
context->varInOrderBy = save_varinorderby;
}
else if (!expr || IsA(expr, Var))
get_rule_expr(expr, context, true);
else
{
/*
@ -3246,7 +3240,8 @@ get_rule_windowspec(WindowClause *wc, List *targetList,
* ----------
*/
static void
get_insert_query_def(Query *query, deparse_context *context)
get_insert_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *select_rte = NULL;
@ -3427,7 +3422,7 @@ get_insert_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3436,7 +3431,8 @@ get_insert_query_def(Query *query, deparse_context *context)
* ----------
*/
static void
get_update_query_def(Query *query, deparse_context *context)
get_update_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *rte;
@ -3505,7 +3501,7 @@ get_update_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3665,7 +3661,8 @@ get_update_query_targetlist_def(Query *query, List *targetList,
* ----------
*/
static void
get_delete_query_def(Query *query, deparse_context *context)
get_delete_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *rte;
@ -3729,7 +3726,7 @@ get_delete_query_def(Query *query, deparse_context *context)
{
appendContextKeyword(context, " RETURNING",
-PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
get_target_list(query->returningList, context);
get_target_list(query->returningList, context, NULL, colNamesVisible);
}
}
@ -3739,7 +3736,8 @@ get_delete_query_def(Query *query, deparse_context *context)
* ----------
*/
static void
get_merge_query_def(Query *query, deparse_context *context)
get_merge_query_def(Query *query, deparse_context *context,
bool colNamesVisible)
{
StringInfo buf = context->buf;
RangeTblEntry *rte;
@ -3979,7 +3977,6 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
deparse_columns *colinfo;
char *refname;
char *attname;
bool need_prefix;
/* Find appropriate nesting depth */
netlevelsup = var->varlevelsup + levelsup;
@ -4180,42 +4177,7 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
attname = get_rte_attribute_name(rte, attnum);
}
need_prefix = (context->varprefix || attname == NULL);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
* clause, we may need to add a table-name prefix to prevent
* findTargetlistEntrySQL92 from misinterpreting the name as an
* output-column name. To avoid cluttering the output with unnecessary
* prefixes, do so only if there is a name match to a SELECT tlist item
* that is different from the Var.
*/
if (context->varInOrderBy && !context->inGroupBy && !need_prefix)
{
int colno = 0;
ListCell *l;
foreach(l, context->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(l);
char *colname;
if (tle->resjunk)
continue; /* ignore junk entries */
colno++;
/* This must match colname-choosing logic in get_target_list() */
if (context->resultDesc && colno <= context->resultDesc->natts)
colname = NameStr(TupleDescAttr(context->resultDesc,
colno - 1)->attname);
else
colname = tle->resname;
if (colname && strcmp(colname, attname) == 0 &&
!equal(var, tle->expr))
{
need_prefix = true;
break;
}
}
}
if (refname && need_prefix)
if (refname && (context->varprefix || attname == NULL))
{
appendStringInfoString(buf, quote_identifier(refname));
appendStringInfoChar(buf, '.');
@ -6813,7 +6775,7 @@ get_func_expr(FuncExpr *expr, deparse_context *context,
argnames, argtypes,
expr->funcvariadic,
&use_variadic,
context->inGroupBy));
context->special_exprkind));
nargs = 0;
foreach(l, expr->args)
{
@ -6856,7 +6818,7 @@ get_proc_expr(CallStmt *stmt, deparse_context *context,
namedArgList, argumentTypes,
stmt->funcexpr->funcvariadic,
&use_variadic,
context->inGroupBy));
context->special_exprkind));
int argNumber = 0;
foreach(argumentCell, finalArgumentList)
{
@ -6929,7 +6891,7 @@ get_agg_expr_helper(Aggref *aggref, deparse_context *context,
funcname = generate_function_name(aggref->aggfnoid, nargs, NIL,
argtypes, aggref->aggvariadic,
&use_variadic,
context->inGroupBy);
context->special_exprkind);
/* Print the aggregate name, schema-qualified if needed */
appendStringInfo(buf, "%s(%s", funcname,
@ -7070,7 +7032,7 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
if (!funcname)
funcname = generate_function_name(wfunc->winfnoid, nargs, argnames,
argtypes, false, NULL,
context->inGroupBy);
context->special_exprkind);
appendStringInfo(buf, "%s(", funcname);
@ -7109,7 +7071,7 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
if (wc->name)
appendStringInfoString(buf, quote_identifier(wc->name));
else
get_rule_windowspec(wc, context->targetList, context);
get_rule_windowspec(wc, context->windowTList, context);
break;
}
}
@ -8585,7 +8547,7 @@ get_tablesample_def(TableSampleClause *tablesample, deparse_context *context)
appendStringInfo(buf, " TABLESAMPLE %s (",
generate_function_name(tablesample->tsmhandler, 1,
NIL, argtypes,
false, NULL, false));
false, NULL, EXPR_KIND_NONE));
nargs = 0;
foreach(l, tablesample->args)
@ -8932,14 +8894,12 @@ generate_fragment_name(char *schemaName, char *tableName)
* the output. For non-FuncExpr cases, has_variadic should be false and
* use_variadic_p can be NULL.
*
* inGroupBy must be true if we're deparsing a GROUP BY clause.
*
* The result includes all necessary quoting and schema-prefixing.
*/
static char *
generate_function_name(Oid funcid, int nargs, List *argnames, Oid *argtypes,
bool has_variadic, bool *use_variadic_p,
bool inGroupBy)
ParseExprKind special_exprkind)
{
char *result;
HeapTuple proctup;
@ -8964,9 +8924,9 @@ generate_function_name(Oid funcid, int nargs, List *argnames, Oid *argtypes,
/*
* Due to parser hacks to avoid needing to reserve CUBE, we need to force
* qualification of some function names within GROUP BY.
* qualification in some special cases.
*/
if (inGroupBy)
if (special_exprkind == EXPR_KIND_GROUP_BY)
{
if (strcmp(proname, "cube") == 0 || strcmp(proname, "rollup") == 0)
force_qualify = true;

View File

@ -171,7 +171,6 @@
#include "distributed/repartition_join_execution.h"
#include "distributed/resource_lock.h"
#include "distributed/shared_connection_stats.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/subplan_execution.h"
#include "distributed/transaction_identifier.h"
#include "distributed/transaction_management.h"
@ -402,7 +401,7 @@ typedef struct WorkerPool
/*
* Placement executions destined for worker node, but not assigned to any
* connection and ready to start.
* connection and not ready to start.
*/
dlist_head readyTaskQueue;
int readyTaskCount;
@ -493,6 +492,8 @@ typedef struct WorkerSession
} WorkerSession;
struct TaskPlacementExecution;
/* GUC, determining whether Citus opens 1 connection per task */
bool ForceMaxQueryParallelization = false;
int MaxAdaptiveExecutorPoolSize = 16;
@ -584,7 +585,7 @@ typedef enum TaskPlacementExecutionState
} TaskPlacementExecutionState;
/*
* TaskPlacementExecution represents the execution of a command
* TaskPlacementExecution represents the an execution of a command
* on a shard placement.
*/
typedef struct TaskPlacementExecution
@ -691,7 +692,7 @@ static bool SendNextQuery(TaskPlacementExecution *placementExecution,
WorkerSession *session);
static void ConnectionStateMachine(WorkerSession *session);
static bool HasUnfinishedTaskForSession(WorkerSession *session);
static void HandleMultiConnectionSuccess(WorkerSession *session, bool newConnection);
static void HandleMultiConnectionSuccess(WorkerSession *session);
static bool HasAnyConnectionFailure(WorkerPool *workerPool);
static void Activate2PCIfModifyingTransactionExpandsToNewNode(WorkerSession *session);
static bool TransactionModifiedDistributedTable(DistributedExecution *execution);
@ -719,18 +720,15 @@ static void RebuildWaitEventSetForSessions(DistributedExecution *execution);
static void AddLatchWaitEventToExecution(DistributedExecution *execution);
static void ProcessWaitEvents(DistributedExecution *execution, WaitEvent *events, int
eventCount, bool *cancellationReceived);
#if PG_VERSION_NUM >= PG_VERSION_15
static void RemoteSocketClosedForAnySession(DistributedExecution *execution);
static void ProcessWaitEventsForSocketClosed(WaitEvent *events, int eventCount);
#endif
static long MillisecondsBetweenTimestamps(instr_time startTime, instr_time endTime);
static uint64 MicrosecondsBetweenTimestamps(instr_time startTime, instr_time endTime);
static int WorkerPoolCompare(const void *lhsKey, const void *rhsKey);
static void SetAttributeInputMetadata(DistributedExecution *execution,
ShardCommandExecution *shardCommandExecution);
static ExecutionParams * CreateDefaultExecutionParams(RowModifyLevel modLevel,
List *taskList,
TupleDestination *tupleDest,
bool expectResults,
ParamListInfo paramListInfo);
/*
@ -1017,14 +1015,14 @@ ExecuteTaskListOutsideTransaction(RowModifyLevel modLevel, List *taskList,
/*
* CreateDefaultExecutionParams returns execution params based on given (possibly null)
* bind params (presumably from executor state) with defaults for some of the arguments.
* ExecuteTaskListIntoTupleDestWithParam is a proxy to ExecuteTaskListExtended() which uses
* bind params from executor state, and with defaults for some of the arguments.
*/
static ExecutionParams *
CreateDefaultExecutionParams(RowModifyLevel modLevel, List *taskList,
TupleDestination *tupleDest,
bool expectResults,
ParamListInfo paramListInfo)
uint64
ExecuteTaskListIntoTupleDestWithParam(RowModifyLevel modLevel, List *taskList,
TupleDestination *tupleDest,
bool expectResults,
ParamListInfo paramListInfo)
{
int targetPoolSize = MaxAdaptiveExecutorPoolSize;
bool localExecutionSupported = true;
@ -1038,24 +1036,6 @@ CreateDefaultExecutionParams(RowModifyLevel modLevel, List *taskList,
executionParams->tupleDestination = tupleDest;
executionParams->paramListInfo = paramListInfo;
return executionParams;
}
/*
* ExecuteTaskListIntoTupleDestWithParam is a proxy to ExecuteTaskListExtended() which uses
* bind params from executor state, and with defaults for some of the arguments.
*/
uint64
ExecuteTaskListIntoTupleDestWithParam(RowModifyLevel modLevel, List *taskList,
TupleDestination *tupleDest,
bool expectResults,
ParamListInfo paramListInfo)
{
ExecutionParams *executionParams = CreateDefaultExecutionParams(modLevel, taskList,
tupleDest,
expectResults,
paramListInfo);
return ExecuteTaskListExtended(executionParams);
}
@ -1069,11 +1049,17 @@ ExecuteTaskListIntoTupleDest(RowModifyLevel modLevel, List *taskList,
TupleDestination *tupleDest,
bool expectResults)
{
ParamListInfo paramListInfo = NULL;
ExecutionParams *executionParams = CreateDefaultExecutionParams(modLevel, taskList,
tupleDest,
expectResults,
paramListInfo);
int targetPoolSize = MaxAdaptiveExecutorPoolSize;
bool localExecutionSupported = true;
ExecutionParams *executionParams = CreateBasicExecutionParams(
modLevel, taskList, targetPoolSize, localExecutionSupported
);
executionParams->xactProperties = DecideTransactionPropertiesForTaskList(
modLevel, taskList, false);
executionParams->expectResults = expectResults;
executionParams->tupleDestination = tupleDest;
return ExecuteTaskListExtended(executionParams);
}
@ -1783,8 +1769,11 @@ FindOrCreateWorkerSession(WorkerPool *workerPool, MultiConnection *connection)
session->commandsSent = 0;
session->waitEventSetIndex = WAIT_EVENT_SET_INDEX_NOT_INITIALIZED;
#if PG_VERSION_NUM >= PG_VERSION_15
/* always detect closed sockets */
UpdateConnectionWaitFlags(session, WL_SOCKET_CLOSED);
#endif
dlist_init(&session->pendingTaskQueue);
dlist_init(&session->readyTaskQueue);
@ -1828,6 +1817,7 @@ FindOrCreateWorkerSession(WorkerPool *workerPool, MultiConnection *connection)
* the events, even ignores cancellation events. Future callers of this
* function should consider its limitations.
*/
#if PG_VERSION_NUM >= PG_VERSION_15
static void
RemoteSocketClosedForAnySession(DistributedExecution *execution)
{
@ -1845,6 +1835,9 @@ RemoteSocketClosedForAnySession(DistributedExecution *execution)
}
#endif
/*
* SequentialRunDistributedExecution gets a distributed execution and
* executes each individual task in the execution sequentially, one
@ -1915,7 +1908,7 @@ RunDistributedExecution(DistributedExecution *execution)
/*
* Iterate until all the tasks are finished. Once all the tasks
* are finished, ensure that all the connection initializations
* are finished, ensure that that all the connection initializations
* are also finished. Otherwise, those connections are terminated
* abruptly before they are established (or failed). Instead, we let
* the ConnectionStateMachine() to properly handle them.
@ -2036,7 +2029,6 @@ ProcessSessionsWithFailedWaitEventSetOperations(DistributedExecution *execution)
else
{
connection->connectionState = MULTI_CONNECTION_FAILED;
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
}
@ -2181,6 +2173,8 @@ ProcessWaitEvents(DistributedExecution *execution, WaitEvent *events, int eventC
}
#if PG_VERSION_NUM >= PG_VERSION_15
/*
* ProcessWaitEventsForSocketClosed mainly checks for WL_SOCKET_CLOSED event.
* If WL_SOCKET_CLOSED is found, the function sets the underlying connection's
@ -2213,6 +2207,9 @@ ProcessWaitEventsForSocketClosed(WaitEvent *events, int eventCount)
}
#endif
/*
* ManageWorkerPool ensures the worker pool has the appropriate number of connections
* based on the number of pending tasks.
@ -2707,6 +2704,7 @@ OpenNewConnections(WorkerPool *workerPool, int newConnectionCount,
* Instead, we prefer this slight difference, which in effect has almost no
* difference, but doing things in different points in time.
*/
#if PG_VERSION_NUM >= PG_VERSION_15
/* we added new connections, rebuild the waitEventSet */
RebuildWaitEventSetForSessions(execution);
@ -2726,6 +2724,9 @@ OpenNewConnections(WorkerPool *workerPool, int newConnectionCount,
* of the execution.
*/
AddLatchWaitEventToExecution(execution);
#else
execution->rebuildWaitEventSet = true;
#endif
WorkerSession *session = NULL;
foreach_declared_ptr(session, newSessionsList)
@ -2812,21 +2813,21 @@ CheckConnectionTimeout(WorkerPool *workerPool)
logLevel = ERROR;
}
ereport(logLevel, (errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("could not establish any connections to the node "
"%s:%d after %u ms", workerPool->nodeName,
workerPool->nodePort,
NodeConnectionTimeout)));
/*
* We hit the connection timeout. In that case, we should not let the
* connection establishment to continue because the execution logic
* pretends that failed sessions are not going to be used anymore.
*
* That's why we mark the connection as timed out to trigger the state
* changes in the executor, if we don't throw an error below.
* changes in the executor.
*/
MarkEstablishingSessionsTimedOut(workerPool);
ereport(logLevel, (errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("could not establish any connections to the node "
"%s:%d after %u ms", workerPool->nodeName,
workerPool->nodePort,
NodeConnectionTimeout)));
}
else
{
@ -2854,7 +2855,6 @@ MarkEstablishingSessionsTimedOut(WorkerPool *workerPool)
connection->connectionState == MULTI_CONNECTION_INITIAL)
{
connection->connectionState = MULTI_CONNECTION_TIMED_OUT;
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
}
}
}
@ -3012,10 +3012,6 @@ ConnectionStateMachine(WorkerSession *session)
* the state machines might have already progressed and used
* new pools/sessions instead. That's why we terminate the
* connection, clear any state associated with it.
*
* Note that here we don't increment the failed connection
* stat counter because MarkEstablishingSessionsTimedOut()
* already did that.
*/
connection->connectionState = MULTI_CONNECTION_FAILED;
break;
@ -3026,12 +3022,7 @@ ConnectionStateMachine(WorkerSession *session)
ConnStatusType status = PQstatus(connection->pgConn);
if (status == CONNECTION_OK)
{
/*
* Connection was already established, possibly a cached
* connection.
*/
bool newConnection = false;
HandleMultiConnectionSuccess(session, newConnection);
HandleMultiConnectionSuccess(session);
UpdateConnectionWaitFlags(session,
WL_SOCKET_READABLE | WL_SOCKET_WRITEABLE);
break;
@ -3039,7 +3030,6 @@ ConnectionStateMachine(WorkerSession *session)
else if (status == CONNECTION_BAD)
{
connection->connectionState = MULTI_CONNECTION_FAILED;
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
break;
}
@ -3055,7 +3045,6 @@ ConnectionStateMachine(WorkerSession *session)
if (pollMode == PGRES_POLLING_FAILED)
{
connection->connectionState = MULTI_CONNECTION_FAILED;
IncrementStatCounterForMyDb(STAT_CONNECTION_ESTABLISHMENT_FAILED);
}
else if (pollMode == PGRES_POLLING_READING)
{
@ -3073,12 +3062,7 @@ ConnectionStateMachine(WorkerSession *session)
}
else
{
/*
* Connection was not established befoore (!= CONNECTION_OK)
* but PQconnectPoll() did so now.
*/
bool newConnection = true;
HandleMultiConnectionSuccess(session, newConnection);
HandleMultiConnectionSuccess(session);
UpdateConnectionWaitFlags(session,
WL_SOCKET_READABLE | WL_SOCKET_WRITEABLE);
@ -3134,7 +3118,7 @@ ConnectionStateMachine(WorkerSession *session)
*
* We can only retry connection when the remote transaction has
* not started over the connection. Otherwise, we'd have to deal
* with restoring the transaction state, which is beyond our
* with restoring the transaction state, which iis beyond our
* purpose at this time.
*/
RemoteTransaction *transaction = &connection->remoteTransaction;
@ -3156,11 +3140,6 @@ ConnectionStateMachine(WorkerSession *session)
break;
}
/*
* Here we don't increment the connection stat counter for failed
* connections because we don't track the connections that we could
* establish but lost later.
*/
connection->connectionState = MULTI_CONNECTION_FAILED;
break;
}
@ -3323,12 +3302,12 @@ HasUnfinishedTaskForSession(WorkerSession *session)
* connection's state.
*/
static void
HandleMultiConnectionSuccess(WorkerSession *session, bool newConnection)
HandleMultiConnectionSuccess(WorkerSession *session)
{
MultiConnection *connection = session->connection;
WorkerPool *workerPool = session->workerPool;
MarkConnectionConnected(connection, newConnection);
MarkConnectionConnected(connection);
ereport(DEBUG4, (errmsg("established connection to %s:%d for "
"session %ld in %ld microseconds",
@ -3684,8 +3663,13 @@ UpdateConnectionWaitFlags(WorkerSession *session, int waitFlags)
return;
}
#if PG_VERSION_NUM >= PG_VERSION_15
/* always detect closed sockets */
connection->waitFlags = waitFlags | WL_SOCKET_CLOSED;
#else
connection->waitFlags = waitFlags;
#endif
/* without signalling the execution, the flag changes won't be reflected */
execution->waitFlagsChanged = true;
@ -3710,11 +3694,13 @@ CheckConnectionReady(WorkerSession *session)
return false;
}
#if PG_VERSION_NUM >= PG_VERSION_15
if ((session->latestUnconsumedWaitEvents & WL_SOCKET_CLOSED) != 0)
{
connection->connectionState = MULTI_CONNECTION_LOST;
return false;
}
#endif
/* try to send all pending data */
int sendStatus = PQflush(connection->pgConn);

View File

@ -43,9 +43,8 @@
#include "distributed/multi_executor.h"
#include "distributed/multi_router_planner.h"
#include "distributed/multi_server_executor.h"
#include "distributed/query_stats.h"
#include "distributed/shard_utils.h"
#include "distributed/stats/query_stats.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/subplan_execution.h"
#include "distributed/worker_log_messages.h"
#include "distributed/worker_protocol.h"
@ -207,7 +206,7 @@ CitusBeginScan(CustomScanState *node, EState *estate, int eflags)
if (distributedPlan->modifyQueryViaCoordinatorOrRepartition != NULL)
{
/*
* INSERT..SELECT / MERGE via coordinator or re-partitioning are special because
* INSERT..SELECT via coordinator or re-partitioning are special because
* the SELECT part is planned separately.
*/
return;
@ -263,19 +262,8 @@ CitusExecScan(CustomScanState *node)
if (!scanState->finishedRemoteScan)
{
bool isMultiTaskPlan = IsMultiTaskPlan(scanState->distributedPlan);
AdaptiveExecutor(scanState);
if (isMultiTaskPlan)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
scanState->finishedRemoteScan = true;
}

View File

@ -62,7 +62,7 @@ TaskListRequiresRollback(List *taskList)
}
Task *task = (Task *) linitial(taskList);
if (task->cannotBeExecutedInTransaction)
if (task->cannotBeExecutedInTransction)
{
/* vacuum, create index concurrently etc. */
return false;
@ -165,7 +165,7 @@ TaskListCannotBeExecutedInTransaction(List *taskList)
Task *task = NULL;
foreach_declared_ptr(task, taskList)
{
if (task->cannotBeExecutedInTransaction)
if (task->cannotBeExecutedInTransction)
{
return true;
}

View File

@ -50,7 +50,6 @@
#include "distributed/repartition_executor.h"
#include "distributed/resource_lock.h"
#include "distributed/shardinterval_utils.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/subplan_execution.h"
#include "distributed/transaction_management.h"
#include "distributed/version_compat.h"
@ -179,22 +178,6 @@ NonPushableInsertSelectExecScan(CustomScanState *node)
targetRelation,
binaryFormat);
if (list_length(distSelectTaskList) <= 1)
{
/*
* Probably we will never get here for a repartitioned
* INSERT..SELECT because when the source is a single shard
* table, we should most probably choose to use
* MODIFY_WITH_SELECT_VIA_COORDINATOR, but we still keep this
* here.
*/
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
/*
* At this point select query has been executed on workers and results
* have been fetched in such a way that they are colocated with corresponding
@ -215,15 +198,6 @@ NonPushableInsertSelectExecScan(CustomScanState *node)
taskList, tupleDest,
hasReturning);
if (list_length(taskList) <= 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
executorState->es_processed = rowsInserted;
if (SortReturning && hasReturning)
@ -298,15 +272,6 @@ NonPushableInsertSelectExecScan(CustomScanState *node)
SortTupleStore(scanState);
}
}
if (list_length(prunedTaskList) <= 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
}
else
{
@ -348,12 +313,6 @@ ExecutePlanIntoColocatedIntermediateResults(Oid targetRelationId,
int partitionColumnIndex = PartitionColumnIndexFromColumnList(targetRelationId,
columnNameList);
/*
* We don't track query counters for the COPY commands that are executed to
* prepare intermediate results.
*/
const bool trackQueryCounters = false;
/* set up a DestReceiver that copies into the intermediate table */
const bool publishableData = true;
CitusCopyDestReceiver *copyDest = CreateCitusCopyDestReceiver(targetRelationId,
@ -361,8 +320,7 @@ ExecutePlanIntoColocatedIntermediateResults(Oid targetRelationId,
partitionColumnIndex,
executorState,
intermediateResultIdPrefix,
publishableData,
trackQueryCounters);
publishableData);
ExecutePlanIntoDestReceiver(selectPlan, paramListInfo, (DestReceiver *) copyDest);
@ -391,20 +349,13 @@ ExecutePlanIntoRelation(Oid targetRelationId, List *insertTargetList,
int partitionColumnIndex = PartitionColumnIndexFromColumnList(targetRelationId,
columnNameList);
/*
* We want to track query counters for the COPY commands that are executed to
* perform the final INSERT for such INSERT..SELECT queries.
*/
const bool trackQueryCounters = true;
/* set up a DestReceiver that copies into the distributed table */
const bool publishableData = true;
CitusCopyDestReceiver *copyDest = CreateCitusCopyDestReceiver(targetRelationId,
columnNameList,
partitionColumnIndex,
executorState, NULL,
publishableData,
trackQueryCounters);
publishableData);
ExecutePlanIntoDestReceiver(selectPlan, paramListInfo, (DestReceiver *) copyDest);

View File

@ -295,6 +295,7 @@ PrepareIntermediateResultBroadcast(RemoteFileDestReceiver *resultDest)
if (resultDest->writeLocalFile)
{
const int fileFlags = (O_APPEND | O_CREAT | O_RDWR | O_TRUNC | PG_BINARY);
const int fileMode = (S_IRUSR | S_IWUSR);
/* make sure the directory exists */
CreateIntermediateResultsDirectory();
@ -302,7 +303,8 @@ PrepareIntermediateResultBroadcast(RemoteFileDestReceiver *resultDest)
const char *fileName = QueryResultFileName(resultId);
resultDest->fileCompat = FileCompatFromFileStart(FileOpenForTransmit(fileName,
fileFlags));
fileFlags,
fileMode));
}
WorkerNode *workerNode = NULL;
@ -604,7 +606,7 @@ CreateIntermediateResultsDirectory(void)
{
char *resultDirectory = IntermediateResultsDirectory();
int makeOK = MakePGDirectory(resultDirectory);
int makeOK = mkdir(resultDirectory, S_IRWXU);
if (makeOK != 0)
{
if (errno == EEXIST)
@ -974,6 +976,7 @@ FetchRemoteIntermediateResult(MultiConnection *connection, char *resultId)
StringInfo copyCommand = makeStringInfo();
const int fileFlags = (O_APPEND | O_CREAT | O_RDWR | O_TRUNC | PG_BINARY);
const int fileMode = (S_IRUSR | S_IWUSR);
PGconn *pgConn = connection->pgConn;
int socket = PQsocket(pgConn);
@ -995,7 +998,7 @@ FetchRemoteIntermediateResult(MultiConnection *connection, char *resultId)
PQclear(result);
File fileDesc = FileOpenForTransmit(localPath, fileFlags);
File fileDesc = FileOpenForTransmit(localPath, fileFlags, fileMode);
FileCompat fileCompat = FileCompatFromFileStart(fileDesc);
while (true)

View File

@ -104,8 +104,8 @@
#include "distributed/query_utils.h"
#include "distributed/relation_access_tracking.h"
#include "distributed/remote_commands.h" /* to access LogRemoteCommands */
#include "distributed/stats/stat_tenants.h"
#include "distributed/transaction_management.h"
#include "distributed/utils/citus_stat_tenants.h"
#include "distributed/version_compat.h"
#include "distributed/worker_protocol.h"
@ -569,7 +569,7 @@ LogLocalCommand(Task *task)
*
* One slightly different case is modifications to replicated tables
* (e.g., reference tables) where a single task ends in two separate tasks
* and the local task is added to localTaskList and the remaining ones to
* and the local task is added to localTaskList and the remaning ones to
* the remoteTaskList.
*/
void

View File

@ -26,7 +26,6 @@
#include "distributed/multi_partitioning_utils.h"
#include "distributed/multi_router_planner.h"
#include "distributed/repartition_executor.h"
#include "distributed/stats/stat_counters.h"
#include "distributed/subplan_execution.h"
static void ExecuteSourceAtWorkerAndRepartition(CitusScanState *scanState);
@ -167,21 +166,6 @@ ExecuteSourceAtWorkerAndRepartition(CitusScanState *scanState)
distSourceTaskList, partitionColumnIndex,
targetRelation, binaryFormat);
if (list_length(distSourceTaskList) <= 1)
{
/*
* Probably we will never get here for a repartitioned MERGE
* because when the source is a single shard table, we should
* most probably choose to use ExecuteSourceAtCoordAndRedistribution(),
* but we still keep this here.
*/
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
ereport(DEBUG1, (errmsg("Executing final MERGE on workers using "
"intermediate results")));
@ -209,16 +193,6 @@ ExecuteSourceAtWorkerAndRepartition(CitusScanState *scanState)
tupleDest,
hasReturning,
paramListInfo);
if (list_length(taskList) <= 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
executorState->es_processed = rowsMerged;
}
@ -245,7 +219,6 @@ ExecuteSourceAtCoordAndRedistribution(CitusScanState *scanState)
copyObject(distributedPlan->selectPlanForModifyViaCoordinatorOrRepartition);
char *intermediateResultIdPrefix = distributedPlan->intermediateResultIdPrefix;
bool hasReturning = distributedPlan->expectResults;
bool hasNotMatchedBySource = HasMergeNotMatchedBySource(mergeQuery);
int partitionColumnIndex = distributedPlan->sourceResultRepartitionColumnIndex;
/*
@ -260,7 +233,7 @@ ExecuteSourceAtCoordAndRedistribution(CitusScanState *scanState)
ereport(DEBUG1, (errmsg("Collect source query results on coordinator")));
List *prunedTaskList = NIL, *emptySourceTaskList = NIL;
List *prunedTaskList = NIL;
HTAB *shardStateHash =
ExecuteMergeSourcePlanIntoColocatedIntermediateResults(
targetRelationId,
@ -282,8 +255,7 @@ ExecuteSourceAtCoordAndRedistribution(CitusScanState *scanState)
* We cannot actually execute MERGE INTO ... tasks that read from
* intermediate results that weren't created because no rows were
* written to them. Prune those tasks out by only including tasks
* on shards with connections; however, if the MERGE INTO includes
* a NOT MATCHED BY SOURCE clause we need to include the task.
* on shards with connections.
*/
Task *task = NULL;
foreach_declared_ptr(task, taskList)
@ -296,28 +268,11 @@ ExecuteSourceAtCoordAndRedistribution(CitusScanState *scanState)
{
prunedTaskList = lappend(prunedTaskList, task);
}
else if (hasNotMatchedBySource)
{
emptySourceTaskList = lappend(emptySourceTaskList, task);
}
}
if (emptySourceTaskList != NIL)
{
ereport(DEBUG1, (errmsg("MERGE has NOT MATCHED BY SOURCE clause, "
"execute MERGE on all shards")));
AdjustTaskQueryForEmptySource(targetRelationId, mergeQuery, emptySourceTaskList,
intermediateResultIdPrefix);
prunedTaskList = list_concat(prunedTaskList, emptySourceTaskList);
}
if (prunedTaskList == NIL)
{
/*
* No task to execute, but we still increment STAT_QUERY_EXECUTION_SINGLE_SHARD
* as per our convention.
*/
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
/* No task to execute */
return;
}
@ -337,16 +292,6 @@ ExecuteSourceAtCoordAndRedistribution(CitusScanState *scanState)
tupleDest,
hasReturning,
paramListInfo);
if (list_length(prunedTaskList) == 1)
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_SINGLE_SHARD);
}
else
{
IncrementStatCounterForMyDb(STAT_QUERY_EXECUTION_MULTI_SHARD);
}
executorState->es_processed = rowsMerged;
}
@ -372,12 +317,6 @@ ExecuteMergeSourcePlanIntoColocatedIntermediateResults(Oid targetRelationId,
List *columnNameList =
BuildColumnNameListFromTargetList(targetRelationId, sourceTargetList);
/*
* We don't track query counters for the COPY commands that are executed to
* prepare intermediate results.
*/
const bool trackQueryCounters = false;
/* set up a DestReceiver that copies into the intermediate file */
const bool publishableData = false;
CitusCopyDestReceiver *copyDest = CreateCitusCopyDestReceiver(targetRelationId,
@ -385,8 +324,7 @@ ExecuteMergeSourcePlanIntoColocatedIntermediateResults(Oid targetRelationId,
partitionColumnIndex,
executorState,
intermediateResultIdPrefix,
publishableData,
trackQueryCounters);
publishableData);
/* We can skip when writing to intermediate files */
copyDest->skipCoercions = true;

View File

@ -168,7 +168,7 @@ CitusExecutorRun(QueryDesc *queryDesc,
executorBoundParams = queryDesc->params;
/*
* We do some potentially time consuming operations ourself now before we hand off
* We do some potentially time consuming operations our self now before we hand of
* control to postgres' executor. To make sure that time spent is accurately measured
* we remove the totaltime instrumentation from the queryDesc. Instead we will start
* and stop the instrumentation of the total time and put it back on the queryDesc

View File

@ -32,7 +32,7 @@
#include "distributed/hash_helpers.h"
#include "distributed/multi_executor.h"
#include "distributed/multi_server_executor.h"
#include "distributed/stats/query_stats.h"
#include "distributed/query_stats.h"
#include "distributed/tuplestore.h"
#include "distributed/version_compat.h"
@ -140,6 +140,19 @@ static void CitusQueryStatsRemoveExpiredEntries(HTAB *existingQueryIdHash);
void
InitializeCitusQueryStats(void)
{
/* on PG 15, we use shmem_request_hook_type */
#if PG_VERSION_NUM < PG_VERSION_15
/* allocate shared memory */
if (!IsUnderPostmaster)
{
RequestAddinShmemSpace(CitusQueryStatsSharedMemSize());
elog(LOG, "requesting named LWLockTranch for %s", STATS_SHARED_MEM_NAME);
RequestNamedLWLockTranche(STATS_SHARED_MEM_NAME, 1);
}
#endif
/* Install hook */
prev_shmem_startup_hook = shmem_startup_hook;
shmem_startup_hook = CitusQueryStatsShmemStartup;

Some files were not shown because too many files have changed in this diff Show More