external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,057 Commits
683 Branches
167 Tags
134 MiB
Commit Graph

3506 Commits (261c97d151715fbabb75ea1737201fa06c83e0e8)

Author SHA1 Message Date
SongYoungUk 743c9bbf87
fix #7715 - add assign hook for CDC library path adjustment (#8025) 
DESCRIPTION: Automatically updates dynamic_library_path when CDC is
enabled

fix : #7715 

According to the documentation and `pg_settings`, the context of the
`citus.enable_change_data_capture` parameter is user.

However, changing this parameter — even as a superuser — doesn't work as
expected: while the initial copy phase works correctly, subsequent
change events are not propagated.

This appears to be due to the fact that `dynamic_library_path` is only
updated to `$libdir/citus_decoders:$libdir` when the server is restarted
and the `_PG_init` function is invoked.

To address this, I added an `EnableChangeDataCaptureAssignHook` that
automatically updates `dynamic_library_path` at runtime when
`citus.enable_change_data_capture` is enabled, ensuring that the CDC
decoder libraries are properly loaded.

Note that `dynamic_library_path` is already a `superuser`-context
parameter in base PostgreSQL, so updating it from within the assign hook
should be safe and consistent with PostgreSQL’s configuration model.

If there’s any reason this approach might be problematic or if there’s a
preferred alternative, I’d appreciate any feedback.




cc. @jy-min

---------

Co-authored-by: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Co-authored-by: ibrahim halatci <ihalatci@gmail.com>
 2025-07-18 11:07:17 +03:00
Mehmet YILMAZ 5d805eb10b
PG18 - Adapt columnar stripe metadata updates (#8030) 
Fixes #8019

**Background / Problem**
- PostgreSQL 18 (commit
[a07e03f…](a07e03fd8f))
removed `heap_inplace_update()` and related helpers.
- Citus’ columnar writer relied on that API in
`UpdateStripeMetadataRow()` to patch the `columnar_stripe` catalog row
with the stripe file-offset, size, and row-count.
- Building the extension against PG 18 therefore failed at link-time
and, if stubbed out, left `file_offset = 0`, causing every insert to
abort with
`ERROR: attempted columnar write … to invalid logical offset: 0`



**Scope of This PR**

- Keep the fast-path on PG 12–17 (`heap_inplace_update()` unchanged).
- Switch to `CatalogTupleUpdate()` on PG 18+, matching core’s new
catalog-update API.
- Bump the lock level from `AccessShareLock` → `RowExclusiveLock` when
the normal heap-update path is taken.
- No behavioral changes for users on PG ≤ 17
 2025-07-17 15:15:43 +03:00
Mehmet YILMAZ da24ede835
Support PostgreSQL 18’s new RTE kinds in Citus deparser (#8023) 
Fixes #8020 

PostgreSQL 18 introduces two new, *pseudo* rangetable‐entry kinds that
Citus’ downstream deparser must recognize:

1. **Pulled-up shard RTE clones** (`CITUS_RTE_SHARD` with `relid ==
InvalidOid`)
2. **Grouping-step RTE** (`RTE_GROUP`, alias `*GROUP*`, not actually in
the FROM clause)

Without special handling, Citus crashes or emits invalid SQL when
running against PG 18beta1:

* **`ERROR: could not open relation with OID 0`**
Citus was unconditionally calling `relation_open(rte->relid,…)` on
entries whose `relid` is 0.
* **`ERROR: missing FROM-clause entry for table "*GROUP*"`**
Citus’ `set_rtable_names()` assigned the synthetic `*GROUP*` alias but
never printed a matching FROM item.

This PR teaches Citus’ `ruleutils_18.c` to skip catalog lookups for RTEs
without valid OIDs and to suppress the grouping-RTE alias, restoring
compatibility with both PG 17 and PG 18.

---

## Background

* **Upstream commit
[[247dea8](247dea89f7)**
Introduced `RTE_GROUP` for the grouping step so that multiple subqueries
in `GROUP BY`/`HAVING` can be deduplicated and planned correctly.
* **Citus PR
[[#6428](https://github.com/citusdata/citus/pull/6428)](https://github.com/citusdata/citus/pull/6428)**
Added initial support for treating shard RTEs like real
relations—calling `relation_open()` to pick up renamed-column fixes.
Worked fine on PG 11–17, but PG 18’s pull-up logic clones those shard
RTEs with `relid=0`, leading to OID 0 crashes.

---

## Changes

1. **Guard `relation_open()`**
In `set_relation_column_names()`, only call `relation_open(rte->relid,
…)` when

   ```c
   OidIsValid(rte->relid)
   ```

Prevents the “could not open relation with OID 0” crash on both
pulled-up shards and synthetic RTEs.

2. **Handle pulled-up shards** (`CITUS_RTE_SHARD` with `relid=0`)
Copy column names directly from `rte->eref->colnames` instead of hitting
the catalog.

3. **Handle grouping RTE** (`RTE_GROUP`)

* **In `set_relation_column_names()`**: fallback to
`rte->eref->colnames` for `RTE_GROUP`.
   * **In `set_rtable_names()`**: explicitly assign

     ```c
     refname = NULL;  /* never show *GROUP* in FROM */
     ```

     so that no `*GROUP*` alias is ever printed.

   **Why this is required:**
PostgreSQL 18’s parser now represents the grouping step with a synthetic
RTE whose alias is always `*GROUP*`—and that RTE is **never** actually
listed in the `FROM` clause. If Citus’ deparser assigns and emits
`*GROUP*` as a table reference, the pushed-down SQL becomes:

   ```sql
SELECT *GROUP*.mygroupcol … -- but there is no “*GROUP*” in the FROM
list
   ```

   Workers then fail:

   ```
   ERROR: missing FROM-clause entry for table "*GROUP*"
   ```

By setting `refname = NULL` for `RTE_GROUP` in `set_rtable_names()`, the
deparser prints just the column name unqualified, exactly matching
upstream PG 18’s behavior and yielding valid SQL on the workers.

4. **Maintain existing behavior on PG 15–17**

* Shard RTEs *with* valid `relid` still open the catalog to pick up
renamed-column fixes.
   * No impact on other RTE kinds or versions prior to PG 18.

---
 2025-07-17 13:15:31 +03:00
Mehmet YILMAZ 5005be31e6
PG18 - Handle PG18’s synthetic `RTE_GROUP` in `FindReferencedTableColumn` for correct GROUP BY pushdown (#8034) 
Fixes #8032 

PostgreSQL 18 introduces a dedicated “grouping-step” range table entry
(`RTE_GROUP`) whose target columns are exactly the expressions in our
`GROUP BY` clause, rather than hiding them as `resjunk` items. In
Citus’s distributed planner, the function `FindReferencedTableColumn`
must be able to map from a `Var` referencing a grouped column back to
the underlying table column. Without special handling for `RTE_GROUP`,
queries that rely on pushdown of `GROUP BY` expressions can fail or
mis-identify their target columns.

This PR adds support for `RTE_GROUP` in Citus when built against PG 18
or later, ensuring that:

* Each grouped expression is correctly resolved.
* The pushdown planner can trace a `Var`’s `varattno` into the
corresponding `groupexprs` list.
* Existing behavior on PG < 18 is unchanged.

---

## What’s Changed

In **`src/backend/distributed/planner/multi_logical_optimizer.c`**,
inside `FindReferencedTableColumn`:

* **Under** `#if PG_VERSION_NUM >= PG_VERSION_18`
  Introduce an `else if` branch for

  ```c
  rangeTableEntry->rtekind == RTE_GROUP
  ```

* **Extraction of grouped expressions:**

  ```c
  List *groupexprs   = rangeTableEntry->groupexprs;
  AttrNumber groupIndex = candidateColumn->varattno - 1;
  ```

* **Safety check** to guard against malformed `Var` numbers:

  ```c
  if (groupIndex < 0 || groupIndex >= list_length(groupexprs))
      return;    /* malformed Var */
  ```

* **Recursive descent:**
  Fetch the corresponding expression from `groupexprs` and call

  ```c
  FindReferencedTableColumn(groupExpr, parentQueryList, query,
                            column, rteContainingReferencedColumn,
                            skipOuterVars);
  ```

so that the normal resolution logic applies to the underlying
expression.

* **Unchanged code path** for PG < 18 and for other `rtekind` values.

---
 2025-07-16 23:23:14 +03:00
Mehmet YILMAZ 9e42f3f2c4
Add PG 18Beta1 compatibility (Build + RuleUtils) (#7981) 
This PR provides successful build against PG18Beta1. RuleUtils PR was
reviewed separately: #8010

## PG 18Beta1–related changes for building Citus


### TupleDesc / Attr layout

**What changed in PG:** Postgres consolidated the
`TupleDescData.attrs[]` array into a more compact representation. Direct
field access (tupdesc->attrs[i]) was replaced by the new
`TupleDescAttr()` API.

**Citus adaptation:** Everywhere we previously used
`tupdesc->attrs[...]`, we now call `TupleDescAttr(tupdesc, idx)` (or our
own `Attr()` macro) under a compatibility guard.
*
5983a4cffc

General Logic:

* Use `Attr(...)` in places where `columnar_version_compat.h` is
included. This avoids the need to sprinkle `#if PG_VERSION_NUM` guards
around each attribute access.

* Use `TupleDescAttr(tupdesc, i)` when the relevant PostgreSQL header is
already included and the additional macro indirection is unnecessary.


### Collation‐aware `LIKE`

**What changed in PG:** The `textlike` operator now requires an explicit
collation, to avoid ambiguous‐collation errors. Core code switched from
`DirectFunctionCall2(textlike, ...)` to
`DirectFunctionCall2Coll(textlike, DEFAULT_COLLATION_OID, ...)`.

**Citus adaptation:** In `remote_commands.c` and any other LIKE call, we
now use `DirectFunctionCall2Coll(textlike, DEFAULT_COLLATION_OID, ...)`
and `#include <utils/pg_collation.h>`.

*
85b7efa1cd

### Columnar storage API

* Adapt `columnar_relation_set_new_filelocator` (and related init
routines) for PG 18’s revised SMGR and storage-initialization hooks.
* Pull in the new headers (`explain_format.h`,
`columnar_version_compat.h`) so the columnar module compiles cleanly
against PG 18.
- heap_modify_tuple + heap_inplace_update only exist on PG < 18; on PG18
the in-place helper was removed upstream


-
a07e03fd8f

### OpenSSL / TLS integration

**What changed in PG:** Moved from the legacy `SSL_library_init()` to
`OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL)`, updated certificate
API calls (`X509_getm_notBefore`, `X509_getm_notAfter`), and
standardized on `TLS_method()`.

**Citus adaptation:** We now `#include <openssl/opensslv.h>` and use
`#if OPENSSL_VERSION_NUMBER >= 0x10100000L` to choose between`
OPENSSL_init_ssl()` or `SSL_library_init()`, and wrap`
X509_gmtime_adj()` calls around the new accessor functions.

*
6c66b7443c


### Adapt `ExtractColumns()` to the new PG-18 `expandRTE()` signature

PostgreSQL 18
80feb727c8
added a fourth argument of type `VarReturningType` to `expandRTE()`, so
calls that used the old 7-parameter form no longer compile. This patch:

* Wraps the `expandRTE(...)` call in a `#if PG_VERSION_NUM >= 180000`
guard.
* On PG 18+ passes the new `VAR_RETURNING_DEFAULT` argument before
`location`.
* On PG 15–17 continues to call the original 7-arg form.
* Adds the necessary includes (`parser/parse_relation.h` for `expandRTE`
and `VarReturningType`, and `pg_version_constants.h` for
`PG_VERSION_NUM`).



### Adapt `ExecutorStart`/`ExecutorRun` hooks to PG-18’s new signatures

PostgreSQL 18
525392d572
changed the signatures of the executor hooks:

* `ExecutorStart_hook` now returns `bool` instead of `void`, and
* `ExecutorRun_hook` drops its old `run_once` argument.

This patch preserves Citus’s existing hook logic by:

1. **Adding two adapter functions** under `#if PG_VERSION_NUM >=
PG_VERSION_18`:

   * `citus_executor_start_adapter(QueryDesc *queryDesc, int eflags)`
Calls the old `CitusExecutorStart(queryDesc, eflags)` and then returns
`true` to satisfy the new hook’s `bool` return type.
* `citus_executor_run_adapter(QueryDesc *queryDesc, ScanDirection
direction, uint64 count)`
Calls the old `CitusExecutorRun(queryDesc, direction, count, true)`
(passing `true` for the dropped `run_once` argument), and returns
`void`.

2. **Installing the adapters** in `_PG_init()` instead of the original
hooks when building against PG 18+:

   ```c
   #if PG_VERSION_NUM >= PG_VERSION_18
       ExecutorStart_hook = citus_executor_start_adapter;
       ExecutorRun_hook   = citus_executor_run_adapter;
   #else
       ExecutorStart_hook = CitusExecutorStart;
       ExecutorRun_hook   = CitusExecutorRun;
   #endif
   ```
   
### Adapt to PG-18’s removal of the “run\_once” flag from
ExecutorRun/PortalRun

PostgreSQL commit
[[3eea7a0](3eea7a0c97)
rationalized the executor’s parallelism logic by moving the “execute a
plan only once” check into `ExecutePlan()` itself and dropping the old
`bool run_once` argument from the public APIs:

```diff
- void ExecutorRun(QueryDesc *queryDesc,
-                  ScanDirection direction,
-                  uint64 count,
-                  bool run_once);
+ void ExecutorRun(QueryDesc *queryDesc,
+                  ScanDirection direction,
+                  uint64 count);
```

(and similarly for `PortalRun()`).

To stay compatible across PG 15–18, Citus now:

1. **Updates all internal calls** to `ExecutorRun(...)` and
`PortalRun(...)`:

* On PG 18+, use the new three-argument form (`ExecutorRun(qd, dir,
count)`).
* On PG 15–17, keep the old four-arg form (`ExecutorRun(qd, dir, count,
true)`) under a `#if PG_VERSION_NUM < 180000` guard.

2. **Guards the dispatcher hooks** via the adapter functions (from the
earlier patch) so that Citus’s executor hooks continue to work under
both the old and new signatures.


### Adapt to PG-18’s shortened PortalRun signature

PostgreSQL 18’s refactoring (see commit
[3eea7a0](3eea7a0c97))
also removed the old run_once and alternate‐dest arguments from the
public PortalRun() API. The signature changed from:



```diff
- bool PortalRun(Portal portal,
-                long count,
-                bool isTopLevel,
-                bool run_once,
-                DestReceiver *dest,
-                DestReceiver *altdest,
-                QueryCompletion *qc);
+ bool PortalRun(Portal portal,
+                long count,
+                bool isTopLevel,
+                DestReceiver *dest,
+                DestReceiver *altdest,
+                QueryCompletion *qc);
```

To support both versions in Citus, we:

1. **Version-guard each call** to `PortalRun()`:

   * **On PG 18+** invoke the new 6-argument form.
* **On PG 15–17** fall back to the legacy 7-argument form, passing
`true` for `run_once`.
   
### Add support for PG-18’s new `plansource` argument in
`PortalDefineQuery`**

PostgreSQL 18 extended the `PortalDefineQuery` API to carry a
`CachedPlanSource *plansource` pointer so that the portal machinery can
track cached‐plan invalidation (as introduced alongside deferred-locking
in commit
525392d572.
To remain compatible across PG 15–18, Citus now wraps its calls under a
version guard:

```diff
-   PortalDefineQuery(portal, NULL, sql, commandTag, plantree_list, NULL);
+#if PG_VERSION_NUM >= 180000
+   /* PG 18+: seven-arg signature (adds plansource) */
+   PortalDefineQuery(
+       portal,
+       NULL,            /* no prepared-stmt name */
+       sql,             /* the query text */
+       commandTag,      /* the CommandTag */
+       plantree_list,   /* List of PlannedStmt* */
+       NULL,            /* no CachedPlan */
+       NULL             /* no CachedPlanSource */
+   );
+#else
+   /* PG 15–17: six-arg signature */
+   PortalDefineQuery(
+       portal,
+       NULL,            /* no prepared-stmt name */
+       sql,             /* the query text */
+       commandTag,      /* the CommandTag */
+       plantree_list,   /* List of PlannedStmt* */
+       NULL             /* no CachedPlan */
+   );
+#endif
```


### Adapt ExecInitRangeTable() calls to PG-18’s new signature

PostgreSQL commit
[cbc127917e04a978a788b8bc9d35a70244396d5b](cbc127917e)
overhauled the planner API for range‐table initialization:

**PG 18+**: added a fourth `Bitmapset *unpruned_relids` argument to
support deferred partition pruning

In Citus’s `create_estate_for_relation()` (in `columnar_metadata.c`), we
now wrap the call in a compile‐time guard so that the code compiles
correctly on all supported PostgreSQL versions:

```
/* Prepare permission info on PG 16+ */
#if PG_VERSION_NUM >= PG_VERSION_16
    List *perminfos = NIL;
    addRTEPermissionInfo(&perminfos, rte);
#else
    List *perminfos = NIL;  /* unused on PG 15 */
#endif

/* Initialize the range table, with the right signature for each PG version */
#if PG_VERSION_NUM >= PG_VERSION_18
    /* PG 18+: four‐arg signature (adds unpruned_relids) */
    ExecInitRangeTable(
        estate,
        list_make1(rte),
        perminfos,
        NULL        /* unpruned_relids: not used by columnar */
    );
#elif PG_VERSION_NUM >= PG_VERSION_16
    /* PG 16–17: three‐arg signature (permInfos) */
    ExecInitRangeTable(
        estate,
        list_make1(rte),
        perminfos
    );
#else
    /* PG 15: two‐arg signature */
    ExecInitRangeTable(
        estate,
        list_make1(rte)
    );
#endif

estate->es_output_cid = GetCurrentCommandId(true);
```

### Adapt `pgstat_report_vacuum()` to PG-18’s new timestamp argument

PostgreSQL commit
[[30a6ed0ce4bb18212ec38cdb537ea4b43bc99b83](30a6ed0ce4)
extended the `pgstat_report_vacuum()` API by adding a `TimestampTz
start_time` parameter at the end so that the VACUUM statistics collector
can record when the operation began:

```diff
/* PG ≤17: four-arg signature */
- void pgstat_report_vacuum(Oid tableoid,
-                           bool shared,
-                           double num_live_tuples,
-                           double num_dead_tuples);
+/* PG ≥18: five-arg signature adds a start_time */
+ void pgstat_report_vacuum(Oid tableoid,
+                           bool shared,
+                           double num_live_tuples,
+                           double num_dead_tuples,
+                           TimestampTz start_time);
```

To support both versions, we now wrap the call in `columnar_tableam.c`
with a version guard, supplying `GetCurrentTimestamp()` for PG-18+:

```c
#if PG_VERSION_NUM >= 180000
    /* PG 18+: include start_timestamp */
    pgstat_report_vacuum(
        RelationGetRelid(rel),
        rel->rd_rel->relisshared,
        Max(new_live_tuples, 0),  /* live tuples */
        0,                        /* dead tuples */
        GetCurrentTimestamp()     /* start time */
    );
#else
    /* PG 15–17: original signature */
    pgstat_report_vacuum(
        RelationGetRelid(rel),
        rel->rd_rel->relisshared,
        Max(new_live_tuples, 0),  /* live tuples */
        0                         /* dead tuples */
    );
#endif
```


### Adapt `ExecuteTaskPlan()` to PG-18’s expanded `CreateQueryDesc()`
signature

PostgreSQL 18 changed `CreateQueryDesc()` from an eight-argument to a
nine-argument call by inserting a `CachedPlan *cplan` parameter
immediately after the `PlannedStmt *plannedstmt` argument (see commit
525392d572).
To remain compatible with PG 15–17, Citus now wraps its invocation in
`local_executor.c` with a version guard:

```diff
-    /* PG15–17: eight-arg CreateQueryDesc without cached plan */
-    QueryDesc *queryDesc = CreateQueryDesc(
-        taskPlan,           /* PlannedStmt *plannedstmt */
-        queryString,        /* const char *sourceText */
-        GetActiveSnapshot(),/* Snapshot snapshot */
-        InvalidSnapshot,    /* Snapshot crosscheck_snapshot */
-        destReceiver,       /* DestReceiver *dest */
-        paramListInfo,      /* ParamListInfo params */
-        queryEnv,           /* QueryEnvironment *queryEnv */
-        0                   /* int instrument_options */
-    );
+#if PG_VERSION_NUM >= 180000
+    /* PG18+: nine-arg CreateQueryDesc with a CachedPlan slot */
+    QueryDesc *queryDesc = CreateQueryDesc(
+        taskPlan,           /* PlannedStmt *plannedstmt */
+        NULL,               /* CachedPlan *cplan (none) */
+        queryString,        /* const char *sourceText */
+        GetActiveSnapshot(),/* Snapshot snapshot */
+        InvalidSnapshot,    /* Snapshot crosscheck_snapshot */
+        destReceiver,       /* DestReceiver *dest */
+        paramListInfo,      /* ParamListInfo params */
+        queryEnv,           /* QueryEnvironment *queryEnv */
+        0                   /* int instrument_options */
+    );
+#else
+    /* PG15–17: eight-arg CreateQueryDesc without cached plan */
+    QueryDesc *queryDesc = CreateQueryDesc(
+        taskPlan,           /* PlannedStmt *plannedstmt */
+        queryString,        /* const char *sourceText */
+        GetActiveSnapshot(),/* Snapshot snapshot */
+        InvalidSnapshot,    /* Snapshot crosscheck_snapshot */
+        destReceiver,       /* DestReceiver *dest */
+        paramListInfo,      /* ParamListInfo params */
+        queryEnv,           /* QueryEnvironment *queryEnv */
+        0                   /* int instrument_options */
+    );
+#endif
```



### Adapt `RelationGetPrimaryKeyIndex()` to PG-18’s new “deferrable\_ok”
flag

PostgreSQL commit
14e87ffa5c
added a new Boolean `deferrable_ok` parameter to
`RelationGetPrimaryKeyIndex()` so that the lock manager can defer
unique‐constraint locks when requested. The API changed from:

```c
RelationGetPrimaryKeyIndex(Relation relation)
```

to:

```c
RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok)
 ```
                
```diff
diff --git a/src/backend/distributed/metadata/node_metadata.c
b/src/backend/distributed/metadata/node_metadata.c
index e3a1b2c..f4d5e6f 100644
--- a/src/backend/distributed/metadata/node_metadata.c
+++ b/src/backend/distributed/metadata/node_metadata.c
@@ -2965,8 +2965,18 @@
     */
- Relation replicaIndex =
index_open(RelationGetPrimaryKeyIndex(pgDistNode),
-                                      AccessShareLock);
+    #if PG_VERSION_NUM >= PG_VERSION_18
+        /* PG 18+ adds a bool "deferrable_ok" parameter */
+        Relation replicaIndex =
+            index_open(
+                RelationGetPrimaryKeyIndex(pgDistNode, false),
+                AccessShareLock);
+    #else
+        Relation replicaIndex =
+            index_open(
+                RelationGetPrimaryKeyIndex(pgDistNode),
+                AccessShareLock);
+    #endif

     ScanKeyInit(&scanKey[0], Anum_pg_dist_node_nodename,
BTEqualStrategyNumber, F_TEXTEQ, CStringGetTextDatum(nodeName));

```
  
  ```diff
  diff --git a/src/backend/distributed/operations/node_protocol.c b/src/backend/distributed/operations/node_protocol.c
index e3a1b2c..f4d5e6f 100644
--- a/src/backend/distributed/operations/node_protocol.c
+++ b/src/backend/distributed/operations/node_protocol.c
@@ -746,7 +746,12 @@
     if (!OidIsValid(idxoid))
     {
-        idxoid = RelationGetPrimaryKeyIndex(rel);
+        /* Determine the index OID of the primary key (PG18 adds a second parameter) */
+#if PG_VERSION_NUM >= PG_VERSION_18
+        idxoid = RelationGetPrimaryKeyIndex(rel, false);
+#else
+        idxoid = RelationGetPrimaryKeyIndex(rel);
+#endif
     }

     return idxoid;

```
  
Because Citus has always taken the lock immediately—just as the old
two-arg call did—we pass `false` to keep that same immediate-lock
behavior. Passing `true` would switch to deferred locking, which we
don’t want.



### Adapt `ExplainOnePlan()` to PG-18’s expanded API

PostgreSQL 18 extended
525392d572
the `ExplainOnePlan()` function to carry the `CachedPlan *` and
`CachedPlanSource *` pointers plus an explicit `query_index`, letting
the EXPLAIN machinery track plan‐source invalidation. The old signature:

```c
/* PG ≤17 */
void
ExplainOnePlan(PlannedStmt *plannedstmt,
               IntoClause *into,
               struct ExplainState *es,
               const char *queryString,
               ParamListInfo params,
               QueryEnvironment *queryEnv,
               const instr_time *planduration,
               const BufferUsage *bufusage);
```

became, in PG 18:

```c
/* PG ≥18 */
void
ExplainOnePlan(PlannedStmt *plannedstmt,
               CachedPlan   *cplan,
               CachedPlanSource *plansource,
               int            query_index,
               IntoClause    *into,
               struct ExplainState *es,
               const char   *queryString,
               ParamListInfo params,
               QueryEnvironment *queryEnv,
               const instr_time *planduration,
               const BufferUsage *bufusage,
               const MemoryContextCounters *mem_counters);
```

To compile under both versions, Citus now wraps each call in
`multi_explain.c` with:

```c
#if PG_VERSION_NUM >= PG_VERSION_18
    /* PG 18+: pass NULL for the new cached‐plan fields and zero for query_index */
    ExplainOnePlan(
        plan,         /* PlannedStmt *plannedstmt */
        NULL,         /* CachedPlan *cplan */
        NULL,         /* CachedPlanSource *plansource */
        0,            /* query_index */
        into,         /* IntoClause *into */
        es,           /* ExplainState *es */
        queryString,  /* const char *queryString */
        params,       /* ParamListInfo params */
        NULL,         /* QueryEnvironment *queryEnv */
        &planduration,/* const instr_time *planduration */
        (es->buffers ? &bufusage : NULL),
        (es->memory  ? &mem_counters : NULL)
    );
#elif PG_VERSION_NUM >= PG_VERSION_17
    /* PG 17: same as before, plus passing mem_counters if enabled */
    ExplainOnePlan(
        plan,
        into,
        es,
        queryString,
        params,
        queryEnv,
        &planduration,
        (es->buffers ? &bufusage : NULL),
        (es->memory ? &mem_counters : NULL)
    );
#else
    /* PG 15–16: original seven-arg form */
    ExplainOnePlan(
        plan,
        into,
        es,
        queryString,
        params,
        queryEnv,
        &planduration,
        (es->buffers ? &bufusage : NULL)
    );
#endif
```


### Adapt to the unified “index interpretation” API in PG 18 (commit
a8025f544854)

PostgreSQL commit
a8025f5448
generalized the old btree‐specific operator‐interpretation API into a
single “index interpretation” interface:

* **Renamed type**:
  `OpBtreeInterpretation` → `OpIndexInterpretation`
* **Renamed function**:
`get_op_btree_interpretation(opno)` →
`get_op_index_interpretation(opno)`
* **Unified field**:
  Each interpretation now carries `cmptype` instead of `strategy`.

To build cleanly on PG 18 while still supporting PG 15–17, Citus’s
shard‐pruning code now wraps these changes:

```c
#include "pg_version_constants.h"

#if PG_VERSION_NUM >= PG_VERSION_18
/* On PG 18+ the btree‐only APIs vanished; alias them to the new generic versions */
typedef OpIndexInterpretation OpBtreeInterpretation;
#define get_op_btree_interpretation(opno)  get_op_index_interpretation(opno)
#define ROWCOMPARE_NE  COMPARE_NE
#endif

/* … later, when checking an interpretation … */
OpBtreeInterpretation *interp =
    (OpBtreeInterpretation *) lfirst(cell);

#if PG_VERSION_NUM >= PG_VERSION_18
    /* use cmptype on PG 18+ */
    if (interp->cmptype == ROWCOMPARE_NE)
#else
    /* use strategy on PG 15–17 */
    if (interp->strategy == ROWCOMPARE_NE)
#endif
{
    /* … */
}
```


### Adapt `create_foreignscan_path()` for PG-18’s revised signature

PostgreSQL commit
e222534679
reordered and removed a couple of parameters in the FDW‐path builder:

* **PG 15–17 signature (11 args)**

  ```c
  create_foreignscan_path(PlannerInfo   *root,
                          RelOptInfo    *rel,
                          PathTarget    *target,
                          double         rows,
                          Cost           startup_cost,
                          Cost           total_cost,
                          List          *pathkeys,
                          Relids         required_outer,
                          Path          *fdw_outerpath,
                          List          *fdw_restrictinfo,
                          List          *fdw_private);
  ```
* **PG 18+ signature (9 args)**

  ```c
  create_foreignscan_path(PlannerInfo   *root,
                          RelOptInfo    *rel,
                          PathTarget    *target,
                          double         rows,
                          int            disabled_nodes,
                          Cost           startup_cost,
                          Cost           total_cost,
                          Relids         required_outer,
                          Path          *fdw_outerpath,
                          List          *fdw_private);
  ```

To support both, Citus now defines a compatibility macro in
`pg_version_compat.h`:

```c
#include "nodes/bitmapset.h"   /* for Relids */
#include "nodes/pg_list.h"     /* for List */
#include "optimizer/pathnode.h" /* for create_foreignscan_path() */

#if PG_VERSION_NUM >= PG_VERSION_18

/* PG18+: drop pathkeys & fdw_restrictinfo, add disabled_nodes */
#define create_foreignscan_path_compat(a, b, c, d, e, f, g, h, i, j, k) \
    create_foreignscan_path(                                            \
        (a),          /* root */                                       \
        (b),          /* rel */                                        \
        (c),          /* target */                                     \
        (d),          /* rows */                                       \
        (0),          /* disabled_nodes (unused by Citus) */           \
        (e),          /* startup_cost */                              \
        (f),          /* total_cost */                                \
        (g),          /* required_outer */                            \
        (h),          /* fdw_outerpath */                             \
        (k)           /* fdw_private */                               \
    )

#else

/* PG15–17: original signature */
#define create_foreignscan_path_compat(a, b, c, d, e, f, g, h, i, j, k) \
    create_foreignscan_path(                                            \
        (a), (b), (c), (d),                                            \
        (e), (f),                                                      \
        (g), (h), (i), (j), (k)                                        \
    )
#endif
```

Now every call to `create_foreignscan_path_compat(...)`—even in tests
like `fake_fdw.c`—automatically picks the correct argument list for
PG 15 through PG 18.



### Drop the obsolete bitmap‐scan hooks on PG 18+

PostgreSQL commit
c3953226a0
cleaned up the `TableAmRoutine` API by removing the two bitmap‐scan
callback slots:

* `scan_bitmap_next_block`
* `scan_bitmap_next_tuple`

Since those hook‐slots no longer exist in PG 18, Citus now wraps their
NULL‐initialization in a `#if PG_VERSION_NUM < PG_VERSION_18` guard. On
PG 15–17 we still explicitly set them to `NULL` (to satisfy the old
struct layout), and on PG 18+ we omit them entirely:

```c

#if PG_VERSION_NUM < PG_VERSION_18
    /* PG 15–17 only: these fields were removed upstream in PG 18 */
    .scan_bitmap_next_block = NULL,
    .scan_bitmap_next_tuple = NULL,
#endif


```


### Adapt `vac_update_relstats()` invocation to PG-18’s new
“all\_frozen” argument

PostgreSQL commit
99f8f3fbbc
extended the `vac_update_relstats()` API by inserting a
`num_all_frozen_pages` parameter between the existing
`num_all_visible_pages` and `hasindex` arguments:

```diff
- /* PG ≤17: */
- void
- vac_update_relstats(Relation relation,
-                    BlockNumber num_pages,
-                    double     num_tuples,
-                    BlockNumber num_all_visible_pages,
-                    bool       hasindex,
-                    TransactionId frozenxid,
-                    MultiXactId  minmulti,
-                    bool      *frozenxid_updated,
-                    bool      *minmulti_updated,
-                    bool       in_outer_xact);
+ /* PG ≥18: adds num_all_frozen_pages */
+ void
+ vac_update_relstats(Relation    relation,
+                    BlockNumber num_pages,
+                    double      num_tuples,
+                    BlockNumber num_all_visible_pages,
+                    BlockNumber num_all_frozen_pages,
+                    bool        hasindex,
+                    TransactionId frozenxid,
+                    MultiXactId  minmulti,
+                    bool      *frozenxid_updated,
+                    bool      *minmulti_updated,
+                    bool       in_outer_xact);
```

To compile cleanly on both PG 15–17 and PG 18+, Citus wraps its call in
a version guard and supplies a zero placeholder for the new field:

```c
#if PG_VERSION_NUM >= 180000
    /* PG 18+: supply explicit “all_frozen” count */
    vac_update_relstats(
        rel,
        new_rel_pages,
        new_live_tuples,
        new_rel_allvisible,    /* allvisible */
        0,                     /* all_frozen */
        nindexes > 0,
        newRelFrozenXid,
        newRelminMxid,
        &frozenxid_updated,
        &minmulti_updated,
        false                  /* in_outer_xact */
    );
#else
    /* PG 15–17: original signature */
    vac_update_relstats(
        rel,
        new_rel_pages,
        new_live_tuples,
        new_rel_allvisible,
        nindexes > 0,
        newRelFrozenXid,
        newRelminMxid,
        &frozenxid_updated,
        &minmulti_updated,
        false                  /* in_outer_xact */
    );
#endif
```

**Why all_frozen = 0?**
Columnar storage never embeds transaction IDs in its pages, so it never
needs to track “all‐frozen” pages the way a heap does. Setting both
allvisible and allfrozen to zero simply tells Postgres “there are no
pages with the visibility or frozen‐status bits set,” matching our
existing behavior.

This change ensures Citus’s VACUUM‐statistic updates work unmodified
across all supported Postgres versions.
 2025-07-16 15:30:41 +03:00
naisila 4cd8bb1b67 Bump Citus version to 13.2devel 2025-06-24 16:21:48 +02:00
Onur Tirtir 55a0d1f730
Add skip_qualify_public param to shard_name() to allow qualifying for "public" schema (#8014) 
DESCRIPTION: Adds skip_qualify_public param to `shard_name()` UDF to
allow qualifying for "public" schema when needed.
 2025-06-02 10:15:32 +03:00
Naisila Puka c98341e4ed
Bump PG versions to 17.5, 16.9, 15.13 (#7986) 
Nontrivial bump because of the following PG15.3 commit
317aba70e
https://github.com/postgres/postgres/commit/317aba70e

Previously, when views were converted to RTE_SUBQUERY the relid
would be cleared in PG15. In this patch of PG15, relid is retained.
Therefore, we add a check with the "relkind and rtekind" to
identify the converted views in 15.13

Sister PR https://github.com/citusdata/the-process/pull/164
Using dev image sha because I encountered the libpq
symlink issue again with "-v219b87c"
 2025-05-22 14:08:03 +02:00
Onur Tirtir 8d2fbca8ef
Fix unsafe memory access in citus_unmark_object_distributed() (#7985) 
_Since we've never released a Citus release that contains the commit
that introduced this bug (see #7461), we don't need to have a
DESCRIPTION line that shows up in release changelog._

From 8 valgrind test targets run for release-13.1 with PG 17.5, we got
1344 stack traces and except one of them, they were all about below
unsafe memory access because this is a very hot code-path that we
execute via our drop trigger.

On main, even `make -C src/test/regress/ check-base-vg` dumps this stack
trace with PG 16/17 to src/test/regress/citus_valgrind_test_log.txt when
executing "multi_cluster_management", and this is not the case with this
PR anymore.

```c
==27337== VALGRINDERROR-BEGIN
==27337== Conditional jump or move depends on uninitialised value(s)
==27337==    at 0x7E26B68: citus_unmark_object_distributed (home/onurctirtir/citus/src/backend/distributed/metadata/distobject.c:113)
==27337==    by 0x7E26CC7: master_unmark_object_distributed (home/onurctirtir/citus/src/backend/distributed/metadata/distobject.c:153)
==27337==    by 0x4BD852: ExecInterpExpr (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execExprInterp.c:758)
==27337==    by 0x4BFD00: ExecInterpExprStillValid (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execExprInterp.c:1870)
==27337==    by 0x51D82C: ExecEvalExprSwitchContext (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/../../../src/include/executor/executor.h:355)
==27337==    by 0x51D8A4: ExecProject (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/../../../src/include/executor/executor.h:389)
==27337==    by 0x51DADB: ExecResult (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/nodeResult.c:136)
==27337==    by 0x4D72ED: ExecProcNodeFirst (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execProcnode.c:464)
==27337==    by 0x4CA394: ExecProcNode (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/../../../src/include/executor/executor.h:273)
==27337==    by 0x4CD34C: ExecutePlan (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execMain.c:1670)
==27337==    by 0x4CAA7C: standard_ExecutorRun (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execMain.c:365)
==27337==    by 0x7E1E475: CitusExecutorRun (home/onurctirtir/citus/src/backend/distributed/executor/multi_executor.c:238)
==27337==  Uninitialised value was created by a heap allocation
==27337==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==27337==    by 0x9AB1F7: AllocSetContextCreateInternal (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/utils/mmgr/aset.c:438)
==27337==    by 0x4E0D56: CreateExprContextInternal (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execUtils.c:261)
==27337==    by 0x4E0E3E: CreateExprContext (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execUtils.c:311)
==27337==    by 0x4E10D9: ExecAssignExprContext (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execUtils.c:490)
==27337==    by 0x51EE09: ExecInitSeqScan (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/nodeSeqscan.c:147)
==27337==    by 0x4D6CE1: ExecInitNode (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execProcnode.c:210)
==27337==    by 0x5243C7: ExecInitSubqueryScan (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/nodeSubqueryscan.c:126)
==27337==    by 0x4D6DD9: ExecInitNode (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execProcnode.c:250)
==27337==    by 0x4F05B2: ExecInitAppend (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/nodeAppend.c:223)
==27337==    by 0x4D6C46: ExecInitNode (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/execProcnode.c:182)
==27337==    by 0x52003D: ExecInitSetOp (home/onurctirtir/.pgenv/src/postgresql-16.2/src/backend/executor/nodeSetOp.c:530)
==27337== 
==27337== VALGRINDERROR-END
```
 2025-05-20 15:22:35 +03:00
Alper Kocatas 088ba75057
Add citus_nodes view (#7968) 
DESCRIPTION: Adds `citus_nodes` view that displays the node name, port,
role, and "active" for nodes in the cluster.

This PR adds `citus_nodes` view to the `pg_catalog` schema. The
`citus_nodes` view is created in the `citus` schema and is used to
display the node name, port, role, and active status of each node in the
`pg_dist_node` table.

The view is granted `SELECT` permission to the `PUBLIC` role and is set
to the `pg_catalog` schema.

Test cases was added to `multi_cluster_management` tests. 

structs.py was modified to add white spaces as `citus_indent` required.

---------

Co-authored-by: Alper Kocatas <alperkocatas@microsoft.com>
 2025-05-14 15:05:12 +03:00
Naisila Puka a18040869a
Error out for queries with outer joins and pseudoconstant quals in PG<17 (#7937) 
PG15 commit d1ef5631e620f9a5b6480a32bb70124c857af4f1
and PG16 commit 695f5deb7902865901eb2d50a70523af655c3a00
disallow replacing joins with scans in queries with pseudoconstant quals.
This commit prevents the set_join_pathlist_hook from being called
if any of the join restrictions is a pseudo-constant.
So in these cases, citus has no info on the join, never sees that
the query has an outer join, and ends up producing an incorrect plan.
PG17 fixes this by commit 9e9931d2bf40e2fea447d779c2e133c2c1256ef3
Therefore, we take this extra measure here for PG versions less than 17.
hasOuterJoin can never be true when set_join_pathlist_hook is absent.
 2025-05-11 21:47:28 +00:00
Mehmet YILMAZ a4040ba5da
Planner: lift volatile target‑list items in `WrapSubquery` to coordinator (prevents sequence‑leap in distributed `INSERT … SELECT`) (#7976) 
This PR fixes #7784 and refactors the `WrapSubquery(Query *subquery)`
function to improve clarity and correctness when handling volatile
expressions in subqueries during Citus insert-select rewriting.

### Background

The `WrapSubquery` function rewrites a query of the form:

```sql
INSERT INTO target_table SELECT ... FROM ...
```

...by wrapping the `SELECT` in a subquery:

```sql
SELECT <outer-TL>
  FROM ( <subquery with volatile expressions replaced with NULL> ) citus_insert_select_subquery
```

This transformation allows:

* **Volatile expressions** (e.g., `nextval`, `now`) **not used in `GROUP
BY` or `ORDER BY`** to be evaluated **exactly once on the coordinator**.
* **Stable/immutable or sort-relevant expressions** to remain in the
worker-executed subquery.
* Placeholder `NULL`s to maintain column alignment in the inner
subquery.

### Fix Details

* Restructured the code into labeled logical sections:

  1. Build wrapper query (`SELECT … FROM (subquery)`)
  2. Rewrite target lists with volatility analysis
  3. Assign and return updated query trees
  
* Preserved existing behavior, focusing on clarity and maintainability.

### How the new code handles volatile items

stage | what we look for | what we do | why
-- | -- | -- | --
scan target list once | 1. `expr_is_volatile(te->expr)` 2.
`te->ressortgroupref != 0` (is the column used in GROUP BY / ORDER BY?)
| decide whether to hoist or keep | we must not hoist an expression the
inner query still needs for sorting/grouping, otherwise its
`SortGroupClause` breaks
volatile & not used in sort/group | deep‑copy the expression into the
outer target list | executes once on the coordinator |  
  | leave a typed `NULL `placeholder (visible, not `resjunk`) in the
inner target list | keeps column numbering stable for helpers that
already ran (reorder, cast); the worker sends a cheap constant |  
stable / immutable, or volatile but used in sort/group | keep the
original expression in the inner list; outer list references it via a
`Var `| workers can evaluate it safely and, if needed, the inner
ORDER BY still works |  

###  Example

Given this query:

```sql
INSERT INTO t SELECT nextval('s'), 42 FROM generate_series(1, 2);
```

The planner rewrites it as:

```sql
SELECT nextval('s'), col2
  FROM (SELECT NULL::bigint AS col1, 42 AS col2 FROM generate_series(1, 2)) citus_insert_select_subquery;
```

This ensures `nextval('s')` is evaluated only once per row on the
**coordinator**, not on each worker node, preserving correct sequence
semantics.

#### **Outer‑Var guard (`FindReferencedTableColumn`)**

Because `WrapSubquery` adds an extra query level, lots of Vars that the
old code never expected become “outer” Vars; without teaching
`FindReferencedTableColumn` to climb that extra level reliably, Citus
would intermittently reject valid foreign keys and even hit asserts.

* Re‑implemented the outer‑Var guard so that the function:

* **Walks deterministically up the query stack** when `skipOuterVars =
false` (default for FK / UNION checks). A new while‑loop copies — rather
than truncates — `parentQueryList` on each hop, eliminating
list‑aliasing that made *issue 5248* fail intermittently in parallel
regressions.

* Handles multi‑level `varlevelsup` in a single loop; never mutates the
caller’s list in place.
 2025-05-06 17:45:49 +03:00
Colm d4dd44e715
Propagate SECURITY LABEL on tables and columns. (#7956) 
Issue #7709 asks for security labels on columns to be propagated, to
support the `anon` extension. Before, Citus supported security labels
on roles (#7735) and this PR adds support for propagating security
labels on tables and columns.

All scenarios that involve propagating metadata for a Citus table now
include the security labels on the table and on the columns of the
table. These scenarios are:
- When a table becomes distributed using `create_distributed_table()` or
`create_reference_table()`, its security labels (if any) are propageted.
- When a security label is defined on a distributed table, or one of its
columns, the label is propagated.
- When a node is added to a Citus cluster, all distributed tables have
their security labels propagated.
- When a column of a distributed table is dropped, any security labels
on the column are also dropped.
- When a column is added to a distributed table, security labels can be
defined on the column and are propagated.
- Security labels on a distributed table or its columns are not
propagated when `citus.enable_metadata_sync` is enabled.

Regress test `seclabel` is extended with tests to cover these scenarios.
The implementation is somewhat involved because it impacts DDL
propagation of Citus tables, but can be broken down as follows:
- distributed_object_ops has `Role_SecLabel`, `Table_SecLabel` and
`Column_SecLabel` to take care of security labels on roles, tables and
columns. `Any_SecLabel` is used for all other security labels and is
essentially a nop.
- Deparser support - `DeparseRoleSecLabelStmt()`,
`DeparseTableSecLabelStmt()` and `DeparseColumnSecLabelStmt()` take care
of deparsing security label statements on roles, tables and columns
respectively.
- When reconstructing the DDL for a citus table, security labels on the
table or its columns are included by having
`GetPreLoadTableCreationCommands()` call a new function
`CreateSecurityLabelCommands()` to take care of any security labels on
the table or its columns.
- When changing a distributed table name to a shard name before running
a command locally on a worker, function `RelayEventExtendNames()` checks
for security labels on a table or its columns.
 2025-04-30 18:03:52 +01:00
Onur Tirtir ea7aa6712d
Move stat view implementations into a submodule (#7975) 
Also move serialize_distributed_ddls into commands submodule, seems like
an oversight from last year (by me).
 2025-04-29 14:22:29 +03:00
Onur Tirtir d2e6cf1de0
Fix dev documentation for stat counters (#7974) 
Minor updates on the relevant portion of the tech readme and a code
comment stat_counters.c
 2025-04-29 11:35:58 +05:00
Onur Tirtir 3d61c4dc71
Add citus_stat_counters view and citus_stat_counters_reset() function to reset it (#7917) 
DESCRIPTION: Adds citus_stat_counters view that can be used to query
stat counters that Citus collects while the feature is enabled, which is
controlled by citus.enable_stat_counters. citus_stat_counters() can be
used to query the stat counters for the provided database oid and
citus_stat_counters_reset() can be used to reset them for the provided
database oid or for the current database if nothing or 0 is provided.

Today we don't persist stat counters on server shutdown. In other words,
stat counters are automatically reset in case of a server restart.

Details on the underlying design can be found in header comment of
stat_counters.c and in the technical readme.

-------

Here are the details about what we track as of this PR:

For connection management, we have three statistics about the inter-node
connections initiated by the node itself:

* **connection_establishment_succeeded**
* **connection_establishment_failed**
* **connection_reused**

While the first two are relatively easier to understand, the third one
covers the case where a connection is reused. This can happen when a
connection was already established to the desired node, Citus decided to
cache it for some time (see citus.max_cached_conns_per_worker &
citus.max_cached_connection_lifetime), and then reused it for a new
remote operation. Here are the other important details about these
connection statistics:

1. connection_establishment_failed doesn't care about the connections
that we could establish but are lost later in the transaction. Plus, we
cannot guarantee that the connections that are counted in
connection_establishment_succeeded were not lost later.
2. connection_establishment_failed doesn't care about the optional
connections (see OPTIONAL_CONNECTION flag) that we gave up establishing
because of the connection throttling rules we follow (see
citus.max_shared_pool_size & citus.local_shared_pool_size). The reaason
for this is that we didn't even try to establish these connections.
3. For the rest of the cases where a connection failed for some reason,
we always increment connection_establishment_failed even if the caller
was okay with the failure and know how to recover from it (e.g., the
adaptive executor knows how to fall back local execution when the target
node is the local node and if it cannot establish a connection to the
local node). The reason is that even if it's likely that we can still
serve the operation, we still failed to establish the connection and we
want to track this.
4. Finally, the connection failures that we count in
connection_establishment_failed might be caused by any of the following
reasons and for now we prefer to _not_ further distinguish them for
simplicity:
a. remote node is down or cannot accept any more connections, or
overloaded such that citus.node_connection_timeout is not enough to
establish a connection
b. any internal Citus error that might result in preparing a bad
connection string so that libpq fails when parsing the connection string
even before actually trying to establish a connection via connect() call
c. broken citus.node_conninfo or such Citus configuration that was
incorrectly set by the user can also result in similar outcomes as in b
d. internal waitevent set / poll errors or OOM in local node

We also track two more statistics for query execution:

* **query_execution_single_shard**
* **query_execution_multi_shard**

And more importantly, both query_execution_single_shard and
query_execution_multi_shard are not only tracked for the top-level
queries but also for the subplans etc. The reason is that for some
queries, e.g., the ones that go through recursive planning, after Citus
performs the heavy work as part of subplans, the work that needs to be
done for the top-level query becomes quite straightforward. And for such
query types, it would be deceiving if we only incremented the query stat
counters for the top-level query. Similarly, for non-pushable INSERT ..
SELECT and MERGE queries, we perform separate counter increments for the
SELECT / source part of the query besides the final INSERT / MERGE
query.
 2025-04-28 12:23:52 +00:00
ThomasC02 37e23f44b4
Add Support for CASCADE/RESTRICT in REVOKE statements (#7958) 
Fixes #7105.

DESCRIPTION: Fixes a bug that causes omitting CASCADE clause for the
commands sent to workers for REVOKE commands on tables.

---------

Co-authored-by: ThomasC02 <thomascantrell02@gmail.com>
Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
Co-authored-by: Tiago Silva <tiagos3373@gmail.com>
 2025-04-26 01:13:41 +03:00
Karina 48d89c9c1b
Adjust max_prepared_transactions only when it is default (#7712) 
DESCRIPTION: Adjusts max_prepared_transactions only when it's set to
default on PG >= 16

Fixes #7711.
Change AdjustMaxPreparedTransactions to really check if
max_prepared_transactions is explicitly set by user, and only adjust
max_prepared_transactions when it is default.
This fixes 021_twophase test failure with loaded Citus library after
postgres/postgres@b39c5272.

Co-authored-by: Karina Litskevich <litskevichkarina@gmail.com>
 2025-04-24 11:11:49 +00:00
manaldush 0e6127c4f6
AddressSanitizer: stack-use-after-scope on distributed_planner:HasUnresolvedExternParamsWalker (#7948) 
Var externParamPlaceholder is created on stack, and its address is used
for paramFetch. Postgres code return address of externParamPlaceholder
var to externParam, then code flow go out of scope and dereference
pointer on stack out of scope.

Fixes https://github.com/citusdata/citus/issues/7941.

---------

Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
 2025-04-04 13:27:56 +00:00
manaldush f084b79a4b
AddressSanitizer: stack-use-after-scope on address in CreateBackgroundJob (#7949) 
Var jobTypeName is created on stack and its value over pointer is used
in heap_form_tuple, so we
have stack use out of scope.
Issue was detected with adress sanitizer.

Fixes #7943.
 2025-04-04 13:03:41 +00:00
Cédric Villemain 1dc60e38bb
Propagates GRANT/REVOKE rights on table columns (#7918) 
This commit adds support for GRANT/REVOKE on table columns.

It extends propagated DDL according to this logic:
https://github.com/citusdata/citus/tree/main/src/backend/distributed#ddl

* Unchanged pre-existing behavior related to splitting ddl per relation
during propagation.
* Changed the way ACL are checked in some cases (see
`EnsureTablePermissions()` and associated commits)
* Rewrite `pg_get_table_grants` to include column grants as well
* Add missing `pfree()` in `pg_get_table_grants()`

Fixes https://github.com/citusdata/citus/issues/7287
Also check a box in https://github.com/citusdata/citus/issues/4812
 2025-04-04 11:54:16 +03:00
Cédric Villemain a7e686c106
Make sure to prevent INSERT INTO ... SELECT queries involving subfield or sublink (#7912) 
DESCRIPTION: Makes sure to prevent `INSERT INTO ... SELECT` queries involving subfield or sublink, to avoid crashes

The following query was crashing the backend:

```
INSERT INTO field_indirection_test_1 (
  int_col, ct1_col.int_1,ct1_col.int_2
) SELECT 0, 1, 2;
-- crash
```

En passant, added more tests with sublink in distributed_types and found
another query with wrong behavior:

```
INSERT INTO domain_indirection_test (f1,f3.if1) SELECT 0, 1;
ERROR:  could not find a conversion path from type 23 to 17619
-- not the expected ERROR
```

Fixed them by using `strip_implicit_coercions()` on target entry
expression before checking for the presence of a subscript or
fieldstore, else we fail to find the existing ones and wrongly accept to
execute unsafe query.
 2025-03-27 09:39:43 +00:00
Naisila Puka 4b4fa22b64
Fix mis-deparsing of shard query in "output-table column" name conflict (#7932) 
DESCRIPTION: Fixes a bug in deparsing of shard query in case of
"output-table column" name conflict

If an `ORDER BY` item in `SELECT` is a bare identifier, the parser
_first seeks it as an output column name_ of the `SELECT` (for SQL92
compatibility).  However, ruleutils.c is expecting the SQL99
interpretation _where such a name is an input column name_.  So it's
possible to produce an incorrect display of a view in the (admittedly
pretty ill-advised) case where some other column is renamed in the
`SELECT` output list to match an `ORDER BY` column.

The `DISTINCT ON` expressions are interpreted using the same rules as
for `ORDER BY`.
We had an issue reported that actually uses `DISTINCT ON`: #7684 
Since Citus uses ruleutils deparsing logic to create the shard queries,
it would not
table-qualify the column names as needed.

PG17 fixed this https://github.com/postgres/postgres/commit/a7eb633563c
by table-qualifying such names in the dumped view text. Therefore,
Citus doesn't reproduce the issue in PG17, since PG17 table-qualifies
the column names when needed, and the produced shard queries are
correct.

This PR applies the PG17 patch to `ruleutils_15.c` and `ruleutils_16.c`.
Even though we generally try to avoid modifying the ruleutils files, in
this case
we are applying a Postgres patch that `ruleutils_17.c` already has:

897d996b8f

Thanks @c2main for your discussion and idea in the issue.
Fixes #7684
 2025-03-19 14:21:30 +03:00
German Eichberger 1c09469dd2
Adds a method to determine if current node is primary (#7720) 
DESCRIPTION: Adds citus_is_primary_node() UDF to determine if the
current node is a primary node in the cluster.

---------

Co-authored-by: German Eichberger <geeichbe@microsoft.com>
Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
 2025-03-18 15:12:42 +00:00
naisila 1d0bdbd749 Bump Citus into 13.1devel 2025-03-13 15:13:56 +03:00
naisila caceb35eba Some cleanup from dropping pg14 2025-03-13 15:13:56 +03:00
naisila 08913e27d7 PG17 renamed Anum_pg_database_daticulocale to Anum_pg_database_datlocale 2025-03-13 15:13:56 +03:00
naisila 17b4122e84 Rename some more foreach_ptr to foreach_declared_ptr 2025-03-13 15:13:56 +03:00
Cédric Villemain ed40a0ad02 fix issue #7676: wrong handler around MULTIEXPR (#7914) 
DESCRIPTION: Fixes a bug with `UPDATE SET (...) = (SELECT
some_func(),... )` (#7676)

Citus was checking for presence of sublink, but forgot to manage
multiexpr while evaluating clauses during planning. At this stage (citus
planner), it's not always possible to call PostgreSQL code because the
tree is not yet ready for PostgreSQL pure executor.

Fixes https://github.com/citusdata/citus/issues/7676.

Fixed by adding a new function to check sublink or multiexpr in the
tree.

---------

Co-authored-by: Colm <colmmchugh@microsoft.com>
 2025-03-12 16:03:30 +03:00
Mehmet YILMAZ e50563fbd8 Issue 7887 Enhance AddInsertSelectCasts for Identity Columns (#7920) 
## Enhance `AddInsertSelectCasts` for Identity Columns


This PR fixes #7887 and improves the behavior of partial inserts into
**identity columns** by modifying the **`AddInsertSelectCasts`**
function. Specifically, we introduce **special-case handling** for
`nextval(...)` calls (represented in the parse tree as `NextValueExpr`)
to ensure that if the identity column’s declared type differs from
`nextval`’s default return type (`int8`), we **cast** the expression
properly. This prevents mismatches like `int8` → `int4` from causing
“invalid string enlargement” errors or other type-related failures.

When `INSERT ... SELECT` is processed, `AddInsertSelectCasts` reconciles
each target column’s type with the corresponding SELECT expression’s
type. Historically, for identity columns that rely on `nextval(...)`, we
can end up with a mismatch:
- `nextval` returns **`int8`**,
- The identity column might be **`int4`**, **`bigint`**, or another
integer type.

Without a correct cast, Postgres or Citus can produce plan-time or
runtime errors. By **detecting** `NextValueExpr` and applying a cast to
the column’s type, the final plan ensures consistent insertion without
errors.

## What Changed

1. **Check for `NextValueExpr`**:  
   In `AddInsertSelectCasts`, we now have a code block:
   ```c
   if (IsA(selectEntry->expr, NextValueExpr))
   {
       Oid nextvalType = GetNextvalReturnTypeCatalog();
       ...
// If (targetType != nextvalType), build a cast from int8 -> targetType
   }
   else
   {
       // fallback to generic mismatch logic
   }
   ```
This short-circuits any expression that’s a `nextval(...)` call, letting
us explicitly cast to the correct type.

2. **Fallback Generic Logic**:  
If it isn’t a `NextValueExpr` (i.e. a normal column or expression
mismatch), we still rely on the existing path that compares `sourceType`
vs. `targetType` and calls `CastExpr(...)` if they differ.

3. **`GetNextvalReturnTypeCatalog`**:  
We added or refined a helper function to confirm that `nextval` returns
`int8`, or do a `LookupFuncName("nextval", ...)` to discover the
function’s return type from `pg_proc`—making it robust if future changes
happen.

## Benefits

- **Partial inserts** into identity columns no longer fail with type
mismatches.
- When `nextval` yields `int8` but the identity column is `int4` (or
another type), we properly cast to the column’s type in the plan.
- Preserves the **existing** approach for other columns—only identity
calls get the specialized `NextValueExpr` logic.

## Testing

- Extended `generatedidentity.sql` test scenario to cover partial
inserts into both `GENERATED ALWAYS` and `GENERATED BY DEFAULT` identity
columns, including tests for the `OVERRIDING SYSTEM VALUE` clause and
partial inserts referencing foreign-key columns.
 2025-03-12 12:43:01 +03:00
Muhammad Usama 95da74c47f Fix Deadlock with transaction recovery is possible during Citus upgrades (#7910) 
DESCRIPTION: Fixes deadlock with transaction recovery that is possible
during Citus upgrades.

Fixes #7875.

This commit addresses two interrelated deadlock issues uncovered during Citus
upgrades:
1. Local Deadlock:
   - **Problem:**
     In `RecoverWorkerTransactions()`, a new connection is created for each worker
     node to perform transaction recovery by locking the
     `pg_dist_transaction` catalog table until the end of the transaction. When
     `RecoverTwoPhaseCommits()` calls this function for each worker node, the order
     of acquiring locks on `pg_dist_authinfo` and `pg_dist_transaction` can alternate.
     This reversal can lead to a deadlock if any concurrent process requires locks on
     these tables.
   - **Fix:**
     Pre-establish all worker node connections upfront so that
     `RecoverWorkerTransactions()` operates with a single, consistent connection.
     This ensures that locks on `pg_dist_authinfo` and `pg_dist_transaction` are always
     acquired in the correct order, thereby preventing the local deadlock.

2. Distributed Deadlock:
   - **Problem:**
     After resolving the local deadlock, a distributed deadlock issue emerges. The
     maintenance daemon calls `RecoverWorkerTransactions()` on each worker node—
     including the local node—which leads to a complex locking sequence:
       - A RowExclusiveLock is taken on the `pg_dist_transaction` table in
         `RecoverWorkerTransactions()`.
       - An update extension then tries to acquire an AccessExclusiveLock on the same
         table, getting blocked by the RowExclusiveLock.
       - A subsequent query (e.g., a SELECT on `pg_prepared_xacts`) issued using a
         separate connection on the local node gets blocked due to locks held during a
         call to `BuildCitusTableCacheEntry()`.
       - The maintenance daemon waits for this query, resulting in a circular wait and
         stalling the entire cluster.
   - **Fix:**
     Avoid cache lookups for internal PostgreSQL tables by implementing an early bailout
     for relation IDs below `FirstNormalObjectId` (system objects). This eliminates
     unnecessary calls to `BuildCitusTableCache`, reducing lock contention and mitigating
     the distributed deadlock.
     Furthermore, this optimization improves performance in fast
     connect→query_catalog→disconnect cycles by eliminating redundant
     cache creation and lookups.

3. Also reverts the commit that disabled the relevant test cases.
 2025-03-12 12:43:01 +03:00
Colm 4139370a1d #7782 - catch when Postgres planning removes all Citus tables (#7907) 
DESCRIPTION: fix a planning error caused by a redundant WHERE clause

Fix a Citus planning glitch that occurs in a DML query when the WHERE
clause of the query is of the form:
    ` WHERE true OR <expression with 1 or more citus tables> `
and this is the only place in the query referencing a citus table.
Postgres' standard planner transforms the WHERE clause to:
    ` WHERE true `
So the query now has no citus tables, confusing the Citus planner as
described in issues #7782 and #7783. The fix is to check, after Postgres
standard planner, if the Query has been transformed as shown, and re-run
the check of whether or not the query needs distributed planning.
 2025-03-12 12:43:01 +03:00
Mehmet YILMAZ 87ec3def55 Fix 0-Task Plans in Single-Shard Router When Updating a Local Table with Reference Table in Subquery (#7897) 
This PR fixes an issue #7891 in the Citus planner where an `UPDATE` on a
local table with a subquery referencing a reference table could produce
a 0-task plan. Historically, the planner sometimes failed to detect that
both the target and referenced tables were effectively “local,”
assigning `INVALID_SHARD_ID `and yielding a no-op plan.

### Root Cause

- In the Citus router logic (`PlanRouterQuery`), we relied on `shardId`
to determine whether a query should be routed to a single shard.
- If `shardId == INVALID_SHARD_ID`, but we also had not marked the query
as a “local table modification,” the code path would produce zero tasks.
- Local + reference tables do not require multi-shard routing. Failing
to detect this “purely local” scenario caused Citus to incorrectly route
to zero tasks.

### Changes

**Enhanced Local Table Detection**

- Updated `IsLocalTableModification` and related checks to consider both
local and reference tables as “local” for planning, preventing the
0-task scenario.
- Expanded `ContainsOnlyLocalOrReferenceTables` to return true if there
are no fully distributed tables in the query.

**Added Regress Test**

- Introduced a new regress test (`issue_7891.sql`) which reproduces the
scenario.
- Verifies we get a valid single- or local-task plan rather than a
0-task plan.
 2025-03-12 12:43:01 +03:00
Colm ec141f696a Enhance MERGE .. WHEN NOT MATCHED BY SOURCE for repartitioned source (#7900) 
DESCRIPTION: Ensure that a MERGE command on a distributed table with a
`WHEN NOT MATCHED BY SOURCE` clause runs against all shards of the
distributed table.

The Postgres MERGE command updates a table using a table or a query as a
data source. It provides three ways to match the target table with the
source: `WHEN MATCHED` means that there is a row in both the target and
source; `WHEN NOT MATCHED` means that there is a row in the source that
has no match (is not present) in the target; and, as of PG17, `WHEN NOT
MATCHED BY SOURCE` means that there is a row in the target that has no
match in the source.

In Citus, when a MERGE command updates a distributed table using a
local/reference table or a distributed query as source, that source is
repartitioned, and for each repartitioned shard that has data (i.e. 1 or
more rows) the MERGE is run against the corresponding distributed table
shard. Suppose the distributed table has 32 shards, and the source
repartitions into 4 shards that have data, with the remaining 28 shards
being empty; then the MERGE command is performed on the 4 corresponding
shards of the distributed table. However, the semantics of `WHEN NOT
MATCHED BY SOURCE` are that the specified action must be performed on
the target for each row in the target that is not in the source; so if
the source is empty, all target rows should be updated. To see this,
consider the following MERGE command:
```
MERGE INTO target AS t
USING source AS s ON t.id = s.id
WHEN NOT MATCHED BY SOURCE THEN UPDATE t SET t.col1 = 100
```
If the source has zero rows then every row in the target is updated s.t.
its col1 value is 100. Currently in Citus a MERGE on a distributed table
with a local/reference table or a distributed query as source ignores
shards of the distributed table when the corresponding shard of the
repartitioned source has zero rows. However, if the MERGE command
specifies a `WHEN NOT MATCHED BY SOURCE` clause, then the MERGE should
be performed on all shards of the distributed table, to ensure that the
specified action is performed on the target for each row in the target
that is not in the source. This PR enhances Citus MERGE execution so
that when a repartitioned source shard has zero rows, and the MERGE
command specifies a `WHEN NOT MATCHED BY SOURCE` clause, the MERGE is
performed against the corresponding shard of the distributed table using
an empty (zero row) relation as source, by generating a query of the
form:
```
MERGE INTO target_shard_0002 AS t
USING (SELECT id FROM (VALUES (NULL) ) source_0002(id) WHERE FALSE) AS s ON t.id = s.id
WHEN NOT MATCHED BY SOURCE THEN UPDATE t set t.col1 = 100
```
This works because each row in the target shard will be updated, and
`WHEN MATCHED` and `WHEN NOT MATCHED`, if specified, will be no-ops
because the source has zero rows.

To implement this when the source is a local or reference table involves
teaching function `ExcuteSourceAtCoordAndRedistribution()` in
`merge_executor.c` to not prune tasks when the query has `WHEN NOT
MATCHED BY SOURCE` but to instead replace the task's query to one that
uses an empty relation as source. And when the source is a distributed
query, function
`ExecuteMergeSourcePlanIntoColocatedIntermediateResults()` (also in
`merge_executor.c`) instead of skipping empty tasks now generates a
query that uses an empty relation as source for the corresponding target
shard of the distributed table, but again only when the query has `WHEN
NOT MATCHED BY SOURCE`. A new function `BuildEmptyResultQuery()` is
added to `recursive_planning.c` and it is used by both the
aforementioned functions in `merge_executor.c` to build an empty
relation to use as the source. It applies the appropriate type to each
column of the empty relation so the join with the target makes sense to
the query compiler.
 2025-03-12 12:43:01 +03:00
OlgaSergeyevaB ccd7ddee36 Custom Scan (ColumnarScan): exclude outer_join_rels from CandidateRelids (#7703) 
DESCRIPTION: Fixes a crash in columnar custom scan that happens when a
columnar table is used in a join. Fixes issue #7647.

Co-authored-by: Ольга Сергеева <ob-sergeeva@it-serv.ru>
 2025-03-12 12:43:01 +03:00
Colm 89674d9630 [Bug Fix] SEGV on query with Left Outer Join (#7787) (#7901) 
DESCRIPTION: Fixes a crash in left outer joins that can happen when
there is an an aggregate on a column from the inner side of the join.

Fix the SEGV seen in #7787 and #7899; it occurs because a column in the
targetlist of a worker subquery can contain a non-empty varnullingrels
field if the column is from the inner side of a left outer join. The
issue can also occur with the columns in the HAVING clause, and this is
also tested in the fix. The issue was triggered by the introduction of
the varnullingrels to Vars in Postgres 16 (2489d76c)

There is a related issue, #7705, where a non-empty varnullingrels was
incorrectly copied into the query tree for the combine query. Here, a
non-empty varnullingrels field of a var is incorrectly copied into the
query tree for a worker subquery.

The regress file from #7705 is used (and renamed) to also test this
(#7787). An alternative test output file is required for Postgres 15
because of an optimization to DISTINCT in Postgres 16 (1349d2790bf).
 2025-03-12 12:43:01 +03:00
Onur Tirtir 7004295065 Revert "Release RowExclusiveLock on pg_dist_transaction as soon as remote xacts are recovered" 
This reverts commit 684b4c6b96.
 2025-03-12 12:43:01 +03:00
Naisila Puka 3b1c082791 Drops PG14 support (#7753) 
DESCRIPTION: Drops PG14 support

1. Remove "$version_num" != 'xx' from configure file
2. delete all PG_VERSION_NUM = PG_VERSION_XX references in the code
3. Look at pg_version_compat.h file, remove all _compat functions etc
defined specifically for PGXX differences
4. delete all PG_VERSION_NUM >= PG_VERSION_(XX+1), PG_VERSION_NUM <
PG_VERSION_(XX+1) ifs in the codebase
5. delete ruleutils_xx.c file
6. cleanup normalize.sed file from pg14 specific lines
7. delete all alternative output files for that particular PG version,
server_version_ge variable helps here
 2025-03-12 12:43:01 +03:00
Onur Tirtir d5618b6b4c Release RowExclusiveLock on pg_dist_transaction as soon as remote xacts are recovered 
As of this commit, after recovering the remote transactions, now we release the lock
on pg_dist_transaction while closing it to avoid deadlocks that might occur because
of trying to acquire a lock on pg_dist_authinfo while holding a lock on
pg_dist_transaction. Such a scenario can only cause a deadlock if another transaction
is trying to acquire a strong lock on pg_dist_transaction while holding a lock on
pg_dist_authinfo. As of today, we (implicitly) acquire a strong lock on
pg_dist_transaction only when upgrading Citus to 11.3-1 and this happens when creating
a REPLICA IDENTITY on pg_dist_transaction.

And regardless of the code-path we are in, it should be okay to release the lock there
because all we do after that point is to abort the prepared transactions that are not
part of an in-progress distributed transaction and releasing the lock before doing so
should be just fine.

This also changes the blocking behavior between citus_create_restore_point and the
transaction recovery code-path in the sense that now citus_create_restore_point doesn't
until transaction recovery completes aborting the prepared transactions that are not
part of an in-progress distributed transaction. However, this should be fine because
even before this was possible, e.g., if transaction recovery fails to open a remote
connection to a node.
 2025-03-12 12:43:01 +03:00
Mehmet YILMAZ 1bb6c7e95f PG17 Compatibility - Fix crash when pg_class is used in MERGE (#7853) 
This pull request addresses Issue #7846, where specific MERGE queries on
non-distributed and distributed tables can result in crashes in certain
scenarios. The issue stems from the usage of `pg_class` catalog table,
and the `FilterShardsFromPgclass` function in Citus. This function goes
through the query's jointree to hide the shards. However, in PG17,
MERGE's join quals are in a separate structure called
`mergeJoinCondition`. Therefore FilterShardsFromPgclass was not
filtering correctly in a `MERGE` command that involves `pg_class`. To
fix the issue, we handle `mergeJoinCondition` separately in PG17.

Relevant PG commit:

0294df2f1f

**Non-Distributed Tables:**
A MERGE query involving a non-distributed table using
`pg_catalog.pg_class` as the source may execute successfully but needs
testing to ensure stability.

**Distributed Tables:**
Performing a MERGE on a distributed table using `pg_catalog.pg_class` as
the source raises an error:
`ERROR: MERGE INTO a distributed table from Postgres table is not yet
supported`
However, in some cases, this can lead to a server crash if the
unsupported operation is not properly handled.

This is the test output from the same test conducted prior to the code
changes being implemented.

```
-- Issue #7846: Test crash scenarios with MERGE on non-distributed and distributed tables
-- Step 1: Connect to a worker node to verify shard visibility
\c postgresql://postgres@localhost::worker_1_port/regression?application_name=psql
SET search_path TO pg17;
-- Step 2: Create and test a non-distributed table
CREATE TABLE non_dist_table_12345 (id INTEGER);
-- Test MERGE on the non-distributed table
MERGE INTO non_dist_table_12345 AS target_0
USING pg_catalog.pg_class AS ref_0
ON target_0.id = ref_0.relpages
WHEN NOT MATCHED THEN DO NOTHING;
SSL SYSCALL error: EOF detected
connection to server was lost
```
 2025-03-12 12:43:01 +03:00
Naisila Puka 0642a4dc08 Propagate MERGE ... WHEN NOT MATCHED BY SOURCE (#7807) 
DESCRIPTION: Propagates MERGE ... WHEN NOT MATCHED BY SOURCE

It seems like there is not much needed to be done here.
`get_merge_query_def` from `ruleutils_17` is updated with "WHEN NOT
MATCHED BY SOURCE" therefore `deparse_shard_query` parses the merge
query for execution on the shard correctly.

Relevant PG commit:
https://github.com/postgres/postgres/commit/0294df2f1
 2025-03-12 12:43:00 +03:00
Naisila Puka 74d945f5ae PG17 - Propagate EXPLAIN options: MEMORY and SERIALIZE (#7802) 
DESCRIPTION: Propagates MEMORY and SERIALIZE options of EXPLAIN

The options for `MEMORY` can be true or false. Default is false.
The options for `SERIALIZE` can be none, text or binary. Default is
none.

I referred to how we added support for WAL option in this PR [Support
EXPLAIN(ANALYZE, WAL)](https://github.com/citusdata/citus/pull/4196).
For the tests however, I used the same tests as Postgres, not like the
tests in the WAL PR. I used exactly the same tests as Postgres does, I
simply distributed the table beforehand. See below the relevant Postgres
commits from where you can see the tests added as well:
- [Add EXPLAIN
(MEMORY)](https://github.com/postgres/postgres/commit/5de890e36)
- [Invent SERIALIZE option for
EXPLAIN.](https://github.com/postgres/postgres/commit/06286709e)

This PR required a lot of copying of Postgres static functions regarding
how `EXPLAIN` works for `MEMORY` and `SERIALIZE` options. Specifically,
these copy-pastes were required for updating `ExplainWorkerPlan()`
function, which is in fact based on postgres' `ExplainOnePlan()`:
```C
/* copied from explain.c to update ExplainWorkerPlan() in citus according to ExplainOnePlan() in postgres */
#define BYTES_TO_KILOBYTES(b)
typedef struct SerializeMetrics
static bool peek_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_buffer_usage(ExplainState *es, const BufferUsage *usage);
static void show_memory_counters(ExplainState *es, const MemoryContextCounters *mem_counters);
static void ExplainIndentText(ExplainState *es);
static void ExplainPrintSerialize(ExplainState *es, SerializeMetrics *metrics);
static SerializeMetrics GetSerializationMetrics(DestReceiver *dest);
```

_Note_: it looks like we were missing some `buffers` option details as
well. I put them together with the memory option, like the code in
Postgres explain.c, as I didn't want to change the copied code. However,
I tested locally and there is no big deal in previous Citus versions,
and you can also see that existing Citus tests with `buffers true`
didn't change. Therefore, I prefer not to backport "buffers" changes to
previous versions.
 2025-03-12 12:43:00 +03:00
Naisila Puka 8940665d17 Allow configuring sslnegotiation using citus.node_conn_info (#7821) 
Relevant PG commit:
https://github.com/postgres/postgres/commit/d39a49c1e

PR similar to https://github.com/citusdata/citus/pull/5203
 2025-03-12 12:26:06 +03:00
Naisila Puka 658632642a Disallow infinite values for partition interval in create_time_partitions udf (#7822) 
PG17 added +/- infinity values for the interval data type
Relevant PG commit:
https://github.com/postgres/postgres/commit/519fc1bd9
 2025-03-12 12:26:06 +03:00
Naisila Puka 3e96a19606 Adds JSON_TABLE() support, and SQL/JSON constructor/query functions tests (#7816) 
DESCRIPTION: Adds JSON_TABLE() support

PG17 has added basic `JSON_TABLE()` functionality
`JSON_TABLE()` allows `JSON` data to be converted into a relational view
and thus used, for example, in a `FROM` clause, like other tabular data.

We treat `JSON_TABLE` the same as correlated functions (e.g., recurring
tuples). In the end, for multi-shard `JSON_TABLE` commands, we apply the
same restrictions as reference tables (e.g., cannot perform a lateral
outer join when a distributed subquery references a (reference
table)/(json table) etc.)

Relevant PG17 commits:
[basic JSON
table](https://github.com/postgres/postgres/commit/de3600452), [nested
paths in json
table](https://github.com/postgres/postgres/commit/bb766cde6)

Onder had previously added json table support for PG15BETA1, but we
reverted that commit because json table was reverted in PG15.
ce7f1a530f
Previous relevant PG15Beta1 commit:
https://github.com/postgres/postgres/commit/4e34747c8
Therefore, I referred to Onder's commit for this commit as well, with a
few changes due to some differences between PG15/PG17:

1) In PG15Beta1, we had also `PLAN` clauses for `JSON_TABLE`
https://github.com/postgres/postgres/commit/fadb48b00, and Onder's
commit includes tests for those as well. However, `PLAN` nodes are _not_
added in PG17. Therefore, I didn't include the `json_table_select_only`
test, which had mostly queries involving `PLAN`. I only included the
last query from that test.

2) In PG15 timeline (Citus 11.1), we didn't support outer joins where
the outer rel is a recurring one and the inner one is a non-recurring
one. However, [Onur added support for that one in Citus
11.2](https://github.com/citusdata/citus/pull/6512), therefore I updated
the tests from Onder's commit accordingly.

3) PG17 json table has nested paths and columns, therefore I added a
test
with a distributed table, which is exactly the same as the one in
sqljson_jsontable in PG17.
https://github.com/postgres/postgres/commit/bb766cde6

This pull request also adds some basic tests on validation of SQL/JSON
constructor functions JSON(), JSON_SCALAR(), and JSON_SERIALIZE(),
and also SQL/JSON query functions JSON_EXISTS(), JSON_QUERY(), and
JSON_VALUE(). The relevant PG commits are the following:
[JSON(), JSON_SCALAR(),
JSON_SERIALIZE()](https://github.com/postgres/postgres/commit/03734a7fe)
[JSON_EXISTS(), JSON_VALUE(),
JSON_QUERY()](https://github.com/postgres/postgres/commit/6185c9737)
 2025-03-12 12:26:05 +03:00
Mehmet YILMAZ 1cf5c190aa Error out for ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION (#7814) 
PG17 added support for
ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION.
Relevant PG commit: https://github.com/postgres/postgres/commit/5d06e99a3

We currently don't support propagating this command for Citus tables.
It is added to future work.

This PR disallows `ALTER TABLE ... ALTER COLUMN ... SET EXPRESSION` on
all Citus table types (local, distributed, and partitioned distributed)
by adding an error check in `ErrorIfUnsupportedAlterTableStmt`. A new
regression test verifies that each table type fails with a consistent
error message when attempting to set an expression.
 2025-03-12 12:25:49 +03:00
Mehmet YILMAZ 24585a8c04 Error out for ALTER TABLE ... SET ACCESS METHOD DEFAULT (#7803) 
PG17 introduced ALTER TABLE ... SET ACCESS METHOD DEFAULT

This PR introduces and enforces an error check preventing ALTER TABLE
... SET ACCESS METHOD DEFAULT on both Citus local tables (added via
citus_add_local_table_to_metadata) and distributed/partitioned
distributed tables. The regression tests now demonstrate that each table
type raises an error advising users to explicitly specify an access
method, rather than relying on DEFAULT. This ensures consistent behavior
across local and distributed environments in Citus.

The reason why we currently don't support this is that we can't simply
propagate the command as it is, because the default table access method
may be different across Citus cluster nodes.

Relevant PG commit:
https://github.com/postgres/postgres/commit/d61a6cad6
 2025-03-12 12:25:49 +03:00
Naisila Puka 5e9f8d838c Error for COPY FROM ... on_error, log_verbosity with Citus tables (#7811) 
PG17 added the new ON_ERROR option for COPY FROM. When this option is
specified, COPY skips soft errors and
continues copying.
Relevant PG commits:
-- https://github.com/postgres/postgres/commit/9e2d87011
-- https://github.com/postgres/postgres/commit/b725b7eec

I tried it locally with Citus tables.
Without further implementation, it doesn't work correctly.
Therefore, we error out for now, and add it to future work.

PG17 also added log_verbosity option, which controls the
 amount of messages emitted during processing. This is
 currently used in COPY FROM when ON_ERROR option is set to
 ignore. Therefore, we error out for this option as well.
Relevant PG17 commit:
https://github.com/postgres/postgres/commit/f5a227895
 2025-03-12 12:25:49 +03:00
Naisila Puka a383ef6831 Adds PG17.1 support - Regression tests sanity (#7661) 
This is the final commit that adds
PG17 compatibility with Citus's current capabilities.

You can use Citus community, release-13.0 branch, with PG17.1.

---------

Specifically, this commit:

- Enables PG17 in the configure script.

- Adds PG17 tests to CI using test images that have 17.1

- Fixes an upgrade test: see below for details
In `citus_prepare_upgrade()`, don't drop any_value when upgrading from
PG16+, because PG16+ has its own any_value function. Attempting to do so
results in the error seen in [pg16-pg17
upgrade](https://github.com/citusdata/citus/actions/runs/11768444117/job/32778340003?pr=7661):
```
ERROR:  cannot drop function any_value(anyelement) because it is required by the database system
CONTEXT:  SQL statement "DROP AGGREGATE IF EXISTS pg_catalog.any_value(anyelement)"
```
When 16 becomes the minimum supported Postgres version, the drop
statements can be removed.

---------

Several PG17 Compatibility commits have been merged before this final one.
All these subtasks are done https://github.com/citusdata/citus/issues/7653

See the list below:

Compilation PR: https://github.com/citusdata/citus/pull/7699
Ruleutils PR: https://github.com/citusdata/citus/pull/7725
Sister PR for tests: https://github.com/citusdata/the-process/pull/159

Helpful smaller PRs:
- https://github.com/citusdata/citus/pull/7714
- https://github.com/citusdata/citus/pull/7726
- https://github.com/citusdata/citus/pull/7731
- https://github.com/citusdata/citus/pull/7732
- https://github.com/citusdata/citus/pull/7733
- https://github.com/citusdata/citus/pull/7738
- https://github.com/citusdata/citus/pull/7745
- https://github.com/citusdata/citus/pull/7747
- https://github.com/citusdata/citus/pull/7748
- https://github.com/citusdata/citus/pull/7749
- https://github.com/citusdata/citus/pull/7752
- https://github.com/citusdata/citus/pull/7755
- https://github.com/citusdata/citus/pull/7757
- https://github.com/citusdata/citus/pull/7759
- https://github.com/citusdata/citus/pull/7760
- https://github.com/citusdata/citus/pull/7761
- https://github.com/citusdata/citus/pull/7762
- https://github.com/citusdata/citus/pull/7765
- https://github.com/citusdata/citus/pull/7766
- https://github.com/citusdata/citus/pull/7768
- https://github.com/citusdata/citus/pull/7769
- https://github.com/citusdata/citus/pull/7771
- https://github.com/citusdata/citus/pull/7774
- https://github.com/citusdata/citus/pull/7776
- https://github.com/citusdata/citus/pull/7780
- https://github.com/citusdata/citus/pull/7781
- https://github.com/citusdata/citus/pull/7785
- https://github.com/citusdata/citus/pull/7788
- https://github.com/citusdata/citus/pull/7793
- https://github.com/citusdata/citus/pull/7796

---------

Co-authored-by: Colm <colmmchugh@microsoft.com>
 2025-03-12 12:25:49 +03:00
Naisila Puka 28b0b0e7a8 Bump Citus version into 13.0.0 (#7792) 
We are using `release-13.0` branch for both development and release, to
deliver PG17 support in Citus.

Afterwards, we will (probably) merge this branch into main.

Some potential changes for main branch, after we are done working on
release-13.0:
- Merge changes from `release-13.0` to `main`
- Figure out what changes were there on 12.2, move them to 13.1 version.
In a nutshell: rename `12.1--12.2` to `13.0--13.1` and fix issues.
- Set version to 13.1devel
 2025-03-12 12:25:49 +03:00