Relevant PG commits:
Introduce SYSTEM_USER
0823d061b0
0823d061b0b7f1e20fbfd48bef3c2e093493dbd4
Rename shadowed local variables
2d0bbedda7
2d0bbedda7c4fbf7092c39b0c06b56cb238e15d7
Fix EXPLAIN of SEARCH BREADTH FIRST with a constant initial value
d57534740b
d57534740b7682c2816ce6c54318521787fcfacf
Add repalloc0 and repalloc0_array
b4b7ce8061
b4b7ce8061d34cea2b4915c41403b2a74d5fde0e
Improve ruleutils' printout of LATERAL references within subplans
adaf34241a
adaf34241acd83afaa45a8b614b6484a285da847
Switch SQLValueFunction on "name" to use COERCE_SQL_SYNTAX
fb32748e32
fb32748e32e2b6b2fcb32220980b93d5436f855e
Replace SQLValueFunction by COERCE_SQL_SYNTAX
f193883fc9
f193883fc9cebe8fa20359b0797832837a788112
Fix under-parenthesized display of AT TIME ZONE constructs
1dd6700f44
1dd6700f44d2e14616d464a5f490552b7036ad3f
Update copyright for 2023
c8e1ba736b
c8e1ba736b2b9e8c98d37a5b77c4ed31baf94147
Print the correct aliases for DML target tables in ruleutils
393430f575
393430f57544dfd550135e0191cc91139926b682
SQL/JSON: add standard JSON constructor functions
7081ac46ac
7081ac46ace8c459966174400b53418683c9fe5c
Fix inconsistencies and style issues in new SQL/JSON code
60966f56c3
60966f56c3e4db9f9bdff485371ef3a50d53431f
SQL/JSON: support the IS JSON predicate
6ee30209a6
6ee30209a6f161d0a267a33f090c70c579c87c00
Add ruleutils support for decompiling MERGE commands.
41e2c52fd6
41e2c52fd6ebed6eff4184f68048813cc8886ec1
Add back SQLValueFunction for SQL keywords
d8c3106bb6
d8c3106bb60e4f87be595f241e173ba3c2b7aa2c
Pre-beta mechanical code beautification
0245f8db36
0245f8db36f375326c2bae0c3420d3c77714e72d
Tradionally our planner works in the following order:
router - > pushdown -> repartition -> pull to coordinator
However, for INSERT .. SELECT commands, we did not support "router".
In practice, that is not a big issue, because pushdown planning
can handle router case as well.
However, with PG 16, certain outer joins are converted to
JOIN without any conditions (e.g., JOIN .. ON (true)) and
the filters are pushed down to the tables.
When the filters are pushed down to the tables, router planner
can detect. However, pushdown planner relies on JOIN conditions.
An example query:
```
INSERT INTO agg_events (user_id)
SELECT raw_events_first.user_id
FROM raw_events_first LEFT JOIN raw_events_second
ON raw_events_first.user_id = raw_events_second.user_id
WHERE raw_events_first.user_id = 10;
```
As a side effect of this change, now we can also relax
certain limitation that "pushdown" planner emposes, but not
"router". So, with this PR, we also allow those.
Previously, we only checked whether the relations are colocated, but we
ignore the shard indexes. That causes certain queries still to be
accidentally router. We should enforce colocation checks for both shard
index and table colocation id to make the check restrictive enough.
For example, the following query should not be router, and after this
patch, it won't:
```SQL
SELECT
user_id
FROM
((SELECT user_id FROM raw_events_first WHERE user_id = 15) EXCEPT
(SELECT user_id FROM raw_events_second where user_id = 17)) as foo;
```
DESCRIPTION: Enforce shard level colocation with
citus.enable_non_colocated_router_query_pushdown
DESCRIPTION: PR description that will go into the change log, up to 78
characters
There are 4 errors arised recently and I fixed them in this PR. Problems
and fixes are as below:
1. When executing make step in packaging pipeline, if it gets error, we
can not detect it since there are additional operations after make in
one line.
With this fix, now if an error occured after make execution, we can
detect and see the step red and failed here,
2. Recently we started to get the error ` fatal: detected dubious
ownership in repository at '/__w/citus/citus' ` as below
https://github.com/citusdata/citus/actions/runs/5542692968/jobs/10117706723#step:7:9
There is a fix for that one as well.
3. fixed the requirements issue arised related to urllib3 library
version
4. Getting errors with centos-8 docker image with the new postgres-dev
packages. Now, changed centos-8 image with almalinux-8 and now it works
This PR provides successful compilation against PG16Beta2. It does some
necessary refactoring to prepare for full support of version 16, in
https://github.com/citusdata/citus/pull/6952 .
Change RelFileNode to RelFileNumber or RelFileLocator
Relevant PG commit
b0a55e43299c4ea2a9a8c757f9c26352407d0ccc
new header for varatt.h
Relevant PG commit:
d952373a987bad331c0e499463159dd142ced1ef
drop support for Abs, use fabs
Relevant PG commit
357cfefb09115292cfb98d504199e6df8201c957
tuplesort PGcommit: d37aa3d35832afde94e100c4d2a9618b3eb76472
Relevant PG commit:
d37aa3d35832afde94e100c4d2a9618b3eb76472
Fix vacuum in columnar
Relevant PG commit:
4ce3afb82ecfbf64d4f6247e725004e1da30f47c
older one:
b6074846cebc33d752f1d9a66e5a9932f21ad177
Add alloc_flags to pg_clean_ascii
Relevant PG commit:
45b1a67a0fcb3f1588df596431871de4c93cb76f
Merge GetNumConfigOptions() into get_guc_variables()
Relevant PG commit:
3057465acfbea2f3dd7a914a1478064022c6eecd
Minor PG refactor PG_FUNCNAME_MACRO __func__
Relevant PG commit
320f92b744b44f961e5d56f5f21de003e8027a7f
Pass NULL context to stringToQualifiedNameList, typeStringToTypeName
The pre-PG16 error behaviour for the following
stringToQualifiedNameList & typeStringToTypeName
was ereport(ERROR, ...)
Now with PG16 we have this context input. We preserve the same behaviour
by passing a NULL context, because of the following:
(copy paste comment from PG16)
If "context" isn't an ErrorSaveContext node, this behaves as
errstart(ERROR, domain), and the errsave() macro ends up acting
exactly like ereport(ERROR, ...).
Relevant PG commit
858e776c84f48841e7e16fba7b690b76e54f3675
Use RangeVarCallbackMaintainsTable instead of RangeVarCallbackOwnsTable
Relevant PG commit:
60684dd834a222fefedd49b19d1f0a6189c1632e
FIX THIS: Not implemented grant-level control of role inheritance
see PG commit
e3ce2de09d814f8770b2e3b3c152b7671bcdb83f
Make Scan node abstract
PG commit:
8c73c11a0d39049de2c1f400d8765a0eb21f5228
Change in Var representations, get_relids_in_jointree
PG commit
2489d76c4906f4461a364ca8ad7e0751ead8aa0d
Deadlock detection changes because SHM_QUEUE is removed
Relevant PG Commit:
d137cb52cb7fd44a3f24f3c750fbf7924a4e9532
TU_UpdateIndexes
Relevant PG commit
19d8e2308bc51ec4ab993ce90077342c915dd116
Use object_ownercheck and object_aclcheck functions
Relevant PG commits:
afbfc02983f86c4d71825efa6befd547fe81a926
c727f511bd7bf3c58063737bcf7a8f331346f253
Rework Permission Info for successful compilation
Relevant PG commits:
postgres/postgres@a61b1f7postgres/postgres@b803b7d
---------
Co-authored-by: onderkalaci <onderkalaci@gmail.com>
Index scans in PG16 return empty sets because of extra compatibility
enforcement for `ScanKeyInit` arguments.
Could be one of the relevant PG commits:
c8b2ef05f4
This PR fixes all incompatible `RegProcedure` and `Datum` arguments in
all `ScanKeyInit` functions used throughout the codebase.
Helpful for https://github.com/citusdata/citus/pull/6952
The link in our readme directly goes to our channel, meaning people
finding the link here for the first time are unable to join slack this
way.
Given that the target audience using this link is most likely not part
of the slack channel yet it would be better to link to our self serve
signup flow at slack.citusdata.com, which is the same we use on
citusdata.com.
From simple testing you should still get redirected to the channel if
you are already joined and signed in.
This PR fixes the following:
- in oraclelinux-7 `Make` step
```
/usr/bin/ld: utils/replication_origin_session_utils.o: relocation R_X86_64_PC32 against undefined symbol
`IsLocalReplicationOriginSessionActive' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
```
`IsLocalReplicationOriginSessionActive` function has improper inline
declaration, fixed that
- in centos-7 `Make` step
```
utils/background_jobs.c: In function 'StartCitusBackgroundTaskExecutor':
utils/background_jobs.c:1746:6: warning: function might be possible candidate for 'gnu_printf' format attribute
[-Wsuggest-attribute=format]
database, user, jobId, taskId);
^
```
should use `pg_attribute_printf(3,4)` instead of
`pg_attribute_printf(3,0)` since the number of arguments varies for
`SafeSnprintf(char *str, rsize_t count, const char *fmt, ...)`
---------
Co-authored-by: naisila <nicypp@gmail.com>
Some clients send ALTER TABLE .. ADD COLUMN .. commands together
with some other DDLs and this makes it impossible to directly send
the original DDL command to the workers.
For this reason, this commit adds support for deparsing such ALTER
TABLE commands so that we can avoid from directly sending the original
one to the workers.
Partially fixes https://github.com/citusdata/citus/issues/690.
Fixes#3678
We allow materialized view to exist in distrbuted schema but they should
not be tried to be converted to a tenant table since they cannot be
distributed.
Fixes https://github.com/citusdata/citus/issues/7041
Inserting into `pg_dist_schema` causes unexpected duplicate key errors,
for distributed schemas that already exist. With this commit we skip the
insertion if the schema already exists in `pg_dist_schema`.
The error:
```sql
SET citus.enable_schema_based_sharding TO ON;
CREATE SCHEMA sc2;
CREATE SCHEMA IF NOT EXISTS sc2;
NOTICE: schema "sc2" already exists, skipping
ERROR: duplicate key value violates unique constraint "pg_dist_schema_pkey"
DETAIL: Key (schemaid)=(17294) already exists.
```
fixes: #7042
This PR
* Addresses a concurrency issue in the probabilistic approach of tenant
monitoring by acquiring a shared lock for tenant existence checks.
* Changes `citus.stat_tenants_sample_rate_for_new_tenants` type to
double
* Renames `citus.stat_tenants_sample_rate_for_new_tenants` to
`citus.stat_tenants_untracked_sample_rate`
DESCRIPTION: Change default rebalance strategy to by_disk_size
When introducing rebalancing by disk size we didn't make it the default
initially. The main reason was, because we expected some problems with
it. We have indeed had some problems/bugs with it over the years, and
have fixed all of them. By now we're quite confident in its stability,
and that it pretty much always gives better results than by_shard_count.
So this PR makes by_disk_size the new default. We don't change the
default when some other strategy than by_shard_count is the current
default. This is in case someone defined their own rebalance strategy
and marked this as the default themselves.
Note: It explicitly does nothing during a downgrade, because there's no
way of knowing if the rebalance strategy before the upgrade was
by_disk_size or by_shard_count. And even in previous versions
by_disk_size is considered superior for quite some time.
One problem with rebalancing by disk size is that shards in newly
created collocation groups are considered extremely small. This can
easily result in bad balances if there are some other collocation groups
that do have some data. One extremely bad example of this is:
1. You have 2 workers
2. Both contain about 100GB of data, but there's a 70MB difference.
3. You create 100 new distributed schemas with a few empty tables in
them
4. You run the rebalancer
5. Now all new distributed schemas are placed on the node with that had
70MB less.
6. You start loading some data in these shards and quickly the balance
is completely off
To address this edge case, this PR changes the by_disk_size rebalance
strategy to add a a base size of 100MB to the actual size of each
shard group. This can still result in a bad balance when shard groups
are empty, but it solves some of the worst cases.
We did not properly handle the error at ownership check method, which
causes `max stack depth for errors` as in
https://github.com/citusdata/citus/issues/6980.
**Fix:**
In case of an error, we should rollback subtransaction and throw the
message with log level to `LOG_SERVER_ONLY`.
Note: We prevent logs from the client to prevent pg vanilla test
failures due to Citus logs which differs from the actual Postgres logs.
(For context: https://github.com/citusdata/citus/pull/6130)
I also needed to fix a flaky test: `multi_schema_support`
DESCRIPTION: Fixes a bug related to non-existent objects in DDL
commands.
Fixes https://github.com/citusdata/citus/issues/6980
This commit is the second and last phase of dropping PG13 support.
It consists of the following:
- Removes all PG_VERSION_13 & PG_VERSION_14 from codepaths
- Removes pg_version_compat entries and columnar_version_compat entries
specific for PG13
- Removes alternative pg13 test outputs
- Removes PG13 normalize lines and fix the test outputs based on that
It is a continuation of 5bf163a27d
onderkalaci
naisila
Carol Smith
zhjwpku
ahmet gedemenli
Nils Dijk
Gürkan İndibay
Naisila Puka
Teja Mupparti
mulander
aykut-bozkurt
Halil Ozan Akgül
Onur Tirtir
Gokhan Gulbiz
Jelte Fennema