Commit Graph

3950 Commits (users/saawasek/non_blocking_split_integrated)

Author SHA1 Message Date
Onder Kalaci 0a5112964d Call relation access hash clean-up irrespective of remote transaction state
Mainly because local-only transactions should be cleaned up
2022-07-28 11:27:59 +02:00
Onder Kalaci d67cf907a2 Detach relation access tracking from connection management 2022-07-28 11:27:59 +02:00
Ying Xu fdf090758b
Bugfix for IN clause to be considered during planner phase in Columnar (#6030)
Reported bug #5803 shows that we are currently not sending the IN clause to our planner for columnar. This PR fixes it by checking for ScalarArrayOpExpr in ExtractPushdownClause so that we do not skip it. Also added a test case for this new addition.
2022-07-27 11:06:49 -07:00
Jelte Fennema 0f50bef696
Avoid possible information leakage about existing users (#6090) 2022-07-27 17:46:32 +02:00
Ahmet Gedemenli 2b2a529653
Error out for views with circular dependencies (#6051)
Adds error check for views with circular dependencies
2022-07-27 17:57:45 +03:00
aykut-bozkurt b08e5ec29d
added some missing object address callbacks (#6056) 2022-07-27 17:36:04 +03:00
Naisila Puka 1259d83511
Smallfix in CreateCollationDDL logic (#6089) 2022-07-27 14:33:31 +03:00
Onder Kalaci 5bc8a81aa7 Add colocation checks for shard splits 2022-07-27 10:01:19 +02:00
Onder Kalaci 12fa3aaf6b Concurrent shard move/copy and colocated table creation fix
It turns out that create_distributed_table
and citus_move/copy_shard_placement does not
work well concurrently.

To fix that, we need to acquire a lock, which
sounds like a good use of colocation lock.

However, the current usage of colocation lock is
limited to higher level UDFs like rebalance_table_shards
etc. Those usage of lock is still useful, but
we cannot acquire the same lock on citus_move_shard_placement
etc. because the coordinator connects to itself to acquire
the lock. Hence, the high level UDF blocks itself.

To fix that, we use one more colocation lock, with the placements
are the main objects to consider.
2022-07-27 10:01:19 +02:00
Onder Kalaci f076e81166 Do not cache all the metadata during fix_all_partition_shard_index_names 2022-07-27 09:49:08 +02:00
Onder Kalaci 26fdcb68f0 Optimize StringJoin() for when prefix-postfix is needed
Before this commit, we required multiple copies of the
same stringInfo if we needed to append/prepend data to
the stringInfo. Now, we optionally get prefix/postfix.

For large string operations, this can save up to %10
memory.
2022-07-27 09:49:08 +02:00
Onder Kalaci b8008999dc Reduce memory consumption while adjust partition index names
Previously, CreateFixPartitionShardIndexNames() created all
the relevant query strings for all the shards, and executed
the large query string. And, in terms of the memory consumption,
this huge command (and its ExprContext generated while running
the command) is the main bottleneck/

With this change, we are reducing the total amount of memory
usage to almost 1/shard_count.

On my local machine, a distributed partitioned table with 120 partitions,
each 32 shards, the total memory consumption reduced from ~3GB
to ~0.1GB. And, the total execution time increased from ~28 seconds
to ~30 seconds. This seems like a good trade-off.
2022-07-27 09:49:08 +02:00
aykut-bozkurt 5f27445b69
enable propagation warnings before postgres vanilla tests (#6081) 2022-07-27 10:34:41 +03:00
Onder Kalaci 6c65d29924 Check the PGPROC's validity properly
We used to only check whether the PID is valid
or not. However, Postgres does not necessarily
set the PID of the backend to 0 when it exists.

Instead, we need to be able to check it from procArray.
IsBackendPid() is what pg_stat_activity also relies
on for a similar purpose.
2022-07-26 17:44:44 +02:00
Hanefi Onaldi f944f97d01
Normalize messages from different libpq versions
Historically we have been testing with the 'latest' version of libpq
when the CI images were build. This has the downside that rebuilding the
images often break our tests due to different errors returned from
libpq.

With this change we will actually test with a stable version of libpq
that is based on the postgres minor version that we test against.

This will make it easier to maintain postgres images over time, as well
as running _all_ tests locally, where we change libpq in sync with the
postgres server version.
2022-07-26 01:41:34 +03:00
aykut-bozkurt 67ac3da2b0
added citus_depended_objects udf and HideCitusDependentObjects GUC to hide citus depended objects from pg meta queries (#6055)
use RecurseObjectDependencies api to find if an object is citus depended

make vanilla tests runnable to see if citus_depended function is working correctly
2022-07-25 16:43:34 +03:00
Sameer Awasekar aef758a4a3 Add testcase 2022-07-24 21:02:33 +05:30
Sameer Awasekar 474fe7be1f Reindent 2022-07-24 17:19:45 +05:30
Sameer Awasekar 0f72d7f706 Rename connection params 2022-07-24 16:34:16 +05:30
Sameer Awasekar f6c11bf09c Reindent 2022-07-24 15:16:44 +05:30
Sameer Awasekar 9073564900 Adding comments to the workflow 2022-07-24 14:16:37 +05:30
Sameer Awasekar 0a6875de5a Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-24 09:19:50 +05:30
Sameer Awasekar b4f8226f96 Add comments for the functions 2022-07-23 22:58:00 +05:30
Sameer Awasekar 5a5505db55 Add comments for functions 2022-07-23 15:47:14 +05:30
Sameer Awasekar 8c871bcd10 Adding comments and code for cleanup 2022-07-22 20:10:18 +05:30
Marco Slot 5fabf94e39 Allow WITH HOLD cursors with parameters 2022-07-21 12:00:59 +02:00
Sameer Awasekar 9e88c14096 Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-21 11:19:27 +05:30
Sameer Awasekar 515f1e2346 Refactor 2022-07-21 10:42:04 +05:30
Hanefi Onaldi eb3e5ee227 Introduce citus_locks view
citus_locks combines the pg_locks views from all nodes and adds
global_pid, nodeid, and relation_name. The columns of citus_locks don't
change based on the Postgres version, however the pg_locks's columns do.
Postgres 14 added one more column to pg_locks (waitstart timestamptz).
citus_locks has the most expansive column set, including the newly added
column. If citus_locks is queried in a Postgres version where pg_locks
doesn't have some columns, the values for those columns in citus_locks
will be NULL
2022-07-21 03:06:57 +03:00
Nitish Upreti 3d569cc49a
Shard Split support for Columnar and Partitioned Table (#6067)
DESCRIPTION:
This PR extends support for Partitioned and Columnar tables in blocking 'citus_split_shard_by_split_points' workflow.
Columnar Support : No special handling required. Just removing checks that fails split for columnar table and adding test coverage.
Partitioned Table Support :

Skip copying of parent table as they are empty, The partitions contain data and are treated as co-located shards that will be copied separately.
Attach partitions to parent on destination after inserting new shard metadata and before creating foreign key constraints.
MISC:
Fix Bug #4949 where Blocking shard moves fails if there is a foreign key between partitioned distributed tables (from child to parent).

TEST:
Added new test 'citus_split_shards_columnar_partitioned' for splitting 'partitioned' and 'columnar + partitioned' table.
Added new test 'shard_move_constraints_blocking' to add coverage for shard move bug fix.
Updated test 'citus_split_shard_by_split_points_negative' to allow columnar and partitioned table.
2022-07-20 12:24:50 -07:00
Naisila Puka 7d6410c838
Drop postgres 12 support (#6040)
* Remove if conditions with PG_VERSION_NUM < 13

* Remove server_above_twelve(&eleven) checks from tests

* Fix tests

* Remove pg12 and pg11 alternative test output files

* Remove pg12 specific normalization rules

* Some more if conditions in the code

* Change RemoteCollationIdExpression and some pg12/pg13 comments

* Remove some more normalization rules
2022-07-20 17:49:36 +03:00
aykutbozkurt 108ca875ad fix assertion bugs related to list length 2022-07-20 10:53:12 +03:00
Hanefi Onaldi 6a32061c08
Renames configure.in to fix warnings (#6034)
When building packages on ubuntu jammy, we started to see some warnings.

autoreconf: warning: autoconf input should be named 'configure.ac', not
'configure.in'
2022-07-19 18:24:15 +02:00
Sameer Awasekar a4bf6a1dbb Change version of worker_split_shard_replication_setup UDF 2022-07-19 21:02:35 +05:30
aykutbozkurt ebb6d1c8c0 refactor code where GetObjectAddressFromParseTree is called because it returns list of addresses now 2022-07-19 18:13:12 +03:00
aykutbozkurt 9d232d7b00 change address method to return list of addresses 2022-07-19 18:13:11 +03:00
Önder Kalacı 90b1afe31e
Merge branch 'main' into baby_step_pg_15 2022-07-18 15:02:39 +02:00
Nitish Upreti 5b3537cdff
Shard Split for Citus (#6029)
* Blocking split setup

* Add missing type

* Missing API from Metadata Sync

* Shard Split e2e code

* Worker Split Copy DestReceiver skeleton

* Basic destreceiver code

* worker_split_copy UDF

* UDF calling

* Split points are text

* Isolate Tenant and Split Shard Unification

* Fixing executor and misc

* Reindent code

* Fixing UDF definitions

* Hello World Local Copy works

* Remote copy hello world works

* Local and Remote binary test

* Fixing text local copy and adding tests

* Hello World shard split works

* Negative tests

* Blocking Split workflow works

* Refactor

* Bug fix

* Reindent

* Cleaning up and adding comments

* Basic test for shard split workflow

* ReIndent

* Circle CI integration

* Removing include causing circle-ci build failure

* Remove SplitCopyDestReceiver and use PartitionedResultDestReceiver

* Add support for citus.enable_binary_protocol

* Reindent

* Fix build break

* Update Test

* Cleanup on catch

* Addressing open comments

* Update downgrade script and quote schema/table in COPY statement

* Fix metadata sync issue. Update regression test

* Isolation test and bug fix

* Add Isolation test, fix foreign constraint deadlock issue

* Misc code review comments

* Test name needing to be quoted

* Refactor code from review comments

* Explaining shardGroupSplitIntervalListList

* Fix upgrade & downgrade

* Fix broken test

* Test fix Round 2

* Fixing bug and modifying test appropriately

* Fully qualify copy udf name. Run Reindent

* Address PR comments

* Fix null handling when creating AuxiliaryStructures

* Ensure local copy is triggered in tests

* Limit max shards that can be created with split

* Test failure fix

* Remove split_mode and use shard_transfer_mode instead'

* Fix test failure

* Fix test failure

* Fixing permission issue when splitting non-superuser owned tables

* Fix test expected output

* Remove extra space

* Fix test

* attempt to fix test

* Addressing Marco's PR comment

* Only clean shards created by workflow

* Remove from merge

* Update test
2022-07-18 02:54:15 -07:00
Sameer Awasekar 1243fe20ae Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-18 14:27:59 +05:30
Sameer Awasekar 10ea295d6c Flow completed 2022-07-18 13:51:58 +05:30
Nitish Upreti 5fe9bac4b9
Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-18 01:02:12 -07:00
Nitish Upreti 1c16060bd6 Only clean shards created by workflow 2022-07-18 00:47:42 -07:00
Sameer Awasekar b26bab32de Refactor subscriber code 2022-07-17 15:32:34 +05:30
Sameer Awasekar 1c617e7d1d Added code to drop publications and subscriptions 2022-07-16 15:38:51 +05:30
Onder Kalaci 3eaef027e2 Remove unused code
Probably left over from removing old repartitioning code
2022-07-15 10:28:46 +02:00
Onder Kalaci 483a3a5875 PG 15 Compat: Resolve compile issues + shmem requests
Similar to #5897, one more step for running Citus with PG 15.

This PR at least make Citus run with PG 15. I have not tried running the tests with PG 15.

Shmem changes are based on 4f2400cb3f

Compile breaks are mostly due to #6008
2022-07-15 10:11:39 +02:00
Nitish Upreti b1405d5eaf Addressing Marco's PR comment 2022-07-14 13:21:19 -07:00
Nitish Upreti 151b549a67 attempt to fix test 2022-07-14 11:19:33 -07:00
Nitish Upreti 07e69fc587 Fix test 2022-07-14 11:03:59 -07:00
Nitish Upreti cb3ddf18a7 Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-14 09:49:41 -07:00
Hanefi Onaldi ae58ca5783
Replace isolation tester func only once on enterprise tests (#6064)
This is a continuation of a refactor (with commit sha
2b7cf0c097) that aimed to use Citus helper
UDFs by default in iso tests.

PostgreSQL isolation test infrastructure uses some UDFs to detect
whether concurrent sessions block each other. Citus implements
alternatives to that UDF so that we are able to detect and report
distributed transactions that get blocked on the worker nodes as well.

We needed to explicitly replace PG helper functions with Citus
implementations in each isolation file. Now we replace them by default.
2022-07-14 19:16:53 +03:00
Sameer Awasekar 2fa1dac051 Add:
1) Drop colocated shards
2) Swap Metadata
3) Drop dummy shards
2022-07-14 16:23:00 +05:30
Sameer Awasekar 662a4aaec1 Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-14 09:52:27 +05:30
ywj 1675519f93
Support citus_columnar as separate extension (#5911)
* Support upgrade and downgrade and separate columnar as citus_columnar extension

Co-authored-by: Yanwen Jin <yanwjin@microsoft.com>
Co-authored-by: Jeff Davis <jeff@j-davis.com>
2022-07-13 21:08:29 -07:00
Sameer Awasekar 028abaa3cb Add:
1. Add publication/subscription logic
2. Add Catchup code
3. Blockwrites
4. Create Indexs, Replica Identity
2022-07-14 09:25:07 +05:30
Nitish Upreti 4b89493e36 Remove extra space 2022-07-13 14:29:26 -07:00
Nitish Upreti e87a8cd303 Fix test expected output 2022-07-13 14:19:20 -07:00
Nitish Upreti aca0a03d21 Fixing permission issue when splitting non-superuser owned tables 2022-07-13 14:09:02 -07:00
Onder Kalaci 6cd7319f12 Add more generic read-replica tests 2022-07-13 14:58:30 +02:00
Onder Kalaci 3c343d4563 Add regression tests for LOCK command citus.use_secondary_nodes=always mode 2022-07-13 14:27:11 +02:00
Onder Kalaci b2e9a5baf1 Make sure citus_is_coordinator works on read replicas 2022-07-13 14:11:18 +02:00
Onder Kalaci 8ab696f7e2 LOCK COMMAND does not require primaries at the start 2022-07-13 14:08:49 +02:00
Sameer Awasekar 51242a21d7 Added pub/sub 2022-07-13 13:16:01 +05:30
Nitish Upreti 8f56a30a21 Fix test failure 2022-07-12 16:01:30 -07:00
Nitish Upreti 8244db7d7c Fix test failure 2022-07-12 15:54:04 -07:00
Nitish Upreti 6fc7544f18 Remove split_mode and use shard_transfer_mode instead' 2022-07-12 15:42:20 -07:00
Nitish Upreti a0084a6590 Test failure fix 2022-07-12 14:37:46 -07:00
Nitish Upreti c4a0d55b45 Limit max shards that can be created with split 2022-07-12 13:42:35 -07:00
Nitish Upreti 0445d72cc7 Ensure local copy is triggered in tests 2022-07-12 10:38:18 -07:00
Nitish Upreti 8f8761ed40 Fix null handling when creating AuxiliaryStructures 2022-07-11 17:45:32 -07:00
Nitish Upreti cf8fea5a3f Address PR comments 2022-07-11 15:44:48 -07:00
Sameer Awasekar fd46d8011d Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-10 19:35:36 +05:30
Sameer Awasekar 0d26c4c8c7 Non-Blocking Shard Split 2022-07-10 19:14:41 +05:30
Nitish Upreti 5a1a8d5dcb Fully qualify copy udf name. Run Reindent 2022-07-09 16:35:10 -07:00
Nitish Upreti a6ca23feda Fixing bug and modifying test appropriately 2022-07-09 16:27:53 -07:00
Nitish Upreti d091a6bee2 Test fix Round 2 2022-07-09 00:05:00 -07:00
Nitish Upreti 7eafc95fd3 Fix broken test 2022-07-08 23:50:22 -07:00
Nitish Upreti f5f352c620 Fix upgrade & downgrade 2022-07-08 23:13:49 -07:00
Nitish Upreti eb1e2f1450 Merging with main 2022-07-08 22:57:11 -07:00
Nitish Upreti 5c635511d9 Explaining shardGroupSplitIntervalListList 2022-07-08 22:45:07 -07:00
Nitish Upreti a30c791eeb Refactor code from review comments 2022-07-08 22:23:38 -07:00
Nitish Upreti 7ef8ea948d Test name needing to be quoted 2022-07-08 21:44:35 -07:00
Nitish Upreti a515a49f4c Misc code review comments 2022-07-08 19:07:25 -07:00
Nitish Upreti f8c2c2555f Add Isolation test, fix foreign constraint deadlock issue 2022-07-08 15:03:46 -07:00
Nitish Upreti fb42c17ac3 Isolation test and bug fix 2022-07-08 11:52:36 -07:00
Nitish Upreti 55770d2816 Fix metadata sync issue. Update regression test 2022-07-06 21:27:25 -07:00
aykutbozkurt da089d72c5 we should check if relation is valid after fetching a relation 2022-07-06 16:35:01 +03:00
Halil Ozan Akgul 1490acbbe9 Removes incorrect parameter from get_all_active_transactions 2022-07-06 11:35:46 +03:00
Hanefi Onaldi 2b7cf0c097
Replace iso tester func only once (#5964)
Use Citus helper UDFs by default in iso tests

PostgreSQL isolation test infrastructure uses some UDFs to detect
whether concurrent sessions block each other. Citus implements
alternatives to that UDF so that we are able to detect and report
distributed transactions that get blocked on the worker nodes as well.

We needed to explicitly replace PG helper functions with Citus
implementations in each isolation file. Now we replace them by default.
2022-07-06 11:04:31 +03:00
Sameer Awasekar 49cada7e9e fix makefile to exclude enterprise-split 2022-07-06 12:40:16 +05:30
Sameer Awasekar ea9eb103e0 Fix nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar b6662ff872 Add hashmap, cleanup methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 213b15071b Rename methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 6d55154529 Remove pg_sleep and address nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar 99a21757d4 Fix name 2022-07-06 11:22:19 +05:30
Sameer Awasekar 0b11a324a4 Change value of static variable 2022-07-06 11:22:15 +05:30
Sameer Awasekar 91c9394ec4 Remove '.so' of plugin and have it embedded within Citus 2022-07-06 11:21:46 +05:30
Sameer Awasekar 23237e50a2 Add UDF description 2022-07-06 11:21:46 +05:30
Sameer Awasekar a23beeb43f Change return type, shard range as text 2022-07-06 11:21:43 +05:30
Sameer Awasekar b66067d09f Addressing review comments
1. Changed SM memory header
2. Changed API prototype
3. Refactored testcase using gset
2022-07-06 11:21:04 +05:30