external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,121 Commits
645 Branches
162 Tags
103 MiB
Commit Graph

3055 Commits (users/saawasek/non_blocking_split_integrated)

Author SHA1 Message Date
Sameer Awasekar 0f72d7f706 Rename connection params 2022-07-24 16:34:16 +05:30
Sameer Awasekar f6c11bf09c Reindent 2022-07-24 15:16:44 +05:30
Sameer Awasekar 9073564900 Adding comments to the workflow 2022-07-24 14:16:37 +05:30
Sameer Awasekar 0a6875de5a Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-24 09:19:50 +05:30
Sameer Awasekar b4f8226f96 Add comments for the functions 2022-07-23 22:58:00 +05:30
Sameer Awasekar 5a5505db55 Add comments for functions 2022-07-23 15:47:14 +05:30
Sameer Awasekar 8c871bcd10 Adding comments and code for cleanup 2022-07-22 20:10:18 +05:30
Marco Slot 5fabf94e39 Allow WITH HOLD cursors with parameters 2022-07-21 12:00:59 +02:00
Sameer Awasekar 9e88c14096 Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-21 11:19:27 +05:30
Sameer Awasekar 515f1e2346 Refactor 2022-07-21 10:42:04 +05:30
Hanefi Onaldi eb3e5ee227 Introduce citus_locks view 
citus_locks combines the pg_locks views from all nodes and adds
global_pid, nodeid, and relation_name. The columns of citus_locks don't
change based on the Postgres version, however the pg_locks's columns do.
Postgres 14 added one more column to pg_locks (waitstart timestamptz).
citus_locks has the most expansive column set, including the newly added
column. If citus_locks is queried in a Postgres version where pg_locks
doesn't have some columns, the values for those columns in citus_locks
will be NULL
 2022-07-21 03:06:57 +03:00
Nitish Upreti 3d569cc49a
Shard Split support for Columnar and Partitioned Table (#6067) 
DESCRIPTION:
This PR extends support for Partitioned and Columnar tables in blocking 'citus_split_shard_by_split_points' workflow.
Columnar Support : No special handling required. Just removing checks that fails split for columnar table and adding test coverage.
Partitioned Table Support :

Skip copying of parent table as they are empty, The partitions contain data and are treated as co-located shards that will be copied separately.
Attach partitions to parent on destination after inserting new shard metadata and before creating foreign key constraints.
MISC:
Fix Bug #4949 where Blocking shard moves fails if there is a foreign key between partitioned distributed tables (from child to parent).

TEST:
Added new test 'citus_split_shards_columnar_partitioned' for splitting 'partitioned' and 'columnar + partitioned' table.
Added new test 'shard_move_constraints_blocking' to add coverage for shard move bug fix.
Updated test 'citus_split_shard_by_split_points_negative' to allow columnar and partitioned table.
 2022-07-20 12:24:50 -07:00
Naisila Puka 7d6410c838
Drop postgres 12 support (#6040) 
* Remove if conditions with PG_VERSION_NUM < 13

* Remove server_above_twelve(&eleven) checks from tests

* Fix tests

* Remove pg12 and pg11 alternative test output files

* Remove pg12 specific normalization rules

* Some more if conditions in the code

* Change RemoteCollationIdExpression and some pg12/pg13 comments

* Remove some more normalization rules
 2022-07-20 17:49:36 +03:00
aykutbozkurt 108ca875ad fix assertion bugs related to list length 2022-07-20 10:53:12 +03:00
Sameer Awasekar a4bf6a1dbb Change version of worker_split_shard_replication_setup UDF 2022-07-19 21:02:35 +05:30
aykutbozkurt ebb6d1c8c0 refactor code where GetObjectAddressFromParseTree is called because it returns list of addresses now 2022-07-19 18:13:12 +03:00
aykutbozkurt 9d232d7b00 change address method to return list of addresses 2022-07-19 18:13:11 +03:00
Önder Kalacı 90b1afe31e
Merge branch 'main' into baby_step_pg_15 2022-07-18 15:02:39 +02:00
Nitish Upreti 5b3537cdff
Shard Split for Citus (#6029) 
* Blocking split setup

* Add missing type

* Missing API from Metadata Sync

* Shard Split e2e code

* Worker Split Copy DestReceiver skeleton

* Basic destreceiver code

* worker_split_copy UDF

* UDF calling

* Split points are text

* Isolate Tenant and Split Shard Unification

* Fixing executor and misc

* Reindent code

* Fixing UDF definitions

* Hello World Local Copy works

* Remote copy hello world works

* Local and Remote binary test

* Fixing text local copy and adding tests

* Hello World shard split works

* Negative tests

* Blocking Split workflow works

* Refactor

* Bug fix

* Reindent

* Cleaning up and adding comments

* Basic test for shard split workflow

* ReIndent

* Circle CI integration

* Removing include causing circle-ci build failure

* Remove SplitCopyDestReceiver and use PartitionedResultDestReceiver

* Add support for citus.enable_binary_protocol

* Reindent

* Fix build break

* Update Test

* Cleanup on catch

* Addressing open comments

* Update downgrade script and quote schema/table in COPY statement

* Fix metadata sync issue. Update regression test

* Isolation test and bug fix

* Add Isolation test, fix foreign constraint deadlock issue

* Misc code review comments

* Test name needing to be quoted

* Refactor code from review comments

* Explaining shardGroupSplitIntervalListList

* Fix upgrade & downgrade

* Fix broken test

* Test fix Round 2

* Fixing bug and modifying test appropriately

* Fully qualify copy udf name. Run Reindent

* Address PR comments

* Fix null handling when creating AuxiliaryStructures

* Ensure local copy is triggered in tests

* Limit max shards that can be created with split

* Test failure fix

* Remove split_mode and use shard_transfer_mode instead'

* Fix test failure

* Fix test failure

* Fixing permission issue when splitting non-superuser owned tables

* Fix test expected output

* Remove extra space

* Fix test

* attempt to fix test

* Addressing Marco's PR comment

* Only clean shards created by workflow

* Remove from merge

* Update test
 2022-07-18 02:54:15 -07:00
Sameer Awasekar 1243fe20ae Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-18 14:27:59 +05:30
Sameer Awasekar 10ea295d6c Flow completed 2022-07-18 13:51:58 +05:30
Nitish Upreti 5fe9bac4b9
Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-18 01:02:12 -07:00
Nitish Upreti 1c16060bd6 Only clean shards created by workflow 2022-07-18 00:47:42 -07:00
Sameer Awasekar b26bab32de Refactor subscriber code 2022-07-17 15:32:34 +05:30
Sameer Awasekar 1c617e7d1d Added code to drop publications and subscriptions 2022-07-16 15:38:51 +05:30
Onder Kalaci 3eaef027e2 Remove unused code 
Probably left over from removing old repartitioning code
 2022-07-15 10:28:46 +02:00
Onder Kalaci 483a3a5875 PG 15 Compat: Resolve compile issues + shmem requests 
Similar to #5897, one more step for running Citus with PG 15.

This PR at least make Citus run with PG 15. I have not tried running the tests with PG 15.

Shmem changes are based on 4f2400cb3f

Compile breaks are mostly due to #6008
 2022-07-15 10:11:39 +02:00
Nitish Upreti b1405d5eaf Addressing Marco's PR comment 2022-07-14 13:21:19 -07:00
Nitish Upreti cb3ddf18a7 Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-14 09:49:41 -07:00
Sameer Awasekar 2fa1dac051 Add: 
1) Drop colocated shards
2) Swap Metadata
3) Drop dummy shards
 2022-07-14 16:23:00 +05:30
Sameer Awasekar 662a4aaec1 Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-14 09:52:27 +05:30
ywj 1675519f93
Support citus_columnar as separate extension (#5911) 
* Support upgrade and downgrade and separate columnar as citus_columnar extension

Co-authored-by: Yanwen Jin <yanwjin@microsoft.com>
Co-authored-by: Jeff Davis <jeff@j-davis.com>
 2022-07-13 21:08:29 -07:00
Sameer Awasekar 028abaa3cb Add: 
1. Add publication/subscription logic
2. Add Catchup code
3. Blockwrites
4. Create Indexs, Replica Identity
 2022-07-14 09:25:07 +05:30
Nitish Upreti aca0a03d21 Fixing permission issue when splitting non-superuser owned tables 2022-07-13 14:09:02 -07:00
Onder Kalaci b2e9a5baf1 Make sure citus_is_coordinator works on read replicas 2022-07-13 14:11:18 +02:00
Onder Kalaci 8ab696f7e2 LOCK COMMAND does not require primaries at the start 2022-07-13 14:08:49 +02:00
Sameer Awasekar 51242a21d7 Added pub/sub 2022-07-13 13:16:01 +05:30
Nitish Upreti 8244db7d7c Fix test failure 2022-07-12 15:54:04 -07:00
Nitish Upreti 6fc7544f18 Remove split_mode and use shard_transfer_mode instead' 2022-07-12 15:42:20 -07:00
Nitish Upreti c4a0d55b45 Limit max shards that can be created with split 2022-07-12 13:42:35 -07:00
Nitish Upreti 0445d72cc7 Ensure local copy is triggered in tests 2022-07-12 10:38:18 -07:00
Nitish Upreti 8f8761ed40 Fix null handling when creating AuxiliaryStructures 2022-07-11 17:45:32 -07:00
Nitish Upreti cf8fea5a3f Address PR comments 2022-07-11 15:44:48 -07:00
Sameer Awasekar fd46d8011d Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-10 19:35:36 +05:30
Sameer Awasekar 0d26c4c8c7 Non-Blocking Shard Split 2022-07-10 19:14:41 +05:30
Nitish Upreti 5a1a8d5dcb Fully qualify copy udf name. Run Reindent 2022-07-09 16:35:10 -07:00
Nitish Upreti a6ca23feda Fixing bug and modifying test appropriately 2022-07-09 16:27:53 -07:00
Nitish Upreti 7eafc95fd3 Fix broken test 2022-07-08 23:50:22 -07:00
Nitish Upreti f5f352c620 Fix upgrade & downgrade 2022-07-08 23:13:49 -07:00
Nitish Upreti eb1e2f1450 Merging with main 2022-07-08 22:57:11 -07:00
Nitish Upreti 5c635511d9 Explaining shardGroupSplitIntervalListList 2022-07-08 22:45:07 -07:00
Nitish Upreti a30c791eeb Refactor code from review comments 2022-07-08 22:23:38 -07:00
Nitish Upreti 7ef8ea948d Test name needing to be quoted 2022-07-08 21:44:35 -07:00
Nitish Upreti a515a49f4c Misc code review comments 2022-07-08 19:07:25 -07:00
Nitish Upreti f8c2c2555f Add Isolation test, fix foreign constraint deadlock issue 2022-07-08 15:03:46 -07:00
Nitish Upreti fb42c17ac3 Isolation test and bug fix 2022-07-08 11:52:36 -07:00
Nitish Upreti 55770d2816 Fix metadata sync issue. Update regression test 2022-07-06 21:27:25 -07:00
aykutbozkurt da089d72c5 we should check if relation is valid after fetching a relation 2022-07-06 16:35:01 +03:00
Halil Ozan Akgul 1490acbbe9 Removes incorrect parameter from get_all_active_transactions 2022-07-06 11:35:46 +03:00
Sameer Awasekar ea9eb103e0 Fix nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar b6662ff872 Add hashmap, cleanup methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 213b15071b Rename methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 6d55154529 Remove pg_sleep and address nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar 99a21757d4 Fix name 2022-07-06 11:22:19 +05:30
Sameer Awasekar 0b11a324a4 Change value of static variable 2022-07-06 11:22:15 +05:30
Sameer Awasekar 91c9394ec4 Remove '.so' of plugin and have it embedded within Citus 2022-07-06 11:21:46 +05:30
Sameer Awasekar 23237e50a2 Add UDF description 2022-07-06 11:21:46 +05:30
Sameer Awasekar a23beeb43f Change return type, shard range as text 2022-07-06 11:21:43 +05:30
Sameer Awasekar b66067d09f Addressing review comments 
1. Changed SM memory header
2. Changed API prototype
3. Refactored testcase using gset
 2022-07-06 11:21:04 +05:30
Sameer Awasekar 5b82fd2ea3 Address review comments 2022-07-06 11:20:37 +05:30
Sameer Awasekar 672d198019 Renamed UDF to worker_split_shard_replication_setup 2022-07-06 11:20:34 +05:30
Sameer Awasekar 01540d78d6 Rename method to IsCommitRecursive 2022-07-06 11:15:23 +05:30
Sameer Awasekar f03a0343bc fix indentations 2022-07-06 11:15:23 +05:30
Sameer Awasekar f1be4888b9 Address comments 
1) Created new schema for tests
2) Variable renaming
 2022-07-06 11:15:18 +05:30
Sameer Awasekar 65eb62e723 Change encoding of replication slot names. Include table owners 2022-07-06 11:14:52 +05:30
Sameer Awasekar 04c2a92455 Reduced test timings 2022-07-06 11:14:52 +05:30
Sameer Awasekar 5a711a9176 Address comments 
- Fix warnings
 2022-07-06 11:14:52 +05:30
Sameer Awasekar 60ee33cfcc Add testcase and handling of UPDATE for non-partition column. 
- Change nodeId to uint32
- some test enchancement
- fix comments
 2022-07-06 11:14:52 +05:30
Sameer Awasekar 3864cd3187 Rename file to split_shard_replication_setup.c 2022-07-06 11:14:52 +05:30
Sameer Awasekar cbc4a4a11c Change datatype from integer to bigint 2022-07-06 11:14:52 +05:30
Sameer Awasekar daae0bb10d Added colocated shard test 
- added a new test helper file
 2022-07-06 11:14:48 +05:30
Sameer Awasekar d71ead0e43 Remove unwanted headers 2022-07-06 11:14:02 +05:30
Sameer Awasekar 5da75b84ac Removed some methods. Handling review comments 2022-07-06 11:14:02 +05:30
Sameer Awasekar 4b064fa321 change name of logical decoding plugin 2022-07-06 11:14:02 +05:30
Sameer Awasekar 25bae46cd6 Fix identations 2022-07-06 11:14:02 +05:30
Sameer Awasekar 121ec8cd51 Change prototype of GetShardSplitInfoSMHeaderFromDSMHandle 2022-07-06 11:14:02 +05:30
Sameer Awasekar 59c3c93aa4 Addressing comments 
- Changed name of functions
- Moved some functions from .h to .c file
 2022-07-06 11:14:02 +05:30
Sameer Awasekar 75c6484e02 Address review comments(partially done) 
- Parameterize testcase methods
- remove code for handline 'UPDATE'.
- replication slot encoding/decoding logic changed now
 2022-07-06 11:14:02 +05:30
Sameer Awasekar a43cab7e71 Add DELETE scenario 2022-07-06 11:08:16 +05:30
Sameer Awasekar c9844abea9 Introducing 'split_shard_replication_setup' UDF to handle the catchup phase of 
citus_split_shard.
 2022-07-06 11:08:10 +05:30
aykutbozkurt d53a7760b0 * alter index/table rename weird syntax supported, 
* correct the wrong level of lock if the weird syntax is used
 2022-07-04 21:27:47 +03:00
aykutbozkurt ba62c0a148 auto is a valid option for vacuum index_cleanup. 2022-07-04 19:27:55 +03:00
Ahmet Gedemenli c8e1e243b8
Fix matviews for citus_add_local_table_to_metadata (#6023) 2022-07-04 17:00:07 +03:00
Nitish Upreti e29222458c Update downgrade script and quote schema/table in COPY statement 2022-07-01 13:35:27 -07:00
Hanefi Onaldi f60809a6c1
Fix downgrade scripts from 11.0-2 to 11.0-1 (#6039) 2022-06-29 22:43:50 +03:00
Nitish Upreti 4fcdff53ae Addressing open comments 2022-06-28 16:45:26 -07:00
Nitish Upreti ebd0e9a617 Cleanup on catch 2022-06-28 16:22:37 -07:00
Nitish Upreti 383fc2b6f0 Fix build break 2022-06-28 09:37:11 -07:00
Onder Kalaci bab4c0a8c3 Fixes a bug that prevents upgrades when there are no worker nodes 2022-06-28 15:54:49 +02:00
Onder Kalaci bd3a070369 Fixes a bug that prevents upgrades when there COMPRESSION and DEFAULT columns 2022-06-28 13:36:00 +02:00