external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,106 Commits
645 Branches
162 Tags
103 MiB
Commit Graph

3935 Commits (9d73bb889d25aa3896fb193c0fc8533f5153e34e)

Author SHA1 Message Date
aykut-bozkurt 67ac3da2b0
added citus_depended_objects udf and HideCitusDependentObjects GUC to hide citus depended objects from pg meta queries (#6055) 
use RecurseObjectDependencies api to find if an object is citus depended

make vanilla tests runnable to see if citus_depended function is working correctly
 2022-07-25 16:43:34 +03:00
Sameer Awasekar aef758a4a3 Add testcase 2022-07-24 21:02:33 +05:30
Sameer Awasekar 474fe7be1f Reindent 2022-07-24 17:19:45 +05:30
Sameer Awasekar 0f72d7f706 Rename connection params 2022-07-24 16:34:16 +05:30
Sameer Awasekar f6c11bf09c Reindent 2022-07-24 15:16:44 +05:30
Sameer Awasekar 9073564900 Adding comments to the workflow 2022-07-24 14:16:37 +05:30
Sameer Awasekar 0a6875de5a Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-24 09:19:50 +05:30
Sameer Awasekar b4f8226f96 Add comments for the functions 2022-07-23 22:58:00 +05:30
Sameer Awasekar 5a5505db55 Add comments for functions 2022-07-23 15:47:14 +05:30
Sameer Awasekar 8c871bcd10 Adding comments and code for cleanup 2022-07-22 20:10:18 +05:30
Marco Slot 5fabf94e39 Allow WITH HOLD cursors with parameters 2022-07-21 12:00:59 +02:00
Sameer Awasekar 9e88c14096 Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-21 11:19:27 +05:30
Sameer Awasekar 515f1e2346 Refactor 2022-07-21 10:42:04 +05:30
Hanefi Onaldi eb3e5ee227 Introduce citus_locks view 
citus_locks combines the pg_locks views from all nodes and adds
global_pid, nodeid, and relation_name. The columns of citus_locks don't
change based on the Postgres version, however the pg_locks's columns do.
Postgres 14 added one more column to pg_locks (waitstart timestamptz).
citus_locks has the most expansive column set, including the newly added
column. If citus_locks is queried in a Postgres version where pg_locks
doesn't have some columns, the values for those columns in citus_locks
will be NULL
 2022-07-21 03:06:57 +03:00
Nitish Upreti 3d569cc49a
Shard Split support for Columnar and Partitioned Table (#6067) 
DESCRIPTION:
This PR extends support for Partitioned and Columnar tables in blocking 'citus_split_shard_by_split_points' workflow.
Columnar Support : No special handling required. Just removing checks that fails split for columnar table and adding test coverage.
Partitioned Table Support :

Skip copying of parent table as they are empty, The partitions contain data and are treated as co-located shards that will be copied separately.
Attach partitions to parent on destination after inserting new shard metadata and before creating foreign key constraints.
MISC:
Fix Bug #4949 where Blocking shard moves fails if there is a foreign key between partitioned distributed tables (from child to parent).

TEST:
Added new test 'citus_split_shards_columnar_partitioned' for splitting 'partitioned' and 'columnar + partitioned' table.
Added new test 'shard_move_constraints_blocking' to add coverage for shard move bug fix.
Updated test 'citus_split_shard_by_split_points_negative' to allow columnar and partitioned table.
 2022-07-20 12:24:50 -07:00
Naisila Puka 7d6410c838
Drop postgres 12 support (#6040) 
* Remove if conditions with PG_VERSION_NUM < 13

* Remove server_above_twelve(&eleven) checks from tests

* Fix tests

* Remove pg12 and pg11 alternative test output files

* Remove pg12 specific normalization rules

* Some more if conditions in the code

* Change RemoteCollationIdExpression and some pg12/pg13 comments

* Remove some more normalization rules
 2022-07-20 17:49:36 +03:00
aykutbozkurt 108ca875ad fix assertion bugs related to list length 2022-07-20 10:53:12 +03:00
Hanefi Onaldi 6a32061c08
Renames configure.in to fix warnings (#6034) 
When building packages on ubuntu jammy, we started to see some warnings.

autoreconf: warning: autoconf input should be named 'configure.ac', not
'configure.in'
 2022-07-19 18:24:15 +02:00
Sameer Awasekar a4bf6a1dbb Change version of worker_split_shard_replication_setup UDF 2022-07-19 21:02:35 +05:30
aykutbozkurt ebb6d1c8c0 refactor code where GetObjectAddressFromParseTree is called because it returns list of addresses now 2022-07-19 18:13:12 +03:00
aykutbozkurt 9d232d7b00 change address method to return list of addresses 2022-07-19 18:13:11 +03:00
Önder Kalacı 90b1afe31e
Merge branch 'main' into baby_step_pg_15 2022-07-18 15:02:39 +02:00
Nitish Upreti 5b3537cdff
Shard Split for Citus (#6029) 
* Blocking split setup

* Add missing type

* Missing API from Metadata Sync

* Shard Split e2e code

* Worker Split Copy DestReceiver skeleton

* Basic destreceiver code

* worker_split_copy UDF

* UDF calling

* Split points are text

* Isolate Tenant and Split Shard Unification

* Fixing executor and misc

* Reindent code

* Fixing UDF definitions

* Hello World Local Copy works

* Remote copy hello world works

* Local and Remote binary test

* Fixing text local copy and adding tests

* Hello World shard split works

* Negative tests

* Blocking Split workflow works

* Refactor

* Bug fix

* Reindent

* Cleaning up and adding comments

* Basic test for shard split workflow

* ReIndent

* Circle CI integration

* Removing include causing circle-ci build failure

* Remove SplitCopyDestReceiver and use PartitionedResultDestReceiver

* Add support for citus.enable_binary_protocol

* Reindent

* Fix build break

* Update Test

* Cleanup on catch

* Addressing open comments

* Update downgrade script and quote schema/table in COPY statement

* Fix metadata sync issue. Update regression test

* Isolation test and bug fix

* Add Isolation test, fix foreign constraint deadlock issue

* Misc code review comments

* Test name needing to be quoted

* Refactor code from review comments

* Explaining shardGroupSplitIntervalListList

* Fix upgrade & downgrade

* Fix broken test

* Test fix Round 2

* Fixing bug and modifying test appropriately

* Fully qualify copy udf name. Run Reindent

* Address PR comments

* Fix null handling when creating AuxiliaryStructures

* Ensure local copy is triggered in tests

* Limit max shards that can be created with split

* Test failure fix

* Remove split_mode and use shard_transfer_mode instead'

* Fix test failure

* Fix test failure

* Fixing permission issue when splitting non-superuser owned tables

* Fix test expected output

* Remove extra space

* Fix test

* attempt to fix test

* Addressing Marco's PR comment

* Only clean shards created by workflow

* Remove from merge

* Update test
 2022-07-18 02:54:15 -07:00
Sameer Awasekar 1243fe20ae Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-18 14:27:59 +05:30
Sameer Awasekar 10ea295d6c Flow completed 2022-07-18 13:51:58 +05:30
Nitish Upreti 5fe9bac4b9
Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-18 01:02:12 -07:00
Nitish Upreti 1c16060bd6 Only clean shards created by workflow 2022-07-18 00:47:42 -07:00
Sameer Awasekar b26bab32de Refactor subscriber code 2022-07-17 15:32:34 +05:30
Sameer Awasekar 1c617e7d1d Added code to drop publications and subscriptions 2022-07-16 15:38:51 +05:30
Onder Kalaci 3eaef027e2 Remove unused code 
Probably left over from removing old repartitioning code
 2022-07-15 10:28:46 +02:00
Onder Kalaci 483a3a5875 PG 15 Compat: Resolve compile issues + shmem requests 
Similar to #5897, one more step for running Citus with PG 15.

This PR at least make Citus run with PG 15. I have not tried running the tests with PG 15.

Shmem changes are based on 4f2400cb3f

Compile breaks are mostly due to #6008
 2022-07-15 10:11:39 +02:00
Nitish Upreti b1405d5eaf Addressing Marco's PR comment 2022-07-14 13:21:19 -07:00
Nitish Upreti 151b549a67 attempt to fix test 2022-07-14 11:19:33 -07:00
Nitish Upreti 07e69fc587 Fix test 2022-07-14 11:03:59 -07:00
Nitish Upreti cb3ddf18a7 Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-14 09:49:41 -07:00
Hanefi Onaldi ae58ca5783
Replace isolation tester func only once on enterprise tests (#6064) 
This is a continuation of a refactor (with commit sha
2b7cf0c097) that aimed to use Citus helper
UDFs by default in iso tests.

PostgreSQL isolation test infrastructure uses some UDFs to detect
whether concurrent sessions block each other. Citus implements
alternatives to that UDF so that we are able to detect and report
distributed transactions that get blocked on the worker nodes as well.

We needed to explicitly replace PG helper functions with Citus
implementations in each isolation file. Now we replace them by default.
 2022-07-14 19:16:53 +03:00
Sameer Awasekar 2fa1dac051 Add: 
1) Drop colocated shards
2) Swap Metadata
3) Drop dummy shards
 2022-07-14 16:23:00 +05:30
Sameer Awasekar 662a4aaec1 Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-14 09:52:27 +05:30
ywj 1675519f93
Support citus_columnar as separate extension (#5911) 
* Support upgrade and downgrade and separate columnar as citus_columnar extension

Co-authored-by: Yanwen Jin <yanwjin@microsoft.com>
Co-authored-by: Jeff Davis <jeff@j-davis.com>
 2022-07-13 21:08:29 -07:00
Sameer Awasekar 028abaa3cb Add: 
1. Add publication/subscription logic
2. Add Catchup code
3. Blockwrites
4. Create Indexs, Replica Identity
 2022-07-14 09:25:07 +05:30
Nitish Upreti 4b89493e36 Remove extra space 2022-07-13 14:29:26 -07:00
Nitish Upreti e87a8cd303 Fix test expected output 2022-07-13 14:19:20 -07:00
Nitish Upreti aca0a03d21 Fixing permission issue when splitting non-superuser owned tables 2022-07-13 14:09:02 -07:00
Onder Kalaci 6cd7319f12 Add more generic read-replica tests 2022-07-13 14:58:30 +02:00
Onder Kalaci 3c343d4563 Add regression tests for LOCK command citus.use_secondary_nodes=always mode 2022-07-13 14:27:11 +02:00
Onder Kalaci b2e9a5baf1 Make sure citus_is_coordinator works on read replicas 2022-07-13 14:11:18 +02:00
Onder Kalaci 8ab696f7e2 LOCK COMMAND does not require primaries at the start 2022-07-13 14:08:49 +02:00
Sameer Awasekar 51242a21d7 Added pub/sub 2022-07-13 13:16:01 +05:30
Nitish Upreti 8f56a30a21 Fix test failure 2022-07-12 16:01:30 -07:00
Nitish Upreti 8244db7d7c Fix test failure 2022-07-12 15:54:04 -07:00
Nitish Upreti 6fc7544f18 Remove split_mode and use shard_transfer_mode instead' 2022-07-12 15:42:20 -07:00
Nitish Upreti a0084a6590 Test failure fix 2022-07-12 14:37:46 -07:00
Nitish Upreti c4a0d55b45 Limit max shards that can be created with split 2022-07-12 13:42:35 -07:00
Nitish Upreti 0445d72cc7 Ensure local copy is triggered in tests 2022-07-12 10:38:18 -07:00
Nitish Upreti 8f8761ed40 Fix null handling when creating AuxiliaryStructures 2022-07-11 17:45:32 -07:00
Nitish Upreti cf8fea5a3f Address PR comments 2022-07-11 15:44:48 -07:00
Sameer Awasekar fd46d8011d Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-10 19:35:36 +05:30
Sameer Awasekar 0d26c4c8c7 Non-Blocking Shard Split 2022-07-10 19:14:41 +05:30
Nitish Upreti 5a1a8d5dcb Fully qualify copy udf name. Run Reindent 2022-07-09 16:35:10 -07:00
Nitish Upreti a6ca23feda Fixing bug and modifying test appropriately 2022-07-09 16:27:53 -07:00
Nitish Upreti d091a6bee2 Test fix Round 2 2022-07-09 00:05:00 -07:00
Nitish Upreti 7eafc95fd3 Fix broken test 2022-07-08 23:50:22 -07:00
Nitish Upreti f5f352c620 Fix upgrade & downgrade 2022-07-08 23:13:49 -07:00
Nitish Upreti eb1e2f1450 Merging with main 2022-07-08 22:57:11 -07:00
Nitish Upreti 5c635511d9 Explaining shardGroupSplitIntervalListList 2022-07-08 22:45:07 -07:00
Nitish Upreti a30c791eeb Refactor code from review comments 2022-07-08 22:23:38 -07:00
Nitish Upreti 7ef8ea948d Test name needing to be quoted 2022-07-08 21:44:35 -07:00
Nitish Upreti a515a49f4c Misc code review comments 2022-07-08 19:07:25 -07:00
Nitish Upreti f8c2c2555f Add Isolation test, fix foreign constraint deadlock issue 2022-07-08 15:03:46 -07:00
Nitish Upreti fb42c17ac3 Isolation test and bug fix 2022-07-08 11:52:36 -07:00
Nitish Upreti 55770d2816 Fix metadata sync issue. Update regression test 2022-07-06 21:27:25 -07:00
aykutbozkurt da089d72c5 we should check if relation is valid after fetching a relation 2022-07-06 16:35:01 +03:00
Halil Ozan Akgul 1490acbbe9 Removes incorrect parameter from get_all_active_transactions 2022-07-06 11:35:46 +03:00
Hanefi Onaldi 2b7cf0c097
Replace iso tester func only once (#5964) 
Use Citus helper UDFs by default in iso tests

PostgreSQL isolation test infrastructure uses some UDFs to detect
whether concurrent sessions block each other. Citus implements
alternatives to that UDF so that we are able to detect and report
distributed transactions that get blocked on the worker nodes as well.

We needed to explicitly replace PG helper functions with Citus
implementations in each isolation file. Now we replace them by default.
 2022-07-06 11:04:31 +03:00
Sameer Awasekar 49cada7e9e fix makefile to exclude enterprise-split 2022-07-06 12:40:16 +05:30
Sameer Awasekar ea9eb103e0 Fix nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar b6662ff872 Add hashmap, cleanup methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 213b15071b Rename methods 2022-07-06 11:22:19 +05:30
Sameer Awasekar 6d55154529 Remove pg_sleep and address nits 2022-07-06 11:22:19 +05:30
Sameer Awasekar 99a21757d4 Fix name 2022-07-06 11:22:19 +05:30
Sameer Awasekar 0b11a324a4 Change value of static variable 2022-07-06 11:22:15 +05:30
Sameer Awasekar 91c9394ec4 Remove '.so' of plugin and have it embedded within Citus 2022-07-06 11:21:46 +05:30
Sameer Awasekar 23237e50a2 Add UDF description 2022-07-06 11:21:46 +05:30
Sameer Awasekar a23beeb43f Change return type, shard range as text 2022-07-06 11:21:43 +05:30
Sameer Awasekar b66067d09f Addressing review comments 
1. Changed SM memory header
2. Changed API prototype
3. Refactored testcase using gset
 2022-07-06 11:21:04 +05:30
Sameer Awasekar 5b82fd2ea3 Address review comments 2022-07-06 11:20:37 +05:30
Sameer Awasekar 672d198019 Renamed UDF to worker_split_shard_replication_setup 2022-07-06 11:20:34 +05:30
Sameer Awasekar b425573b08 Revert "Fix testcases" 
This reverts commit 2b926f6115b3ba11d7bbcdcdf02001dfd64a89fd.
 2022-07-06 11:16:58 +05:30
Sameer Awasekar aeaa1738bd Fix testcases 2022-07-06 11:16:21 +05:30
Sameer Awasekar 01540d78d6 Rename method to IsCommitRecursive 2022-07-06 11:15:23 +05:30
Sameer Awasekar f03a0343bc fix indentations 2022-07-06 11:15:23 +05:30
Sameer Awasekar f1be4888b9 Address comments 
1) Created new schema for tests
2) Variable renaming
 2022-07-06 11:15:18 +05:30
Sameer Awasekar 65eb62e723 Change encoding of replication slot names. Include table owners 2022-07-06 11:14:52 +05:30
Sameer Awasekar 04c2a92455 Reduced test timings 2022-07-06 11:14:52 +05:30
Sameer Awasekar 5a711a9176 Address comments 
- Fix warnings
 2022-07-06 11:14:52 +05:30
Sameer Awasekar 60ee33cfcc Add testcase and handling of UPDATE for non-partition column. 
- Change nodeId to uint32
- some test enchancement
- fix comments
 2022-07-06 11:14:52 +05:30
Sameer Awasekar 3864cd3187 Rename file to split_shard_replication_setup.c 2022-07-06 11:14:52 +05:30
Sameer Awasekar 028344b05c Remove unintentionally added pgoutput.h file 2022-07-06 11:14:52 +05:30
Sameer Awasekar cbc4a4a11c Change datatype from integer to bigint 2022-07-06 11:14:52 +05:30
Sameer Awasekar daae0bb10d Added colocated shard test 
- added a new test helper file
 2022-07-06 11:14:48 +05:30