Commit Graph

1292 Commits (140e8db3c248a5bc235d61b2b20d6219e2cf7d3d)

Author SHA1 Message Date
Sameer Awasekar 140e8db3c2 Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-08-04 16:30:41 +05:30
Sameer Awasekar 00c3830bee 1)Revert the original order of workflow
2) Introduce GetNextShardIdForSplitChild method
2022-08-04 16:26:15 +05:30
aykut-bozkurt 3ddc089651
stop distributing views with no distributed dependency if GUC DistributeLocalViews is set false. (#6083) 2022-08-04 12:34:40 +03:00
Sameer Awasekar b92b00e3cf Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-08-04 13:25:45 +05:30
Sameer Awasekar ad240b2904 Reorder split workflow to create empty publications first and then alter to add actual shards. 2022-08-04 12:42:53 +05:30
aykut-bozkurt 4ffe436bf9
we validate constraint as well if the statement is alter domain drop constraint (#6125) 2022-08-03 23:06:33 +03:00
Sameer Awasekar 15014f405f Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-08-03 08:17:19 +05:30
aykutbozkurt 7387c7ed3d address method should take parameter isPostprocess 2022-08-02 21:00:23 +03:00
aykutbozkurt c98a68662a introduces operation type for dist ops 2022-08-02 20:42:32 +03:00
aykutbozkurt 57ce4cf8c4 use address method to decide if we should run preprocess and postprocess steps for a distributed object 2022-08-02 20:42:32 +03:00
Sameer Awasekar d6258fe0db Address review comments.
1) Add partitioning test
2) Refactor some methods
2022-08-02 19:09:32 +05:30
Sameer Awasekar a430b0bffc Address review comments 2022-08-01 23:58:20 +05:30
Sameer Awasekar c56b79b6f7 Address review comments 2022-08-01 12:54:30 +05:30
aykut-bozkurt f372e93d22
we supress notice log during looking up function oid to not break pg vanilla tests. (#6082) 2022-08-01 10:14:35 +03:00
Sameer Awasekar adf73387f4 1. Added an isolation test
2. Added DisableAndDropShardSplitSubscription
3. Added creation of replica identities and validated through isolation test
2022-07-30 13:36:10 +05:30
Onder Kalaci 149771792b Remove useless version compats
most likely leftover from earlier versions
2022-07-29 10:31:55 +02:00
Marco Slot cff013a057 Fix issues with insert..select casts and column ordering 2022-07-28 13:23:57 +02:00
Onder Kalaci d67cf907a2 Detach relation access tracking from connection management 2022-07-28 11:27:59 +02:00
aykut-bozkurt b08e5ec29d
added some missing object address callbacks (#6056) 2022-07-27 17:36:04 +03:00
Onder Kalaci 12fa3aaf6b Concurrent shard move/copy and colocated table creation fix
It turns out that create_distributed_table
and citus_move/copy_shard_placement does not
work well concurrently.

To fix that, we need to acquire a lock, which
sounds like a good use of colocation lock.

However, the current usage of colocation lock is
limited to higher level UDFs like rebalance_table_shards
etc. Those usage of lock is still useful, but
we cannot acquire the same lock on citus_move_shard_placement
etc. because the coordinator connects to itself to acquire
the lock. Hence, the high level UDF blocks itself.

To fix that, we use one more colocation lock, with the placements
are the main objects to consider.
2022-07-27 10:01:19 +02:00
Onder Kalaci 26fdcb68f0 Optimize StringJoin() for when prefix-postfix is needed
Before this commit, we required multiple copies of the
same stringInfo if we needed to append/prepend data to
the stringInfo. Now, we optionally get prefix/postfix.

For large string operations, this can save up to %10
memory.
2022-07-27 09:49:08 +02:00
aykut-bozkurt 5f27445b69
enable propagation warnings before postgres vanilla tests (#6081) 2022-07-27 10:34:41 +03:00
Onder Kalaci 6c65d29924 Check the PGPROC's validity properly
We used to only check whether the PID is valid
or not. However, Postgres does not necessarily
set the PID of the backend to 0 when it exists.

Instead, we need to be able to check it from procArray.
IsBackendPid() is what pg_stat_activity also relies
on for a similar purpose.
2022-07-26 17:44:44 +02:00
aykut-bozkurt 67ac3da2b0
added citus_depended_objects udf and HideCitusDependentObjects GUC to hide citus depended objects from pg meta queries (#6055)
use RecurseObjectDependencies api to find if an object is citus depended

make vanilla tests runnable to see if citus_depended function is working correctly
2022-07-25 16:43:34 +03:00
Sameer Awasekar 0f72d7f706 Rename connection params 2022-07-24 16:34:16 +05:30
Sameer Awasekar b4f8226f96 Add comments for the functions 2022-07-23 22:58:00 +05:30
Sameer Awasekar 8c871bcd10 Adding comments and code for cleanup 2022-07-22 20:10:18 +05:30
Sameer Awasekar 9e88c14096 Merge remote-tracking branch 'origin' into users/saawasek/non_blocking_split_integrated 2022-07-21 11:19:27 +05:30
Naisila Puka 7d6410c838
Drop postgres 12 support (#6040)
* Remove if conditions with PG_VERSION_NUM < 13

* Remove server_above_twelve(&eleven) checks from tests

* Fix tests

* Remove pg12 and pg11 alternative test output files

* Remove pg12 specific normalization rules

* Some more if conditions in the code

* Change RemoteCollationIdExpression and some pg12/pg13 comments

* Remove some more normalization rules
2022-07-20 17:49:36 +03:00
Hanefi Onaldi 6a32061c08
Renames configure.in to fix warnings (#6034)
When building packages on ubuntu jammy, we started to see some warnings.

autoreconf: warning: autoconf input should be named 'configure.ac', not
'configure.in'
2022-07-19 18:24:15 +02:00
aykutbozkurt ebb6d1c8c0 refactor code where GetObjectAddressFromParseTree is called because it returns list of addresses now 2022-07-19 18:13:12 +03:00
aykutbozkurt 9d232d7b00 change address method to return list of addresses 2022-07-19 18:13:11 +03:00
Önder Kalacı 90b1afe31e
Merge branch 'main' into baby_step_pg_15 2022-07-18 15:02:39 +02:00
Nitish Upreti 5b3537cdff
Shard Split for Citus (#6029)
* Blocking split setup

* Add missing type

* Missing API from Metadata Sync

* Shard Split e2e code

* Worker Split Copy DestReceiver skeleton

* Basic destreceiver code

* worker_split_copy UDF

* UDF calling

* Split points are text

* Isolate Tenant and Split Shard Unification

* Fixing executor and misc

* Reindent code

* Fixing UDF definitions

* Hello World Local Copy works

* Remote copy hello world works

* Local and Remote binary test

* Fixing text local copy and adding tests

* Hello World shard split works

* Negative tests

* Blocking Split workflow works

* Refactor

* Bug fix

* Reindent

* Cleaning up and adding comments

* Basic test for shard split workflow

* ReIndent

* Circle CI integration

* Removing include causing circle-ci build failure

* Remove SplitCopyDestReceiver and use PartitionedResultDestReceiver

* Add support for citus.enable_binary_protocol

* Reindent

* Fix build break

* Update Test

* Cleanup on catch

* Addressing open comments

* Update downgrade script and quote schema/table in COPY statement

* Fix metadata sync issue. Update regression test

* Isolation test and bug fix

* Add Isolation test, fix foreign constraint deadlock issue

* Misc code review comments

* Test name needing to be quoted

* Refactor code from review comments

* Explaining shardGroupSplitIntervalListList

* Fix upgrade & downgrade

* Fix broken test

* Test fix Round 2

* Fixing bug and modifying test appropriately

* Fully qualify copy udf name. Run Reindent

* Address PR comments

* Fix null handling when creating AuxiliaryStructures

* Ensure local copy is triggered in tests

* Limit max shards that can be created with split

* Test failure fix

* Remove split_mode and use shard_transfer_mode instead'

* Fix test failure

* Fix test failure

* Fixing permission issue when splitting non-superuser owned tables

* Fix test expected output

* Remove extra space

* Fix test

* attempt to fix test

* Addressing Marco's PR comment

* Only clean shards created by workflow

* Remove from merge

* Update test
2022-07-18 02:54:15 -07:00
Sameer Awasekar 1243fe20ae Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-18 14:27:59 +05:30
Sameer Awasekar 10ea295d6c Flow completed 2022-07-18 13:51:58 +05:30
Nitish Upreti 5fe9bac4b9
Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-18 01:02:12 -07:00
Sameer Awasekar b26bab32de Refactor subscriber code 2022-07-17 15:32:34 +05:30
Sameer Awasekar 1c617e7d1d Added code to drop publications and subscriptions 2022-07-16 15:38:51 +05:30
Onder Kalaci 3eaef027e2 Remove unused code
Probably left over from removing old repartitioning code
2022-07-15 10:28:46 +02:00
Onder Kalaci 483a3a5875 PG 15 Compat: Resolve compile issues + shmem requests
Similar to #5897, one more step for running Citus with PG 15.

This PR at least make Citus run with PG 15. I have not tried running the tests with PG 15.

Shmem changes are based on 4f2400cb3f

Compile breaks are mostly due to #6008
2022-07-15 10:11:39 +02:00
Nitish Upreti cb3ddf18a7 Merge branch 'main' into users/niupre/BlockingSplitAPI 2022-07-14 09:49:41 -07:00
Sameer Awasekar 2fa1dac051 Add:
1) Drop colocated shards
2) Swap Metadata
3) Drop dummy shards
2022-07-14 16:23:00 +05:30
ywj 1675519f93
Support citus_columnar as separate extension (#5911)
* Support upgrade and downgrade and separate columnar as citus_columnar extension

Co-authored-by: Yanwen Jin <yanwjin@microsoft.com>
Co-authored-by: Jeff Davis <jeff@j-davis.com>
2022-07-13 21:08:29 -07:00
Sameer Awasekar 028abaa3cb Add:
1. Add publication/subscription logic
2. Add Catchup code
3. Blockwrites
4. Create Indexs, Replica Identity
2022-07-14 09:25:07 +05:30
Onder Kalaci b2e9a5baf1 Make sure citus_is_coordinator works on read replicas 2022-07-13 14:11:18 +02:00
Sameer Awasekar 51242a21d7 Added pub/sub 2022-07-13 13:16:01 +05:30
Sameer Awasekar fd46d8011d Merge remote-tracking branch 'origin/users/niupre/BlockingSplitAPI' into users/saawasek/non_blocking_split_integrated 2022-07-10 19:35:36 +05:30
Sameer Awasekar 0d26c4c8c7 Non-Blocking Shard Split 2022-07-10 19:14:41 +05:30
Nitish Upreti eb1e2f1450 Merging with main 2022-07-08 22:57:11 -07:00