external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,615 Commits
645 Branches
162 Tags
103 MiB
Commit Graph

3616 Commits (46b4b55c7bc1671bfbe136b0d34b8f1d670ee762)

Author SHA1 Message Date
Burak Velioglu 790bf73ec0
Comment out isolation test 2022-01-17 17:01:12 +03:00
Burak Velioglu 4dab78a55a
Minor updates 2022-01-17 16:40:50 +03:00
Ahmet Gedemenli 8936543b80
Create wrapper function CreateObjectAddressDependencyDefList (#5623) 2022-01-17 15:35:40 +03:00
Burak Velioglu 5d1eae732c
Update metadata sync function names 2022-01-17 15:02:53 +03:00
Burak Velioglu 016c6179de
Fix test style 2022-01-17 14:19:44 +03:00
Burak Velioglu 8c27096d2c
Update master_metadata_snapshot to activate_node_snapshot in tests 2022-01-17 12:07:07 +03:00
Burak Velioglu 1ebc2fd6c5
activate_node_snapshot 2022-01-17 11:38:08 +03:00
Burak Velioglu 6fd5f8e658
Move functions 2022-01-16 23:54:04 +03:00
Burak Velioglu aedd09ffdf
Separate command list generation and execution 2022-01-16 23:49:31 +03:00
Burak Velioglu 381871890f
Move DetachPartitionCommandList back 2022-01-16 21:20:39 +03:00
Burak Velioglu e107fb7519
Move DistributedObjectMetadataSyncCommandList back 2022-01-16 21:02:16 +03:00
Burak Velioglu 4ae87e8057
Merge branch 'master' into velioglu/table_wo_seq_prototype 2022-01-16 17:11:04 +03:00
Burak Velioglu cc6c0cf7eb
Address review 2022-01-16 16:55:28 +03:00
Burak Velioglu 04511cc5e0
Adress reviews 2022-01-16 16:44:21 +03:00
Burak Velioglu 5e2e5c3470
Fix indentation for shell table func 2022-01-16 16:24:59 +03:00
Burak Velioglu 427bdf08a0
Use function to get seq dep definition list 2022-01-16 16:15:01 +03:00
Burak Velioglu ef7ddef07d
Test style 2022-01-16 15:57:18 +03:00
Burak Velioglu 9dc9d0d4dd
citus indent 2022-01-16 15:44:45 +03:00
Burak Velioglu fc307afdc4
Use function to get table command list and fix isolation test 2022-01-16 15:42:54 +03:00
Burak Velioglu 041a8c3d73
Revert truncate back 2022-01-16 15:10:04 +03:00
Burak Velioglu 060e6bf02f
Update truncate with delete from for metadata table 2022-01-15 23:24:17 +03:00
Ying Xu 4dca662e97
Making Columnar Dependency Free from Citus (#5622) 
* Removed distributed dependency in columnar_metadata.c

* Changed columnar_debug.c so that it no longer needed distributed/tuplestore and made it return a record instead of a tuplestore

* removed distributed/commands.h dependency

* Made columnar_tableam.c dependency-free

* Fixed spacing for columnar_store_memory_stats function

* indentation fix

* fixed test failures
 2022-01-14 09:43:05 -08:00
Burak Velioglu f29af1e23e
Check isolation test 2022-01-14 12:41:54 +03:00
Burak Velioglu 16807343b4
Update tests 2022-01-14 11:35:19 +03:00
Burak Velioglu 2d85858c26
Check isolation tests 2022-01-14 11:31:04 +03:00
Burak Velioglu f0536fa299
Check manual value for test 2022-01-14 11:18:20 +03:00
Onur Tirtir 70d8e1fe97
Assert that we will create indexes on shards via local execution (#5620) 2022-01-13 17:09:57 +01:00
Burak Velioglu 2789d45e9a
Fix failure test 2022-01-13 16:52:32 +03:00
Burak Velioglu b7f3e1e766
Fix multi partitioning test 2022-01-13 15:29:55 +03:00
Burak Velioglu 2e046ea8e5
Continue fixing multi-1 tests 2022-01-13 15:12:10 +03:00
Burak Velioglu 727b40e064
Fix multimetadata sync test 2022-01-13 13:31:33 +03:00
Burak Velioglu d84e1c1835
Fix style of UDF 2022-01-13 13:00:58 +03:00
Burak Velioglu 718a864232
Fix mx test 2022-01-13 12:46:12 +03:00
Burak Velioglu 4fe8ec3e61
Fix function create-drop scripts 2022-01-13 12:12:36 +03:00
Burak Velioglu d1f3f6e5e4
Fix upgrade tests 2022-01-13 11:34:11 +03:00
Burak Velioglu 3eb0121bb6
Normalize output 2022-01-13 11:24:25 +03:00
Burak Velioglu 02526f1bee
Minor refactors 2022-01-13 01:01:47 +03:00
Burak Velioglu 33efade04c
Merge branch 'master' into velioglu/table_wo_seq_prototype 2022-01-13 00:41:45 +03:00
Burak Velioglu 0c506606cd
Fix MX tests 2022-01-12 17:25:16 +03:00
Burak Velioglu 9fc09947ed
Citus indent 2022-01-12 15:26:01 +03:00
Burak Velioglu ad67942cda
Sequence and add node fix 2022-01-12 15:24:09 +03:00
Halil Ozan Akgul 63cd90e5dd Add missing library to dependencies.c 2022-01-11 18:36:43 +03:00
Önder Kalacı 46ec7cd5cf Enable MX for rebalancer tests 2022-01-11 12:07:39 +01:00
Burak Velioglu 5204db187b
Remove stale comment 2022-01-11 11:39:30 +03:00
Burak Velioglu 2b513c4100
Merge branch 'master' into velioglu/table_wo_seq_prototype 2022-01-11 11:35:32 +03:00
Burak Velioglu f3522763e5
Remove hacky group id check and unused UDFs 2022-01-11 11:22:45 +03:00
Burak Velioglu 697d1468fe
Use coordianated transaction for object prop 2022-01-10 22:08:43 +03:00
Önder Kalacı 885601c02c
Require superuser while activating a node (#5609) 
* Require superuser while activating a node

With this change, we require ActiveNode() (hence citus_add_node(),
citus_activate_node()) explicitly require for a superuser.

Before this commit, these functions were designed to work with
non-superuser roles with the relevent GRANTs given.

However, that is not a widely used way for calling the functions
above.

Due to possibility of non-super user calling the UDFs, they were
designed in a way that some commands were using some additional
short-lived superuser connections. That is:
	(a) breaking transactional behavior (e.g., ROLLBACK
 	    wouldn't fully rollback the whole transaction)
        (b) Making it very complicated to reason about which
	    parts of the node activation goes over which connections,
	    and becoming vulnerable to deadlocks / visibility issues.
 2022-01-10 08:30:13 -08:00
Burak Velioglu 4a7a8b8835
Fix sequence owner, reference shards and update tests 2022-01-10 11:30:52 +03:00
Onur Tirtir 3cc44ed8b3
Tell other backends it's safe to ignore the backend that concurrently built the shell table index (#5520) 
In addition to starting a new transaction, we also need to tell other
backends --including the ones spawned for connections opened to
localhost to build indexes on shards of this relation-- that concurrent
index builds can safely ignore us.

Normally, DefineIndex() only does that if index doesn't have any
predicates (i.e.: where clause) and no index expressions at all.
However, now that we already called standard process utility, index
build on the shell table is finished anyway.

The reason behind doing so is that we cannot guarantee not grabbing any
snapshots via adaptive executor, and the backends creating indexes on
local shards (if any) might block on waiting for current xact of the
current backend to finish, which would cause self deadlocks that are not
detectable.
 2022-01-10 10:23:09 +03:00
Burak Velioglu 76e1e1fd6b
Add adjust sequence settings and update tests 2022-01-09 19:35:07 +03:00
Marco Slot ee3b50b026 Disallow remote execution from queries on shards 2022-01-07 17:46:21 +01:00
Önder Kalacı 8d1b188620
Enable MX for the remaining failure tests (#5606) 2022-01-07 17:24:31 +01:00
Ahmet Gedemenli 3c834e6693
Disable foreign distributed tables (#5605) 
* Disable foreign distributed tables
* Add warning for existing distributed foreign tables
 2022-01-07 18:12:23 +03:00
Burak Velioglu 8006765504
Handle foreign tables and update tests 2022-01-07 15:48:20 +03:00
Burak Velioglu 21138784a3
Update comment 2022-01-07 13:03:31 +03:00
Burak Velioglu 942f1a2c48
Revert local sequence break change 2022-01-07 12:55:47 +03:00
Burak Velioglu 36fb662bf4
Merge branch 'master' into velioglu/table_wo_seq_prototype 2022-01-07 12:43:35 +03:00
Onder Kalaci 7cb1d6ae06 Improve metadata connections 
With https://github.com/citusdata/citus/pull/5493 we introduced
metadata specific connections.

With this connection we guarantee that there is a single metadata connection.
But note that this connection can be used for any other operation.
In other words, this connection is not only reserved for metadata
operations.

However, as https://github.com/citusdata/citus-enterprise/issues/715 showed
us that the logic has a flaw. We allowed ineligible connections to be
picked as metadata connections: such as exclusively claimed connections
or not fully initialized connections.

With this commit, we make sure that we only consider eligable connections
for metadata operations.
 2022-01-07 10:36:32 +01:00
Burak Velioglu 1383c442ea
Update multiple table integration sync 2022-01-07 12:18:06 +03:00
Onder Kalaci 9f2d9e1487 Move placement deletion from disable node to activate node 
We prefer the background daemon to only sync node metadata. That's
why we move placement metadata changes from disable node to
activate node. With that, we can make sure that disable node
only changes node metadata, whereas activate node syncs all
the metadata changes. In essence, we already expect all
nodes to be up when a node is activated. So, this does not change
the behavior much.
 2022-01-07 09:56:03 +01:00
Hanefi Onaldi 9edfbe7718 Fix the default value for DeferShardDeleteOnMove 
The default for GUC citus.defer_drop_after_shard_move is true. However
we initialize the global variable with a false value.
 2022-01-07 11:01:49 +03:00
Ahmet Gedemenli 45e423136c
Support foreign tables in MX (#5461) 2022-01-06 18:50:34 +03:00
Önder Kalacı 5305aa4246
Do not drop sequences when dropping metadata (#5584) 
Dropping sequences means we need to recreate
and hence losing the sequence.

With this commit, we keep the existing sequences
such that resyncing wouldn't drop the sequence.

We do that by breaking the dependency of the sequence
from the table.
 2022-01-06 09:48:34 +01:00
Önder Kalacı 8007adda25
Convert the function to a distributed function (#5596) 
so that when metadata is synced, the table is on the worker
 2022-01-06 11:32:40 +03:00
Burak Velioglu 299043dfaa
Update activate node 2022-01-05 16:33:11 +03:00
Önder Kalacı 6d9218540b
Enable single node tests with Citus MX (#5595) 
* Enable single node tests with Citus MX

The test already has comment on the changes
 2022-01-05 16:00:44 +03:00
jeff-davis 2e03efd91e
Columnar: move DDL hooks to citus to remove dependency. (#5547) 
Add a new hook ColumnarTableSetOptions_hook so that citus can get
control when the columnar table options change.
 2022-01-04 23:26:46 -08:00
jeff-davis c9292cfad1
Make pg_version_compat.h and listutils.c dependency-free. (#5548) 
Split distributed/version_compat.h into dependency-free
pg_version_compat.h, and the original which still has
dependencies. The original doesn't have much purpose, but until other
files have better discipline about including the correct header files,
then it's still needed.

Also make distributed/listutils.h dependency-free. Should be moved
outside of 'distributed' subdirectory, but that will cause significant
code churn, so leave for another cleanup patch.

Now both files can be included in columnar without creating a
dependency on citus.
 2022-01-04 23:02:08 -08:00
jeff-davis 1546aa0d9f
Columnar: use proper generic WAL interface. (#5543) 
Previously, we cheated by using the RM_GENERIC_ID record type, but not
actually using the generic WAL API. This worked because we always took
a full page image, and saved the extra work of allocating and copying
to a temporary page.

But it introduced complexity, and perhaps fragility, so better to just
use the API properly. The performance penalty for a serial data load
seems to be less than 1%.
 2022-01-04 22:42:21 -08:00
Onder Kalaci 22b5175fd1 Make sure that the community and enterprise tests produce the same output 2022-01-04 13:30:31 +01:00
Önder Kalacı 0a8b0b06c6
Do not allow distributed functions on non-metadata synced nodes (#5586) 
Before this commit, Citus was triggering metadata syncing
in the background when a function is distributed. However,
with Citus 11, we expect all clusters to have metadata synced
enabled. So, we do not expect any nodes not to have the metadata.

This change:
	(a) pro: simplifies the code and opens up possibilities
		 to simplify futher by reducing the scope of
		 bg worker to only sync node metadata
        (b) pro: explicitly asks users to sync the metadata such that
  	    any unforseen impact can be easily detected
        (c) con: For distributed functions without distribution
		 argument, we do not necessarily require the metadata
		 sycned. However, for completeness and simplicity, we
		 do so.
 2022-01-04 13:12:57 +01:00
Burak Velioglu c60f77bb99
Merge branch 'master' into velioglu/table_wo_seq_prototype 2022-01-04 12:53:14 +03:00
Burak Velioglu 320a8ecdea
Check groupid existence hackily while removing metadata 2022-01-04 12:04:33 +03:00
Burak Velioglu 7e3f2486f3
Remove metadata by checking isactive 2022-01-03 17:55:26 +03:00
Burak Velioglu 070e2afbe5
Remove placements wisely 2021-12-31 17:50:35 +03:00
Burak Velioglu 596e49db8c
Citus Indent 2021-12-31 13:16:30 +03:00
Burak Velioglu 3c1361dd4d
Move detach partition command list 2021-12-31 13:08:00 +03:00
Halil Ozan Akgul 9547228e8d Add isolation_check_mx test 2021-12-30 14:58:30 +03:00
Burak Velioglu 848d13f6eb
Changes depending on the discussion 2021-12-30 14:38:31 +03:00
Halil Ozan Akgul aef2d83c7d Fix metadata sync fails on multi_transaction_recovery 2021-12-29 11:21:32 +03:00
Burak Velioglu a348b6b097
Comment out failing test 2021-12-28 16:28:28 +03:00
Burak Velioglu cc68e87903
Citus indent 2021-12-28 12:48:20 +03:00
Burak Velioglu e610a147a1
Remove unnecessary comments on tests 2021-12-28 11:37:02 +03:00
Burak Velioglu d912cd9a9b
Update todos 2021-12-28 11:27:12 +03:00
Burak Velioglu cfe3f97faf
Create shell always for dist and reference table 2021-12-28 11:08:27 +03:00
Burak Velioglu d18525a906
Create shell always for local table 2021-12-28 10:58:35 +03:00
Burak Velioglu 416196b9be
Unmark dropped tabled on worker 2021-12-28 10:52:31 +03:00
Burak Velioglu 880533a609
Divide object and metadata handling 2021-12-27 18:14:51 +03:00
Önder Kalacı d33650d1c1
Record if any partitioned Citus tables during upgrade (#5555) 
With Citus 11, the default behavior is to sync the metadata.
However, partitioned tables created pre-Citus 11 might have
index names that are not compatiable with metadata syncing.

See https://github.com/citusdata/citus/issues/4962 for the
details.

With this commit, we record the existence of partitioned tables
such that we can fix it later if any exists.
 2021-12-27 03:33:34 -08:00
Halil Ozan Akgul 0c292a74f5 Fix metadata sync fails on multi_truncate 2021-12-27 13:54:53 +03:00
Önder Kalacı c9127f921f
Avoid round trips while fixing index names (#5549) 
With this commit, fix_partition_shard_index_names()
works significantly faster.

For example,

32 shards, 365 partitions, 5 indexes drop from ~120 seconds to ~44 seconds
32 shards, 1095 partitions, 5 indexes drop from ~600 seconds to ~265 seconds

`queryStringList` can be really long, because it may contain #partitions * #indexes entries.

Before this change, we were actually going through the executor where each command
in the query string triggers 1 round trip per entry in queryStringList.

The aim of this commit is to avoid the round-trips by creating a single query string.

I first simply tried sending `q1;q2;..;qn` . However, the executor is designed to
handle `q1;q2;..;qn` type of query executions via the infrastructure mentioned
above (e.g., by tracking the query indexes in the list and doing 1 statement
per round trip).

One another option could have been to change the executor such that only track
the query index when `queryStringList` is provided not with queryString
including multiple `;`s . That is (a) more work (b) could cause weird edge
cases with failure handling (c) felt like coding a special case in to the executor
 2021-12-27 10:29:37 +01:00
Halil Ozan Akgul bb636e6a29 Fix metadata sync fails on multi_function_evaluation 2021-12-24 19:32:58 +03:00
Halil Ozan Akgul 70e68d5312 Fix metadata sync fails on multi_name_lengths 2021-12-24 14:33:32 +03:00
Halil Ozan Akgul 5c2fb06322 Fix metadata sync fails on multi_sequence_default 2021-12-24 14:33:32 +03:00
Halil Ozan Akgul b9c06a6762 Turn metadata sync on in multi_metadata_sync 2021-12-24 10:58:13 +03:00
Hanefi Onaldi 479b2da740 Fix one flaky failure test 2021-12-23 20:11:45 +03:00
Ahmet Gedemenli 042d45b263 Propagate foreign server ops 2021-12-23 17:54:04 +03:00
Burak Velioglu 6598a23963
Dependency update 2021-12-23 17:46:37 +03:00
Onur Tirtir 61b5fb1cfc
Run failure_test_helpers in base schedule (#5559) 2021-12-23 12:54:12 +01:00