external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,900 Commits
645 Branches
162 Tags
103 MiB
Commit Graph

5900 Commits (f78a25f72bfec6b7dc76ab9cdcff17737a84d960)

Author SHA1 Message Date
Yanwen Jin f78a25f72b Merge branch 'dev/yanwjin/upgrade1' of github.com:citusdata/citus into dev/yanwjin/upgrade1 2022-06-14 17:08:00 -07:00
Yanwen Jin 1a88d2eb53 Clean up after merge 2022-06-14 17:04:31 -07:00
Yanwen Jin f52755a0a9 Clean up after merge 2022-06-14 16:32:34 -07:00
Yanwen Jin 06babdbdbe Merge branch 'dev/yanwjin/upgrade1' of github.com:citusdata/citus into dev/yanwjin/upgrade1 2022-06-14 16:21:19 -07:00
Yanwen Jin e3bfb6f741 clean up after merge 2022-06-14 16:20:59 -07:00
ywj 9c0e00fb49
Delete citus--11.1-1--11.2-1.sql 
delete unnecessary file
 2022-06-14 16:03:31 -07:00
ywj 013cff8d5f
Delete citus--11.2-1--11.1-1.sql 
delete unnecessary files
 2022-06-14 16:03:09 -07:00
Yanwen Jin 5041fa6597 Fix some missing parts 2022-06-14 15:58:21 -07:00
Yanwen Jin e99eb59dd9 Merge remote-tracking branch 'origin/separate-extensions' into dev/yanwjin/upgrade1 2022-06-14 15:19:45 -07:00
Yanwen Jin 03c83f0ae8 Test: use columnar_internal instead of citus_internal in citus_finish_pg_upgrade forcolumnar 2022-06-10 13:51:03 -07:00
Yanwen Jin 99be185f52 Test: citus_finish_pg_upgrade 2022-06-10 13:16:14 -07:00
Yanwen Jin 40e4dab3de Test: update latest.sql in citus_finish_pg_upgrade 2022-06-10 13:11:46 -07:00
Yanwen Jin f7b548dc10 Test: add citus_finish_pg_upgrade 11.1-1 2022-06-10 12:55:14 -07:00
Yanwen Jin cf48ab0872 Revert "Test: disable dependencies set for columnar table access method in citus_finish_pg_upgrade latest" 
This reverts commit eab2b56fab.
 2022-06-09 12:57:02 -07:00
Yanwen Jin eab2b56fab Test: disable dependencies set for columnar table access method in citus_finish_pg_upgrade latest 2022-06-09 12:50:45 -07:00
Yanwen Jin 6c9c5b50df Fix test:upgrade_columnar_metapage_after 2022-06-09 11:00:59 -07:00
Yanwen Jin 057ad83619 remove unused variable 2022-06-08 17:42:46 -07:00
Yanwen Jin da83829974 remove unused variable 2022-06-08 17:38:10 -07:00
Yanwen Jin f95d4be924 remove unused variables 2022-06-08 17:31:01 -07:00
Yanwen Jin 64e1ec7488 Fix upgrade & downgrade errors 2022-06-08 17:16:36 -07:00
Yanwen Jin 1f9fb0511c test: removed duplicate declaration 2022-06-07 15:09:45 -07:00
Yanwen Jin 7174fb87a0 Testing purpose: test 'alter extension update' 2022-06-07 12:59:31 -07:00
Jeff Davis 1960e978da fixup columnar_ensure 2022-05-26 07:45:53 -07:00
Jeff Davis 335c2ab5a8 tmp3 2022-05-25 13:58:00 -07:00
Yanwen Jin e06abe3534 install citus_columnar for citus version >= 11.1 2022-05-25 13:55:32 -07:00
Jeff Davis 345b465b18 tmp2 2022-05-25 12:43:12 -07:00
Jeff Davis 07a6d32885 works kinda 2022-05-25 09:47:44 -07:00
Jeff Davis 72fd456870 rm 11.2 2022-05-25 09:21:50 -07:00
Jeff Davis 25b3e1d1c1 tmp1 2022-05-25 09:08:29 -07:00
Jeff Davis 9c8c52445b Add auxiliary control file 2022-05-25 08:42:11 -07:00
Yanwen Jin 5b67bfcc1a modify columnar_drop test 2022-05-25 08:40:31 -07:00
Yanwen Jin a1d7b08f45 add citus_columnar upgrade & downgrade script with new version 11.1-1 2022-05-25 08:40:31 -07:00
Yanwen Jin 60cc332abc Modify columnar sql files in Makefile 2022-05-25 08:40:31 -07:00
Yanwen Jin 4d287c9581 Update version as 11.1-1 and makefile for testing purpose 2022-05-25 08:40:31 -07:00
Yanwen Jin 51ae3ced76 Testing 2022-05-25 08:22:42 -07:00
Yanwen Jin a1adc06e74 Delete whitespace at end of lines 2022-05-25 08:22:42 -07:00
Yanwen Jin 32bd222e11 Fix whitespace in columnar 2022-05-25 08:22:42 -07:00
Yanwen Jin 12927b153c Fix whitespace 2022-05-25 08:22:42 -07:00
Yanwen Jin e869fb3be1 Fix whitespace style issues 2022-05-25 08:22:42 -07:00
Yanwen Jin 98bcf174c7 Fix some build errors 2022-05-25 08:22:41 -07:00
Yanwen Jin 502fb5ae57 Combine sql changes in columnar 2022-05-25 08:22:41 -07:00
Yanwen Jin b2681564d5 Support upgrade and downgrade for current citus customer 2022-05-25 08:22:41 -07:00
Yanwen Jin d5e25bcb3a Have citus_columnar as an extension 2022-05-25 08:21:33 -07:00
jeff-davis 74ce210f8b
Columnar: fix wraparound bug. (#5962) 
columnar_vacuum_rel() now advances relfrozenxid.

Fixes #5958.
 2022-05-25 07:50:48 -07:00
Yanwen Jin 2494604c59 modify columnar_drop test 2022-05-24 15:55:45 -07:00
Burak Velioglu 86781ec85a
Merge pull request #5966 from citusdata/velioglu/fix_depending_view_creation 
Fix schema and ownership of view while altering depending distributed table
 2022-05-24 17:09:20 +03:00
Burak Velioglu 1d7dda991f Create view and materialized views with right schema and owner while 
altering the distributed table.

To be able to alter view's owner without enforcing sequential mode.
Alter view process functions have been udpated to use metadata
connection.
 2022-05-24 15:27:30 +03:00
Gledis Zeneli 27ddb4fc8e
Do not obtain AccessShareLock before actual lock (#5965) 
Do not obtain AccessShareLock before acquiring the distributed locks.

Acquiring an AccessShareLock ensures that the relations which we are trying to get a distributed lock on will not be dropped in the time between when the LOCK command is issued and the LOCK commands are send to the worker. However, this also leads to distributed deadlocks in such scenarios:

```sql
-- for dist lock acquiring order coor, w1, w2

-- on w2
LOCK t1 IN ACCESS EXLUSIVE MODE;
-- acquire AccessShareLock locally on t1 to ensure it is not dropped while we get ready to distribute the lock

      -- concurrently on w1
      LOCK t1 IN ACCESS EXLUSIVE MODE;
      -- acquire AccessShareLock locally on t1 to ensure it is not dropped while we get ready to distribute the lock
      -- acquire dist lock on coor, w1, gets blocked on local AccessShareLock on w2

-- on w2 continuation of the execution above
-- starts to acquire dist locks and gets blocked on the coor by the lock acquired by w1

-- distributed deadlock

``` 

We opt for avoiding such deadlocks with the cost of the possibility of running into errors when the relations on which we are trying to acquire locks on get dropped.
 2022-05-23 13:06:38 +03:00
Önder Kalacı 2a768176c4
Merge pull request #5960 from citusdata/parallelize_node_activate 
Parallelize metadata syncing on node activate
 2022-05-23 09:25:11 +02:00
Onder Kalaci dd02e1755f Parallelize metadata syncing on node activate 
It is often useful to be able to sync the metadata in parallel
across nodes.

Also citus_finalize_upgrade_to_citus11() uses
start_metadata_sync_to_primary_nodes() after this commit.

Note that this commit does not parallelize all pieces of node
activation or metadata syncing. Instead, it tries to parallelize
potenially large parts of metadata, which is the objects and
distributed tables (in general Citus tables).

In the future, it would be nice to sync the reference tables
in parallel across nodes.

Create ~720 distributed tables / ~23450 shards
```SQL
-- declaratively partitioned table
CREATE TABLE github_events_looooooooooooooong_name (
  event_id bigint,
  event_type text,
  event_public boolean,
  repo_id bigint,
  payload jsonb,
  repo jsonb,
  actor jsonb,
  org jsonb,
  created_at timestamp
) PARTITION BY RANGE (created_at);

SELECT create_time_partitions(
  table_name         := 'github_events_looooooooooooooong_name',
  partition_interval := '1 day',
  end_at             := now() + '24 months'
);

CREATE INDEX ON github_events_looooooooooooooong_name USING btree (event_id, event_type, event_public, repo_id);
SELECT create_distributed_table('github_events_looooooooooooooong_name', 'repo_id');

SET client_min_messages TO ERROR;

```

across 1 node: almost same as expected
```SQL

SELECT start_metadata_sync_to_primary_nodes();
Time: 15664.418 ms (00:15.664)

select start_metadata_sync_to_node(nodename,nodeport) from pg_dist_node;
Time: 14284.069 ms (00:14.284)
```

across 7 nodes: ~3.5x improvement
```SQL

SELECT start_metadata_sync_to_primary_nodes();
┌──────────────────────────────────────┐
│ start_metadata_sync_to_primary_nodes │
├──────────────────────────────────────┤
│ t                                    │
└──────────────────────────────────────┘
(1 row)

Time: 25711.192 ms (00:25.711)

-- across 7 nodes
select start_metadata_sync_to_node(nodename,nodeport) from pg_dist_node;
Time: 82126.075 ms (01:22.126)
```
 2022-05-23 09:15:48 +02:00