external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,746 Commits
645 Branches
162 Tags
103 MiB
Commit Graph

6746 Commits (4dff8f3c674a7e7a538ef58cb63ec3803d04da10)

Author SHA1 Message Date
Marco Slot 03cae27782 Add tests for distributing functions with replication_model statement 2019-10-26 23:57:59 +02:00
Marco Slot 067657af26 Disallow distributed functions with distribution arguments unless replication_model is streaming 2019-10-26 23:57:59 +02:00
Önder Kalacı dceaddbe4d
Remove real-time/router executors (step 1) (#3125) 
See #3125 for details on each item.

* Remove real-time/router executor tests-1

These are the ones which doesn't have '_%d' in the test
output files.

* Remove real-time/router executor tests-2

These are the ones which has in the test
output files.

* Move the tests outputs to correct place

* Make sure that single shard commits use 2PC on adaptive executor

It looks like we've messed the tests in #2891. Fixing back.

* Use adaptive executor for all router queries

This becomes important because when task-tracker is picked, we
used to pick router executor, which doesn't make sense.

* Remove explicit references to real-time/router executors in the tests

* JobExecutorType never picks real-time/router executors

* Make sure to go incremental in test output numbers

* Even users cannot pick real-time anymore

* Do not use real-time/router custom scans

* Get rid of unnecessary normalizations

* Reflect unneeded normalizations

* Get rid of unnecessary test output file
 2019-10-25 10:54:54 +02:00
Jelte Fennema 96b8c36723 Better check for compiler flag capability 
When adding a `-Wno-some-error` flag, instead of trying to check for
`-Wno-some-error` support this checks for `-Wsome-error` support.
This is done because currently there's the a problem when compiling
citus with `./configure CFLAGS=-Werror`. If there's a warning (which is
then converted to an error) in addition to this error, the output will contain
the following message with GCC 7:

```
cc1: error: unrecognized command line option ‘-Wno-gnu-variable-sized-type-not-at-end’ [-Werror]
```

This is because of the following behaviour of GCC in case of unknown warnings:

> When an unrecognized warning option is requested (e.g., -Wunknown-warning), GCC
> emits a diagnostic stating that the option is not recognized.  However, if the
> -Wno- form is used, the behavior is slightly different: no diagnostic is
> produced for -Wno-unknown-warning unless other diagnostics are being produced.
> This allows the use of new -Wno- options with old compilers, but if something
> goes wrong, the compiler warns that an unrecognized option is present.

By changing the check to `-Wsome-error`, the check will actually fail
when the warning is not supported. Instead of silently being ignored
when checking, but then coming up when another error happens.
 2019-10-24 17:05:04 +00:00
Marco Slot 86a4c0925b
Revoke usage from the citus schema from public (#3123) 
Revoke usage from the citus schema from public
 2019-10-24 14:22:29 +02:00
Jelte Fennema 32676f9233
Use -std=gnu99 (C99 + GNU extensions) (#3124) 
Everything supports this, works in clang 6 and gcc 4.7

Also removes -Wdeclaration-after-statement, since declarations
after statements are only not supported in ISO C90.

Fixes #3122
 2019-10-24 14:01:24 +02:00
Jelte Fennema a5010e5b17
Add extra foreach convenience macros (#3117) 
This completely hides `ListCell` to the user of the loop

Example usage:
```c
WorkerNode *workerNode = NULL;

foreach_ptr(workerNode, workerNodeList) {
	// Do stuff with workerNode
}
```

Instead of:
```c
ListCell *workerNodeCell = NULL;

foreach(cell, workerNodeList) {
    WorkerNode *workerNode = lfirst(workerNodeCell);
	// Do stuff with workerNode
}
```
 2019-10-23 16:49:12 +02:00
Önder Kalacı 54de466876
Merge pull request #3067 from citusdata/add_upgrade_test_function 
Add upgrade test for distributed functions
 2019-10-23 16:16:43 +02:00
Onder Kalaci c2460a1c31 Add upgrade test for distributed functions 
Simply make sure that Citus can pushdown functions after pg upgrade.
 2019-10-23 12:07:51 +02:00
Philip Dubé b2f084d7f5 UnsetMetadataSyncedForAll: use CatalogTupleUpdateWithInfo 2019-10-23 00:45:11 +00:00
Philip Dubé 2a969fe4bb ssl_by_default: remove stray PG10 check 2019-10-23 00:27:54 +00:00
Marco Slot b8c8fd4612 Fix run_command_on_colocated_placements tests 2019-10-23 00:08:17 +02:00
Marco Slot a1162b2023 Rename 9.1 upgrade script to upgrade from 9.0-2 2019-10-23 00:08:17 +02:00
Marco Slot 04040e0a37 Revoke usage from the citus schema 2019-10-23 00:08:17 +02:00
Philip Dubé 2204a17dbd isolation_multiuser_locking: reorder GRANT to avoid deadlock on enterprise 2019-10-22 21:10:55 +00:00
Önder Kalacı f0f93d9c45
Merge pull request #3114 from citusdata/fix_leak_more_generic 
Fix memory leak on ReceiveResults
 2019-10-22 17:29:52 +02:00
Onder Kalaci a208f8b151 Fix memory leak on ReceiveResults 
It turns out that TupleDescGetAttInMetadata() allocates quite a lot
of memory. And, if the target list is long and there are too many rows
returning, the leak becomes appereant.

You can reproduce the issue wout the fix with the following commands:

```SQL

CREATE TABLE users_table (user_id int, time timestamp, value_1 int, value_2 int, value_3 float, value_4 bigint);
SELECT create_distributed_table('users_table', 'user_id');

insert into users_table SELECT i, now(), i, i, i, i FROM generate_series(0,99999)i;

-- load faster

-- 200,000
INSERT INTO users_table SELECT * FROM users_table;

-- 400,000
INSERT INTO users_table SELECT * FROM users_table;

-- 800,000
INSERT INTO users_table SELECT * FROM users_table;

-- 1,600,000
INSERT INTO users_table SELECT * FROM users_table;

-- 3,200,000
INSERT INTO users_table SELECT * FROM users_table;

-- 6,400,000
INSERT INTO users_table SELECT * FROM users_table;

-- 12,800,000
INSERT INTO users_table SELECT * FROM users_table;

-- making the target list entry wider speeds up the leak to show up
 select *,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,* FROM users_table ;

 ```
 2019-10-22 17:22:26 +02:00
Jelte Fennema 78e495e030
Add shouldhaveshards to pg_dist_node (#2960) 
This is an improvement over #2512.

This adds the boolean shouldhaveshards column to pg_dist_node. When it's false, create_distributed_table for new collocation groups will not create shards on that node. Reference tables will still be created on nodes where it is false.
 2019-10-22 16:47:16 +02:00
Jelte Fennema 5001c44990 Remove trailing whitespace 2019-10-22 11:26:08 +02:00
Hanefi Onaldi 7ebda04494
Update all c-style comments in migration files 2019-10-21 16:05:53 +03:00
Halil Ozan Akgül 210a6cc04b
Merge pull request #3111 from citusdata/refresh_materialized_view_with_subquery 
Refresh Materialized View with Subquery
 2019-10-17 16:07:35 +03:00
Halil Ozan Akgul 5f04ac774f Adds the tests for refresh materialized views 2019-10-17 16:00:56 +03:00
Önder Kalacı e065933928 Remove all the non-adaptive test schedules (#3112) 
Just because we'll remove the executors soon, and it doesn't
make sense to keep them running.

I'll remove the tests files with a follow-up commit, but it seems
safe to remove them from circleci now.
 2019-10-17 10:39:31 +03:00
Jelte Fennema 7abedc38b0
Support subqueries in HAVING (#3098) 
Areas for further optimization:
- Don't save subquery results to a local file on the coordinator when the subquery is not in the having clause
- Push the the HAVING with subquery to the workers if there's a group by on the distribution column
- Don't push down the results to the workers when we don't push down the HAVING clause, only the coordinator needs it

Fixes #520
Fixes #756
Closes #2047
 2019-10-16 16:40:14 +02:00
Onur TIRTIR 3bfb2a078b
Make changes on if-statement in ExtractRangeTableList for furhter walker types (#3110) 2019-10-16 15:50:09 +03:00
Onur TIRTIR d5f83dc110
Refactor range table walkers (#3109) 2019-10-16 01:20:49 +03:00
SaitTalhaNisanci 94a7e6475c
Remove copyright years (#2918) 
* Update year as 2012-2019

* Remove copyright years
 2019-10-15 17:44:30 +03:00
Jelte Fennema 9b2f4d71ac
Make sure some MX tests use defined shard_ids (#3103) 2019-10-12 22:46:14 +02:00
Philip Dubé 74cb168205 Remove Postgres 10 support 2019-10-11 21:56:56 +00:00
SaitTalhaNisanci 95633416f7 remove pg10 from pg upgrade tests (#3102) 2019-10-11 10:30:06 +02:00
Hadi Moshayedi 3fb9c63890
Merge pull request #3100 from citusdata/fix_typo 
Fix a typo
 2019-10-10 11:16:11 -07:00
Hadi Moshayedi b50d216536 Fix a typo 2019-10-10 10:44:41 -07:00
Philip Dubé 4063e7ca67 CALL delegation: apply strip_implicit_coercions to distribution argument 2019-10-10 17:42:43 +00:00
Philip Dubé 7ffd78b6e0 isolation_multiuser_locking 
Introduce a test which checks that locks are only acquired when a user has necessary permissions
Currently tests REINDEX, CREATE INDEX, TRUNCATE
 2019-10-10 16:58:41 +00:00
Philip Dubé dd490b6376 Cache whether an object is in pg_dist_object. Avoids redundant lookups for non-distributed objects 2019-10-10 14:50:38 +00:00
Hanefi Onaldi 2a47cff7b7
Bump citus to 9.0.0 2019-10-09 11:33:58 +03:00
Jelte Fennema f0f702c604
Add sane editorconfig settings (#3051) 
- UNIX line endings
- utf8
- newline at end of file
- trim trailing whitespace
 2019-10-08 15:07:01 +02:00
SaitTalhaNisanci 83667436f7
add support to run citus upgrade tests locally (#3083) 
* add support to run citus upgrade tests locally

* dont build tars if they already exist

* use current code instead of master for upgrade

* always build the current code

* copy the current citus code to have isolated citus upgrade tests

* fix configure and simplify copy
 2019-10-08 15:32:44 +03:00
Nils Dijk 4a4a220945
Fix enum add value order and pg12 (#3082) 
DESCRIPTION: Fix order for enum values and correctly support pg12

PG 12 introduces `ALTER TYPE ... ADD VALUE ...` during transactions. Earlier versions would error out when called in a transaction, hence we connect to workers outside of the transaction which could cause inconsistencies on pg12 now that postgres doesn't error with this syntax anymore.

During the implementation of this fix it became apparent there was an error with the ordering of enum labels when the type was recreated. A patch and test have been included.
 2019-10-07 17:16:19 +02:00
Jelte Fennema 01da11f264
Change citus truncate trigger to AFTER and add more upgrade tests (#3070) 
* Add more upgrade tests

* Fix citus trigger generation after upgrade

citus_truncate_trigger runs before truncate when created by create_distributed_table:
492d1b2cba/src/backend/distributed/commands/create_distributed_table.c (L1163)

* Remove pg_dist_jobid_seq
 2019-10-07 16:43:04 +02:00
SaitTalhaNisanci bffd110446
use citus instead of citusdata repo (#3077) 2019-10-07 12:03:42 +03:00
Onder Kalaci 3be72ce42f Make sure that distributed functions always have the correct user 
Objectives:

(a) both super user and regular user should have the correct owner for the function on the worker
(b) The transactional semantics would work fine for both super user and regular user
(c) non-super-user and non-function owner would get a reasonable error message if tries to distribute the function

Co-authored-by: @serprex
 2019-10-04 21:38:49 +00:00
SaitTalhaNisanci c547664fae
Add Citus upgrade tests with its job (#3003) 
* Add initial citus upgrade test

* Add restart databases and run tests in all nodes

* Add output for citus versions 8.0 8.1 8.2 and 8.3

* Add verify step for citus upgrade

* Add target for citus upgrade test in makefile

* Add check citus upgrade job

* Fix installation file path and add missing tar

* Run citus upgrade for v8.0 v8.1 v8.2 and v8.3

* Create upgrade_common file and rename upgrade check

* Add pg version to citus upgrade test

* Test with postgres 10 and 11 in citus upgrade tests

* Add readme for citus upgrade test

* Add some basic tests to citus upgrade tests

* Add citus upgrade mixed mode test

* Remove citus artifacts before installing another one

* Refactor citus upgrade test according to reviews

* quick and dirty rewrite of citus upgrade tests to support local execution.

I think we need to change the makefile in such a way that the tar files can be injected from the circle ci config file.

Also I removed some of the citus version checks you had to not have the requirement to pass that in separately from the pre tar file. I am not super happy with it, but two flags that need to be kept in sync is also not desirable. Instead I print out the citus version that is installed per node. This will not cause a failure if they are not what one would expect but it lets us verify we are running the expected version.

* use latest citusupgradetester in circleci

* update readme and use common alias for upgrade_common import
 2019-10-04 17:44:49 +03:00
Marco Slot 144a4f4cfa
Merge pull request #3075 from citusdata/fix/citus_schema_grant 
Grant usage on schema citus to public
 2019-10-04 13:22:57 +02:00
Marco Slot 1a3a174f67 Grant usage on schema citus to public 2019-10-04 12:26:08 +02:00
Marco Slot c15ddfb63f
Merge pull request #3068 from citusdata/fix_metadata_sync_locks 
Don't block for locks in SyncMetadataToNodes()
 2019-10-04 12:24:57 +02:00
Marco Slot 89377ee578 Move RowExclusiveLock to start in SyncMetadataToNodes 2019-10-04 12:07:41 +02:00
Hadi Moshayedi 217db2a03e Don't block for locks in SyncMetadataToNodes() 2019-10-03 16:53:36 -07:00
Hadi Moshayedi ae915493e6 Don't send metadata commands to not-synced workers. 
Otherwise some of the dependencies might not exist yet and
commands will error out.
 2019-10-03 16:52:25 -07:00
Marco Slot 7a49f158f3
Merge pull request #3072 from citusdata/drop_rebalancer 
Drop the rebalancer before creating new UDFs
 2019-10-03 18:14:47 +02:00