external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,813 Commits
637 Branches
162 Tags
99 MiB
Commit Graph

2813 Commits (23c26e9cbf793fb507aa4d331dbaa3ef6f57c6c8)

Author SHA1 Message Date
Hanefi Onaldi 23c26e9cbf
Remove PG10 to PG11 upgrade test 2019-10-23 11:33:28 +03:00
Onder Kalaci f8417d184c Fix memory leak on ReceiveResults 
It turns out that TupleDescGetAttInMetadata() allocates quite a lot
of memory. And, if the target list is long and there are too many rows
returning, the leak becomes appereant.

You can reproduce the issue wout the fix with the following commands:

```SQL

CREATE TABLE users_table (user_id int, time timestamp, value_1 int, value_2 int, value_3 float, value_4 bigint);
SELECT create_distributed_table('users_table', 'user_id');

insert into users_table SELECT i, now(), i, i, i, i FROM generate_series(0,99999)i;

-- load faster

-- 200,000
INSERT INTO users_table SELECT * FROM users_table;

-- 400,000
INSERT INTO users_table SELECT * FROM users_table;

-- 800,000
INSERT INTO users_table SELECT * FROM users_table;

-- 1,600,000
INSERT INTO users_table SELECT * FROM users_table;

-- 3,200,000
INSERT INTO users_table SELECT * FROM users_table;

-- 6,400,000
INSERT INTO users_table SELECT * FROM users_table;

-- 12,800,000
INSERT INTO users_table SELECT * FROM users_table;

-- making the target list entry wider speeds up the leak to show up
 select *,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,* FROM users_table ;

 ```
 2019-10-22 17:37:19 +02:00
Onur Tirtir e760dd69ac Re-add runstatedir option to configure 2019-10-10 11:12:57 +03:00
Hanefi Onaldi 566283d202
Bump citus to 9.0.0 
(cherry picked from commit 2a47cff7b7)
 2019-10-09 14:01:56 +03:00
Hanefi Onaldi 9747ca40ae
Bump Citus version to 9.0.0 2019-10-07 21:00:01 +03:00
Nils Dijk 4a4a220945
Fix enum add value order and pg12 (#3082) 
DESCRIPTION: Fix order for enum values and correctly support pg12

PG 12 introduces `ALTER TYPE ... ADD VALUE ...` during transactions. Earlier versions would error out when called in a transaction, hence we connect to workers outside of the transaction which could cause inconsistencies on pg12 now that postgres doesn't error with this syntax anymore.

During the implementation of this fix it became apparent there was an error with the ordering of enum labels when the type was recreated. A patch and test have been included.
 2019-10-07 17:16:19 +02:00
Jelte Fennema 01da11f264
Change citus truncate trigger to AFTER and add more upgrade tests (#3070) 
* Add more upgrade tests

* Fix citus trigger generation after upgrade

citus_truncate_trigger runs before truncate when created by create_distributed_table:
492d1b2cba/src/backend/distributed/commands/create_distributed_table.c (L1163)

* Remove pg_dist_jobid_seq
 2019-10-07 16:43:04 +02:00
SaitTalhaNisanci bffd110446
use citus instead of citusdata repo (#3077) 2019-10-07 12:03:42 +03:00
Onder Kalaci 3be72ce42f Make sure that distributed functions always have the correct user 
Objectives:

(a) both super user and regular user should have the correct owner for the function on the worker
(b) The transactional semantics would work fine for both super user and regular user
(c) non-super-user and non-function owner would get a reasonable error message if tries to distribute the function

Co-authored-by: @serprex
 2019-10-04 21:38:49 +00:00
SaitTalhaNisanci c547664fae
Add Citus upgrade tests with its job (#3003) 
* Add initial citus upgrade test

* Add restart databases and run tests in all nodes

* Add output for citus versions 8.0 8.1 8.2 and 8.3

* Add verify step for citus upgrade

* Add target for citus upgrade test in makefile

* Add check citus upgrade job

* Fix installation file path and add missing tar

* Run citus upgrade for v8.0 v8.1 v8.2 and v8.3

* Create upgrade_common file and rename upgrade check

* Add pg version to citus upgrade test

* Test with postgres 10 and 11 in citus upgrade tests

* Add readme for citus upgrade test

* Add some basic tests to citus upgrade tests

* Add citus upgrade mixed mode test

* Remove citus artifacts before installing another one

* Refactor citus upgrade test according to reviews

* quick and dirty rewrite of citus upgrade tests to support local execution.

I think we need to change the makefile in such a way that the tar files can be injected from the circle ci config file.

Also I removed some of the citus version checks you had to not have the requirement to pass that in separately from the pre tar file. I am not super happy with it, but two flags that need to be kept in sync is also not desirable. Instead I print out the citus version that is installed per node. This will not cause a failure if they are not what one would expect but it lets us verify we are running the expected version.

* use latest citusupgradetester in circleci

* update readme and use common alias for upgrade_common import
 2019-10-04 17:44:49 +03:00
Marco Slot 144a4f4cfa
Merge pull request #3075 from citusdata/fix/citus_schema_grant 
Grant usage on schema citus to public
 2019-10-04 13:22:57 +02:00
Marco Slot 1a3a174f67 Grant usage on schema citus to public 2019-10-04 12:26:08 +02:00
Marco Slot c15ddfb63f
Merge pull request #3068 from citusdata/fix_metadata_sync_locks 
Don't block for locks in SyncMetadataToNodes()
 2019-10-04 12:24:57 +02:00
Marco Slot 89377ee578 Move RowExclusiveLock to start in SyncMetadataToNodes 2019-10-04 12:07:41 +02:00
Hadi Moshayedi 217db2a03e Don't block for locks in SyncMetadataToNodes() 2019-10-03 16:53:36 -07:00
Hadi Moshayedi ae915493e6 Don't send metadata commands to not-synced workers. 
Otherwise some of the dependencies might not exist yet and
commands will error out.
 2019-10-03 16:52:25 -07:00
Marco Slot 7a49f158f3
Merge pull request #3072 from citusdata/drop_rebalancer 
Drop the rebalancer before creating new UDFs
 2019-10-03 18:14:47 +02:00
Marco Slot 0b4b63e647 Drop the rebalancer before creating new UDFs 2019-10-03 16:08:58 +02:00
Marco Slot fd5c2409a5
Merge pull request #3071 from citusdata/check_command_type 
Check command type in TryToDelegateFunctionCall
 2019-10-03 15:49:02 +02:00
Marco Slot 2e50306cf8 Check command type in TryToDelegateFunctionCall 2019-10-03 15:37:15 +02:00
Jelte Fennema 9833c07070 Improve upgrade test runner 
- Update certifi in regress pipenv
- Use normal test ports for upgrade tests
- Make diff behave correctly for upgrade tests
 2019-10-03 13:10:11 +02:00
Halil Ozan Akgül e32592221a
Merge pull request #3060 from citusdata/mx_isolation_dis2ref_foreign_key 
MX Isolation Tests for Foreign keys from Distributrion to Reference tables vs DML
 2019-10-03 09:52:02 +03:00
Halil Ozan Akgul bda8f6f87b Created tests for distribution to reference table foreign keys on mx 2019-10-03 09:31:13 +03:00
SaitTalhaNisanci 19bdca14d8
Add jobs to run tests with pg 12 (#3033) 
* Add PG12 test outputs

* Add jobs to run tests with pg 12

* use POSIX collate for compatibility between pg10/pg11/pg12

* do not override the new default value when running vanilla tests

* fix 2 problems with pg12 tests

* update pg12 images with pg12 rc1

* remove pg10 jobs

* Revert "Add PG12 test outputs"

This reverts commit f3545b92ef.

* change images to use latest instead of dev

* add missing coverage flags
 2019-10-02 15:33:12 +03:00
Halil Ozan Akgül 3f83b726e0
Merge pull request #3044 from citusdata/mx_ref_isolation_update_delete_upsert 
MX Isolation Tests for Update, Delete, Upsert on Reference Tables
 2019-10-02 10:27:47 +03:00
Halil Ozan Akgul e5906bead2 Created isolation tests for update, delete, upsert on reference tables with MX. 2019-10-02 10:11:21 +03:00
Hanefi Onaldi bd416ef68f Fix empty FROM clauses in PG12 2019-10-01 19:54:11 +00:00
Halil Ozan Akgül 9fa4a20148
Merge pull request #3045 from citusdata/mx_ref_isolation_select_for_update 
MX Isolation Tests for SELECT FOR UPDATE on Reference Tables
 2019-10-01 16:48:56 +03:00
Halil Ozan Akgul 1d7030a651 Created isolation tests for select for update on reference tables with MX. 2019-10-01 16:29:15 +03:00
Jelte Fennema ec4a165eec Improve isolation test block detection (#3055) 2019-10-01 14:10:15 +02:00
Jelte Fennema 40f785e6d8 Move citus_isolation_test_session_is_blocked to separate udf sql file 2019-10-01 14:10:15 +02:00
Philip Dubé 89d35e9692 Attempt to force custom plans for prepared statements when trying to delegate function calls 
We discern between PARAM_EXEC & PARAM_EXTERN:
d52eaa0948/src/include/nodes/primnodes.h (L211)
According to primnodes.h we should only run into PARAM_EXEC or PARAM_EXTERN
 2019-09-30 23:49:14 +00:00
Philip Dubé 29f1ea079b PG_VERSION_NUM > 110000 should be PG_VERSION_NUM >= 110000 
Also fix a > 12000 typo
 2019-09-30 23:37:43 +00:00
Hadi Moshayedi 1989ff85b5
Merge pull request #3059 from citusdata/cte_function_calls 
Don't push down queries when in subqueries/ctes
 2019-09-30 15:56:24 -07:00
Hadi Moshayedi 5e97e5c98e Don't push down queries when in subqueries/ctes 2019-09-30 14:22:05 -07:00
Marco Slot 2d1639ad9d
Merge pull request #3052 from citusdata/fix/internal_connection_caching 
Avoid caching connections from backends that service internal connections
 2019-09-30 17:07:03 +02:00
Nils Dijk 01b26cf91a
Disallow distributed functions for functions depending on an extension (#3049) 
DESCRIPTION: Disallow distributed functions for functions depending on an extension

Functions depending on an extension cannot (yet) be distributed by citus. If we would allow this it would cause issues with our dependency following mechanism as we stop following objects depending on an extension.

By not allowing functions to be distributed when they depend on an extension as well as not allowing to make distributed functions depend on an extension we won't break the ability to add new nodes. Allowing functions depending on extensions to be distributed at the moment could cause problems in that area.
 2019-09-30 15:19:47 +02:00
Jelte Fennema 4d991f281c
Update CircleCI codecov orb (#3050) 
Hopefully this fixes the long runtime for coverage uploads that we sometimes have (> 1 minute).
 2019-09-30 15:04:49 +02:00
Nils Dijk 473cbc0115
Propagate CREATE OR REPLACE FUNCTION to workers for distributed functions (#3043) 
DESCRIPTION: Propagate CREATE OR REPLACE FUNCTION

Distributed functions could be replaced, which should be propagated to the workers to keep the function in sync between all nodes.

Due to the complexity of deparsing the `CreateFunctionStmt` we actually produce the plan during the processing phase of our utilityhook. Since the changes have already been made in the catalog tables we can reuse `pg_get_functiondef` to get us the generated `CREATE OR REPLACE` sql.
 2019-09-30 12:41:17 +02:00
Jelte Fennema 82ec918b29
Add explain summary support (#3046) 
Fixes #2922 and also adds explain analyze regression tests
 2019-09-30 10:58:49 +02:00
Marco Slot 35bef0f3db Avoid caching connections from backends that servicei internal connections 2019-09-28 08:32:10 +02:00
Nils Dijk 9c2c50d875
Hookup function/procedure deparsing to our utility hook (#3041) 
DESCRIPTION: Propagate ALTER FUNCTION statements for distributed functions

Using the implemented deparser for function statements to propagate changes to both functions and procedures that are previously distributed.
 2019-09-27 22:06:49 +02:00
Philip Dubé 363409a0c2 Propagate REINDEX TABLE & REINDEX INDEX 2019-09-27 18:14:53 +00:00
Hanefi Onaldi 66b9f2e887 Deparsing and qualifiying for FUNCTION/PROCEDURE statements (#3014) 
This PR aims to add all the necessary logic to qualify and deparse all possible `{ALTER|DROP} .. {FUNCTION|PROCEDURE}` queries.

As Procedures are introduced in PG11, the code contains many PG version checks. I tried my best to make it easy to clean up once we drop PG10 support.


Here are some caveats:
- I assumed that the parse tree is a valid one. There are some queries that are not allowed, but still are parsed successfully by postgres planner. Such queries will result in errors in execution time. (e.g. `ALTER PROCEDURE p STRICT` -> `STRICT` action is valid for functions but not procedures. Postgres decides to parse them nevertheless.)
 2019-09-27 19:02:52 +02:00
Hadi Moshayedi 28acab9d02
Merge pull request #3021 from citusdata/distribute-select 
Distribute select function
 2019-09-27 09:35:24 -07:00
Marco Slot 2868e02a3d Implement SELECT function call delegation. 
When a function is marked as colocated with a distributed table,
we try delegating queries of kind "SELECT func(...)" to workers.

We currently only support this simple form, and don't delegate
forms like "SELECT f1(...), f2(...)", "SELECT f1(...) FROM ...",
or function calls inside transactions.

As a side effect, we also fix the transactional semantics of DO blocks.
Previously we didn't consider a DO block a multi-statement transaction.
Now we do.

Co-authored-by: Marco Slot <marco@citusdata.com>
Co-authored-by: serprex <serprex@users.noreply.github.com>
Co-authored-by: pykello <hadi.moshayedi@microsoft.com>
 2019-09-27 09:13:25 -07:00
Jelte Fennema dab16be283
Set default threshold on get_rebalance_table_shards_plan to 0, like rebalance_table_shards (#3039) 
In this PR the default `threshold` of `rebalance_table_shards` was set to 0: https://github.com/citusdata/shard_rebalancer/pull/73
However, the default for get_rebalance_table_shards_plan was not updated. This
can cause the confusing situation where the actual steps run by
`rebalance_table_shards` are not the same as the ones returned by
`get_rebalance_table_shards_plan`.
 2019-09-27 17:21:36 +02:00
Jelte Fennema 6adf64efdb
Hotfix for circleci problem where directory is owned by root (#3048) 2019-09-27 17:07:30 +02:00
Halil Ozan Akgül bbe3ec0493
Merge pull request #3015 from citusdata/mx_isolation_test_insert_select 
MX Isolation tests for Insert Select
 2019-09-26 18:15:22 +03:00
Halil Ozan Akgul 824a69587c Created isolation tests for insert select on MX 2019-09-26 17:40:36 +03:00