Commit Graph

2121 Commits (4b36938f2823b1b360cbae87ee55a29e64109349)

Author SHA1 Message Date
Onur Tirtir 4b36938f28 update existing tests 2020-03-11 13:04:51 +03:00
Onur Tirtir 4042d21714 enable local execution for DROP command on distributed & reference tables 2020-03-11 13:04:51 +03:00
Onur Tirtir 26cec16b16 enable local execution for DDL commands on distributed & reference tables 2020-03-11 13:04:51 +03:00
Onur Tirtir c162755cc5 enable local execution for TRUNCATE command on distributed & reference tables 2020-03-11 13:04:50 +03:00
Onur Tirtir 38a59b4767 change ErrorIfTransactionAccessedPlacementsLocally logic & usages 2020-03-11 13:04:50 +03:00
Onur Tirtir 82f3157750 introduce functions for local execution of utility commands 2020-03-11 13:04:50 +03:00
Onur Tirtir ed66517e95 reimplement ExecuteUtilityTaskListWithoutResults for local execution and refactor its usages 2020-03-11 13:04:50 +03:00
Önder Kalacı afc942c6af
Remove non-adaptive test schedules (#3605)
As we don't have any other executors to run them.

These schedules were added when we had both the adaptive executor and
the real-time/router executors in the code. Since we only have adaptive
executor anymore, we can remove these.
2020-03-11 09:58:49 +01:00
Onder Kalaci 7d787e3d5e Prevent create_distributed_function() from the workers
As this could cause weird edge cases.
2020-03-10 18:24:20 +01:00
Onur Tirtir e902581cb6
implement DropTaskList before introducing local DROP table execution (#3603) 2020-03-10 19:12:44 +03:00
Marco Slot cb3d90bdc8 Simplify INSERT logic in router planner 2020-03-10 15:54:40 +01:00
Philip Dubé 2b4ea33a2b maintenanced: Don't call proc_exit in SIGTERM handler
Instead set got_SIGTERM to true to signal mainloop to exit
2020-03-09 23:22:19 +00:00
Philip Dubé 81cfa05d3d First phase of addressing HAVING subquery issues
Add failing tests, make changes to avoid crashes at least

Fix HAVING subquery pushdown ignoring reference table only subqueries,
also include HAVING in recursive planning

Given that we have a function IsDistributedTable which includes reference tables,
it seems best to have IsDistributedTableRTE & QueryContainsDistributedTableRTE
reflect that they do not include reference tables in their check

Similarly SublinkList's name should reflect that it only scans WHERE

contain_agg_clause asserts that we don't have SubLinks,
use contain_aggs_of_level as suggested by pg sourcecode
2020-03-09 17:58:30 +00:00
Onder Kalaci 2ed19181fe Improve definition of RelationInfoContainsOnlyRecurringTuples
Before this commit, we considered !ContainsRecurringRTE() enough
for NotContainsOnlyRecurringTuples. However, instead, we can check
for existince of any distributed table.

DESCRIPTION: Fixes a bug that causes wrong results with complex outer joins
2020-03-09 17:28:33 +01:00
SaitTalhaNisanci 321d0152c1
add a utility to get shard oid from relation oid and shard id (#3596) 2020-03-09 15:50:29 +03:00
SaitTalhaNisanci 4509d9a72b
Create a variable SLOW_START_DISABLED (#3593)
When ExecutorSlowStartInterval is set to 0, it has a special meaning
that we do not want to use slow start. Therefore, in the code we have
checks such as ExecutorSlowStartInterval > 0 to understand if it is
enabled or not. However, this is kind of subtle, and it creates an extra
mapping in our mind. Therefore, I thought that using a variable for the
special value removes the mapping and makes it easier to understand.
2020-03-09 14:54:01 +03:00
Hanefi Onaldi 2595b4864b
Remove all GetWorkerNodeCount() references
As @onderkalaci suggested removing the definition of GetWorkerNodeCount() that can potentially cause misunderstandings.

I can advise using ActiveReadableWorkerNodeCount() that returns the number of active primaries is a safer alternative than GetWorkerNodeCount() that returns the total number of workers containing inactives, primaries, and unavailable nodes. I introduced a bug #3556 and in the bugfix #3564 removed the single usage of said function
2020-03-09 13:35:18 +03:00
Philip Dubé 7cdfa1daab Rename LookupCitusTableCacheEntry to GetCitusTableCacheEntry, LookupLookupCitusTableCacheEntry back to LookupCitusTableCacheEntry 2020-03-08 14:08:23 +00:00
Philip Dubé a7cca1bcde Rename DistTableCacheEntry to CitusTableCacheEntry 2020-03-07 14:08:03 +00:00
Philip Dubé b514ab0f55 Fix typos, rename isDistributedRelation to isCitusRelation 2020-03-06 19:20:34 +00:00
Philip Dubé bec58000d6 Given IsDistributedTableRTE, there's ambiguity in what DistributedTable means
Elsewhere we used DistributedTable to include reference tables
Marco suggested we use CitusTable for distributed & reference tables

So renaming:
- IsDistributedTable -> IsCitusTable
- IsDistributedTableViaCatalog -> IsCitusTableViaCatalog
- DistributedTableCacheEntry -> CitusTableCacheEntry
- DistributedTableList -> CitusTableList
- isDistributedTable -> isCitusTable
- InsertSelectIntoDistributedTable -> InsertSelectIntoCitusTable
- ExtractFirstDistributedTableId -> ExtractFirstCitusTableId
2020-03-06 18:57:55 +00:00
Marco Slot 5b1d1dd413 Remove unnecessary use of max_parallel_workers_per_gather 2020-03-06 13:18:58 +01:00
Marco Slot d0fead6691 Disable Postgres parallelism by default in tests 2020-03-06 13:18:58 +01:00
Onur Tirtir bdce9acc30 some refactor around foreign key constraints 2020-03-05 20:20:41 +03:00
Onur Tirtir 88bfd2e4b7 refactor around local group id checks
Mostyl optimizes the calls made to GetLocalGroupId and refactors
its usages
2020-03-05 20:20:41 +03:00
Onur Tirtir 1e128a6ee4 fix a potential infinite loop 2020-03-05 20:20:41 +03:00
SaitTalhaNisanci a75436a54b
refactor CoordinatedTransactionCallback (#3571) 2020-03-05 18:36:12 +03:00
Hanefi Onaldi c0ad44f975
Fix early exit bug on intermediate result pruning
There are 2 problems with our early exit strategy that this commit fixes:

1- When we decide that a subplan results are sent to all worker nodes,
we used to skip traversing the whole distributed plan, instead of
skipping only the subplan.

2- We used to consider all available nodes in the cluster (secondaries
and inactive nodes as well as active primaries) when deciding on early
exit strategy. This resulted in failures to early exit when there are
secondaries or inactive nodes.
2020-03-05 16:41:44 +03:00
Onder Kalaci f72916875f Expand test coverage for combinations of master evalution, deferred pruning, parameters, local execution
- Router           & Remote & Requires Master Evaluation & With Param & Without Param
- Fast Path Router & Remote & Requires Master Evaluation & With Param & Without Param
2020-03-05 12:37:22 +01:00
Marco Slot dc4c0c032e Refactor CitusBeginScan into separate DML / SELECT paths 2020-03-05 12:37:22 +01:00
Nils Dijk 268ad741a9
Refactor the deparsing of a CREATE EXTENSION to prevent NULL POINTER dereferences (#3518)
DESCRIPTION: satisfy static analysis tool for a nullptr dereference

During the static analysis project on the codebase this code has been flagged as having the potential for a null pointer dereference. Funnily enough the author had already made a comment of it in the code this was not possible due to us setting the schema name before we pass in the statement. If we want to reuse this code in a later setting this comment might not always apply and we could actually run into null pointer dereference.

This patch changes a bit of the code around to first of all make sure there is no NULL pointer dereference in this code anymore.
Secondly we allow for better deparsing by setting and adhering to the `if_not_exists` flag on the statement.
And finally add support for all syntax described in the documentation of postgres (FROM was missing).
2020-03-04 16:47:07 +01:00
Marco Slot 27f23d2c89 Add some distribution column = composite type prepared statement tests 2020-03-04 05:01:43 +01:00
Onder Kalaci 087f6eb4c0 For composite types, add cast to the parameter to ease remote node detect
the type.
2020-03-04 11:27:45 +01:00
Onur Tirtir ff9c9d1808 make VacuumTaskList even with other taskList functions and some safety changes
Makees VacuumTaskList function even with other TaskList creator functions.
Also, previously we were generating per-shard vacuum command strings via
unconventional usage of StringInfo struct (setting the stringInfo->len field
manually) which could cause unexepected memory errors (that I cannot foresee now).
2020-03-02 10:25:28 +03:00
Onur Tirtir cf718ffe77 safely error out in DistributedTableCacheEntry function 2020-03-02 10:25:12 +03:00
Onur Tirtir 17d9b934c3 refactor local_executor.c lines with >78 characters 2020-02-29 15:04:34 +03:00
Philip Dubé 34f241af16 Fix create_distributed_table on a table using GENERATED ALWAYS AS
If the generated column does not come at the end of the column list,
columnNameList doesn't line up with the column indexes. Seek past

CREATE TABLE test_table (
    test_id int PRIMARY KEY,
    gen_n int GENERATED ALWAYS AS (1) STORED,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
SELECT create_distributed_table('test_table', 'test_id');

Would raise ERROR: cannot cast 23 to 1184
2020-02-28 09:34:26 -08:00
Philip Dubé 2fae132e45
repartition_join_execution: Don't store 64 bit integers as poin… (#3551)
Pointers are not necessarily 64bit
2020-02-28 15:06:06 +01:00
Philip Dubé 20abc4d2b5
Replace foreach with foreach_ptr/foreach_oid (#3544) 2020-02-27 16:54:49 +01:00
Jelte Fennema c48f0ca7e5 Make bad refactors to foreach_xxx error out
Without this commit you could still use varCell in the body of loop.
This makes it easy for bad refactors that still use the ListCell to slip
through unnoticed, because the new ListCell will be named the same as the
one used in the old code. By renaming the ListCell to varCellDoNotUse
this will not happen.
2020-02-27 10:59:45 +01:00
Jelte Fennema 685b54b3de
Semmle: Check for NULL in some places where it might occur (#3509)
Semmle reported quite some places where we use a value that could be NULL. Most of these are not actually a real issue, but better to be on the safe side with these things and make the static analysis happy.
2020-02-27 10:45:29 +01:00
Jelte Fennema eb8e099f09 Fix Makefile so that it builds safestringlib correctly on OSX 2020-02-26 17:44:44 +01:00
Jelte Fennema 8e7eaaf949 Add clean-full to also clean full builds of vendored libraries 2020-02-26 17:44:44 +01:00
Hadi Moshayedi e7cce40e6e Address pykello's feedback 2020-02-26 07:17:32 -08:00
Hadi Moshayedi 1b3e58f0c3 Merge branch 'improve-shard-pruning' of https://github.com/MarkusSintonen/citus into MarkusSintonen-improve-shard-pruning 2020-02-26 07:13:33 -08:00
SaitTalhaNisanci 82d22b34fe
create temp schemas in parallel (#3540) 2020-02-26 16:20:08 +03:00
SaitTalhaNisanci d94c3fd43d
send repartition cleanup jobs in parallel to all workers (#3485)
* send repartition cleanup jobs in parallel to all workers

* add review items
2020-02-26 13:44:06 +03:00
Marco Slot c7f123947e Make merge tables during re-partitioning unlogged 2020-02-26 10:46:07 +01:00
Jelte Fennema 62bf571ced Make SafeSnprintf work on PG11 2020-02-25 15:39:27 +01:00
Jelte Fennema 7d24cebc80 Add pg11 snprintf file to repo for use in pg11 when it's not compiled 2020-02-25 15:39:27 +01:00