Commit Graph

3827 Commits (1a7ccac6efa19d1b6e1c019898d95da3fed6e12a)

Author SHA1 Message Date
Jelte Fennema c7aa6eddf3
Fix some bugs in string to int functions (#3602)
This fixes 3 bugs:
1. `strtoul` never underflows, so that branch was useless
2. `strtoul` has ULONG_MAX instead of LONG_MAX when it overflows
3. `long` and `unsigned long` are not necessarily 64bit, they can be
    either more or less. So now `strtoll` and `strtoull` are used 
    and 64 bit bounds are checked.
2020-03-11 23:03:02 +01:00
Jelte Fennema c4cc26ed37
Semmle: Ensure stack memory is not leaked through uninitialized… (#3561)
New stack memory can contain anything including passwords/private keys.
In these functions we return structs that can have their padding
bytes uninitialized. By first zeroing out the struct fully, we try to
ensure that any data that is in these padding bytes is at least
overwritten once. It might not be zero anymore after setting the fields,
but at least it shouldn't be private data anymore.
2020-03-11 20:05:36 +01:00
Philip Dubé 7eb678f0f7
Merge pull request #3600 from citusdata/typecheck-agg-combine
Add runtime type checking to AGGREGATE_CUSTOM_COMBINE helper functions
2020-03-11 17:31:17 +00:00
Philip Dubé 11b968bc30 Add runtime type checking to AGGREGATE_CUSTOM_COMBINE helper functions 2020-03-11 17:20:30 +00:00
Jelte Fennema e0bbe1ca38
Semmle: Actively check one possible NULL deref case (#3560)
Calling ErrorIfUnsupportedConstraint was still giving errors on Semmle. This
makes sure that we check for NULL at runtime. This way we can safely ignore all
errors created by this function.
2020-03-11 18:11:56 +01:00
Philip Dubé 468319e638
Merge pull request #3608 from citusdata/fix-non-pushdownable-agg-in-having
Also check aggregates in havingQual when scanning for non pushdownable aggregates
2020-03-11 15:55:12 +00:00
Philip Dubé 4b68ee12c6 Also check aggregates in havingQual when scanning for non pushdownable aggregates
Came across this while coming up with test cases,
'result "68_1" does not exist' I'll seek to address in a future PR,
for now avoid segfault
2020-03-11 15:47:04 +00:00
Önder Kalacı 63ced3d901
Improve master evaluation tests (#3609)
* Add third column to master_evaluation_modify table

It was already added in some tests, but now make it globally
applicable to the test file.

* Add third column to master_evaluation_select table

As we'll use the column in some tests

* Add modify regression tests

For the combinations of: local/remote, router/fast-path:
   - Distribution key is a const.
   - Contains a function
   - A column which is not dist. key is parametrized

* Add select regression tests

    For the combinations of: local/remote, router/fast-path:
       - Distribution key is a const.
       - Contains a function
       - A column which is not dist. key is parametrized

* Make some tests consistent to check-base
2020-03-11 15:38:08 +01:00
Önder Kalacı afc942c6af
Remove non-adaptive test schedules (#3605)
As we don't have any other executors to run them.

These schedules were added when we had both the adaptive executor and
the real-time/router executors in the code. Since we only have adaptive
executor anymore, we can remove these.
2020-03-11 09:58:49 +01:00
Önder Kalacı f7f0fff304
Merge pull request #3604 from citusdata/prevent_worker_mx_create_dist_f
Prevent create_distributed_function() from the workers
2020-03-11 09:47:41 +01:00
Onder Kalaci 7d787e3d5e Prevent create_distributed_function() from the workers
As this could cause weird edge cases.
2020-03-10 18:24:20 +01:00
Onur Tirtir e902581cb6
implement DropTaskList before introducing local DROP table execution (#3603) 2020-03-10 19:12:44 +03:00
Marco Slot c26f99ea82
Merge pull request #3584 from citusdata/simplify_insert_logic
Simplify INSERT logic in router planner
2020-03-10 16:45:38 +01:00
Marco Slot cb3d90bdc8 Simplify INSERT logic in router planner 2020-03-10 15:54:40 +01:00
Philip Dubé d0d51bb8c3
Merge pull request #3601 from citusdata/maintenanced-dont-proc-exit-from-term-handler
maintenanced: Don't call proc_exit in SIGTERM handler
2020-03-10 13:50:48 +00:00
Philip Dubé 2b4ea33a2b maintenanced: Don't call proc_exit in SIGTERM handler
Instead set got_SIGTERM to true to signal mainloop to exit
2020-03-09 23:22:19 +00:00
Philip Dubé 877687cc64
Merge pull request #3568 from citusdata/fix-having-subquery-ref
First phase of addressing HAVING subquery issues
2020-03-09 18:23:02 +00:00
Philip Dubé 81cfa05d3d First phase of addressing HAVING subquery issues
Add failing tests, make changes to avoid crashes at least

Fix HAVING subquery pushdown ignoring reference table only subqueries,
also include HAVING in recursive planning

Given that we have a function IsDistributedTable which includes reference tables,
it seems best to have IsDistributedTableRTE & QueryContainsDistributedTableRTE
reflect that they do not include reference tables in their check

Similarly SublinkList's name should reflect that it only scans WHERE

contain_agg_clause asserts that we don't have SubLinks,
use contain_aggs_of_level as suggested by pg sourcecode
2020-03-09 17:58:30 +00:00
Önder Kalacı 7793d19b71
Merge pull request #3578 from citusdata/fix_wrong_left_join
Improve definition of RelationInfoContainsOnlyRecurringTuples
2020-03-09 17:42:51 +01:00
Onder Kalaci 2ed19181fe Improve definition of RelationInfoContainsOnlyRecurringTuples
Before this commit, we considered !ContainsRecurringRTE() enough
for NotContainsOnlyRecurringTuples. However, instead, we can check
for existince of any distributed table.

DESCRIPTION: Fixes a bug that causes wrong results with complex outer joins
2020-03-09 17:28:33 +01:00
SaitTalhaNisanci 321d0152c1
add a utility to get shard oid from relation oid and shard id (#3596) 2020-03-09 15:50:29 +03:00
SaitTalhaNisanci 4509d9a72b
Create a variable SLOW_START_DISABLED (#3593)
When ExecutorSlowStartInterval is set to 0, it has a special meaning
that we do not want to use slow start. Therefore, in the code we have
checks such as ExecutorSlowStartInterval > 0 to understand if it is
enabled or not. However, this is kind of subtle, and it creates an extra
mapping in our mind. Therefore, I thought that using a variable for the
special value removes the mapping and makes it easier to understand.
2020-03-09 14:54:01 +03:00
Hanefi Onaldi 2595b4864b
Remove all GetWorkerNodeCount() references
As @onderkalaci suggested removing the definition of GetWorkerNodeCount() that can potentially cause misunderstandings.

I can advise using ActiveReadableWorkerNodeCount() that returns the number of active primaries is a safer alternative than GetWorkerNodeCount() that returns the total number of workers containing inactives, primaries, and unavailable nodes. I introduced a bug #3556 and in the bugfix #3564 removed the single usage of said function
2020-03-09 13:35:18 +03:00
Philip Dubé 426b8ff1a9
Merge pull request #3592 from citusdata/rename-lookuplookup
Rename LookupCitusTableCacheEntry to GetCitusTableCacheEntry, LookupLookupCitusTableCacheEntry back to LookupCitusTableCacheEntry
2020-03-08 14:20:24 +00:00
Philip Dubé 7cdfa1daab Rename LookupCitusTableCacheEntry to GetCitusTableCacheEntry, LookupLookupCitusTableCacheEntry back to LookupCitusTableCacheEntry 2020-03-08 14:08:23 +00:00
Philip Dubé 70436ec279
Merge pull request #3587 from citusdata/fix-typos
Fix typos, rename isDistributedRelation to isCitusRelation
2020-03-07 14:21:03 +00:00
Philip Dubé a7cca1bcde Rename DistTableCacheEntry to CitusTableCacheEntry 2020-03-07 14:08:03 +00:00
Philip Dubé b514ab0f55 Fix typos, rename isDistributedRelation to isCitusRelation 2020-03-06 19:20:34 +00:00
Philip Dubé 00a7bc3044
Merge pull request #3586 from citusdata/rename-distributed-to-citus
Try to create clear distinction between DistributedTable vs CitusTable
2020-03-06 19:10:16 +00:00
Philip Dubé bec58000d6 Given IsDistributedTableRTE, there's ambiguity in what DistributedTable means
Elsewhere we used DistributedTable to include reference tables
Marco suggested we use CitusTable for distributed & reference tables

So renaming:
- IsDistributedTable -> IsCitusTable
- IsDistributedTableViaCatalog -> IsCitusTableViaCatalog
- DistributedTableCacheEntry -> CitusTableCacheEntry
- DistributedTableList -> CitusTableList
- isDistributedTable -> isCitusTable
- InsertSelectIntoDistributedTable -> InsertSelectIntoCitusTable
- ExtractFirstDistributedTableId -> ExtractFirstCitusTableId
2020-03-06 18:57:55 +00:00
Onur Tirtir a381074787
Update CHANGELOG for 9.0.2 (#3585)
(cherry picked from commit de6068b2c4)

Co-authored-by: Hanefi Onaldi <hanefionaldi@gmail.com>
2020-03-06 18:26:04 +03:00
Marco Slot fa29fb8c52
Merge pull request #3579 from citusdata/disable_postgres_parallelism
Disable Postgres parallelism by default in tests
2020-03-06 15:54:57 +01:00
Onur Tirtir 50e59f1a61
Update CHANGELOG for 9.2.2 (#3582)
Co-authored-by: Hanefi Onaldi <hanefionaldi@gmail.com>
2020-03-06 16:09:34 +03:00
Marco Slot 5b1d1dd413 Remove unnecessary use of max_parallel_workers_per_gather 2020-03-06 13:18:58 +01:00
Marco Slot d0fead6691 Disable Postgres parallelism by default in tests 2020-03-06 13:18:58 +01:00
Onur Tirtir c5007bc93c
Merge pull request #3563 from citusdata/refactor/local-group-id-and-fkey
Refactor around foreign key constraints and GetLocalGroupId
2020-03-05 20:32:10 +03:00
Onur Tirtir bdce9acc30 some refactor around foreign key constraints 2020-03-05 20:20:41 +03:00
Onur Tirtir 88bfd2e4b7 refactor around local group id checks
Mostyl optimizes the calls made to GetLocalGroupId and refactors
its usages
2020-03-05 20:20:41 +03:00
Onur Tirtir 1e128a6ee4 fix a potential infinite loop 2020-03-05 20:20:41 +03:00
SaitTalhaNisanci a75436a54b
refactor CoordinatedTransactionCallback (#3571) 2020-03-05 18:36:12 +03:00
Hanefi Onaldi 6e6763678c
Merge pull request #3564 from citusdata/fix-early-exits-on-subplan-pruning
Fix early exits on intermediate result pruning

There are 2 problems with our early exit strategy that this commit fixes:

1- When we decide that a subplan results are sent to all worker nodes,
we used to skip traversing the whole distributed plan, instead of
skipping only the subplan.

2- We used to consider all available nodes in the cluster (secondaries
and inactive nodes as well as active primaries) when deciding on early
exit strategy. This resulted in failures to early exit when there are
secondaries or inactive nodes.
2020-03-05 16:51:15 +03:00
Hanefi Onaldi c0ad44f975
Fix early exit bug on intermediate result pruning
There are 2 problems with our early exit strategy that this commit fixes:

1- When we decide that a subplan results are sent to all worker nodes,
we used to skip traversing the whole distributed plan, instead of
skipping only the subplan.

2- We used to consider all available nodes in the cluster (secondaries
and inactive nodes as well as active primaries) when deciding on early
exit strategy. This resulted in failures to early exit when there are
secondaries or inactive nodes.
2020-03-05 16:41:44 +03:00
Marco Slot 241c186603
Merge pull request #3553 from citusdata/refactor/begin_scan
Refactor CitusBeginScan into separate SELECT / DML paths
2020-03-05 13:06:10 +01:00
Onder Kalaci f72916875f Expand test coverage for combinations of master evalution, deferred pruning, parameters, local execution
- Router           & Remote & Requires Master Evaluation & With Param & Without Param
- Fast Path Router & Remote & Requires Master Evaluation & With Param & Without Param
2020-03-05 12:37:22 +01:00
Marco Slot dc4c0c032e Refactor CitusBeginScan into separate DML / SELECT paths 2020-03-05 12:37:22 +01:00
Nils Dijk 268ad741a9
Refactor the deparsing of a CREATE EXTENSION to prevent NULL POINTER dereferences (#3518)
DESCRIPTION: satisfy static analysis tool for a nullptr dereference

During the static analysis project on the codebase this code has been flagged as having the potential for a null pointer dereference. Funnily enough the author had already made a comment of it in the code this was not possible due to us setting the schema name before we pass in the statement. If we want to reuse this code in a later setting this comment might not always apply and we could actually run into null pointer dereference.

This patch changes a bit of the code around to first of all make sure there is no NULL pointer dereference in this code anymore.
Secondly we allow for better deparsing by setting and adhering to the `if_not_exists` flag on the statement.
And finally add support for all syntax described in the documentation of postgres (FROM was missing).
2020-03-04 16:47:07 +01:00
Önder Kalacı 9096c650f6
Merge pull request #3562 from citusdata/add_type_to_deparse
For composite types, add cast to the parameter to ease remote node detect the type
2020-03-04 13:12:05 +01:00
Marco Slot 27f23d2c89 Add some distribution column = composite type prepared statement tests 2020-03-04 05:01:43 +01:00
Onder Kalaci 087f6eb4c0 For composite types, add cast to the parameter to ease remote node detect
the type.
2020-03-04 11:27:45 +01:00
Onur Tirtir c9c6e58c53
Merge pull request #3554 from citusdata/refactor/vacuum-and-local-executor
Refactor vacuumTaskList function and local_executor.c line lengths
2020-03-02 12:04:55 +03:00