external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,819 Commits
636 Branches
162 Tags
98 MiB
Commit Graph

1819 Commits (42a80827217291ab182eb41f7f166fe7b47ec40b)

Author SHA1 Message Date
Önder Kalacı 54ccfb24be
Merge pull request #1876 from citusdata/subqueries 
Recursively plan subqueries that are not safe to pushdown (i.e., requires merge step)
 2017-12-21 09:58:35 +03:00
Onder Kalaci e2a5124830 Add regression tests for recursive subquery planning 2017-12-21 08:37:40 +02:00
Onder Kalaci 0d5a4b9c72 Recursively plan subqueries that are not safe to pushdown 
With this commit, Citus recursively plans subqueries that
are not safe to pushdown, in other words, requires a merge
step.

The algorithm is simple: Recursively traverse the query from bottom
up (i.e., bottom meaning the leaf queries). On each level, check
whether the query is safe to pushdown (or a single repartition
subquery). If the answer is yes, do not touch that subquery. If the
answer is no, plan the subquery seperately (i.e., create a subPlan
for it) and replace the subquery with a call to
`read_intermediate_results(planId, subPlanId)`. During the the
execution, run the subPlans first, and make them avaliable to the
next query executions.

Some of the queries hat this change allows us:

   * Subqueries with LIMIT
   * Subqueries with GROUP BY/DISTINCT on non-partition keys
   * Subqueries involving re-partition joins, router queries
   * Mixed usage of subqueries and CTEs (i.e., use CTEs in
     subqueries as well). Nested subqueries as long as we
     support the subquery inside the nested subquery.
   * Subqueries with local tables (i.e., those subqueries
     has the limitation that they have to be leaf subqueries)

   * VIEWs on the distributed tables just works (i.e., the
     limitations mentioned below still applies to views)

Some of the queries that is still NOT supported:

  * Corrolated subqueries that are not safe to pushdown
  * Window function on non-partition keys
  * Recursively planned subqueries or CTEs on the outer
    side of an outer join
  * Only recursively planned subqueries and CTEs in the FROM
    (i.e., not any distributed tables in the FROM) and subqueries
    in WHERE clause
  * Subquery joins that are not on the partition columns (i.e., each
    subquery is individually joined on partition keys but not the upper
    level subquery.)
  * Any limitation that logical planner applies such as aggregate
    distincts (except for count) when GROUP BY is on non-partition key,
    or array_agg with ORDER BY
 2017-12-21 08:37:40 +02:00
Onder Kalaci e12ea914b9 Refactor ErrorIfQueryNotSupported to defer errors 2017-12-20 09:03:49 +02:00
Onder Kalaci 71ce42b936 Refactor RecursivelyPlanSubqueriesAndCTEs() to make it ready 
to work with subqueries
 2017-12-20 09:03:47 +02:00
Marco Slot 393e625cb2
Merge pull request #1888 from citusdata/subplan_explain 
Show distributed subplan ID in EXPLAIN output
 2017-12-19 22:13:11 +03:00
Marco Slot 6a6e986c2b Add EXPLAIN regression test with subplans 2017-12-19 16:34:56 +01:00
Marco Slot 5e0539efa3 Plan CTEs when subquery pushdown is on 2017-12-19 16:34:56 +01:00
Marco Slot 44a1ea631a Show distributed subplan ID in EXPLAIN output 2017-12-19 16:34:56 +01:00
Marco Slot 7a25ebe257
Merge pull request #1889 from citusdata/fix_my_backend_data 
Do not reinitialise MyBackendData on cache invalidations
 2017-12-19 18:09:38 +03:00
Marco Slot 35dbacdb69 Do not reinitialise MyBackendData 2017-12-19 15:56:26 +01:00
Marco Slot 5c5bd80afc
Merge pull request #1879 from citusdata/result_in_parallel_worker 
Allow intermediate results to be used in parallel workers
 2017-12-19 11:45:48 +03:00
Marco Slot 9b520ae194 Add test for using transaction ID in parallel worker 2017-12-19 09:30:29 +01:00
Marco Slot af201a2f6d Allow intermediate results to be used in parallel workers 2017-12-18 19:05:08 +01:00
Marco Slot 704828b237
Merge pull request #1869 from citusdata/result_cost 
Set cost estimates for read_intermediate_result
 2017-12-18 16:55:28 +01:00
Marco Slot 7dab078e67 Set cost estimates for read_intermediate_result 2017-12-18 16:23:44 +01:00
Marco Slot e49254f876 Revert "Add EXPLAIN regression test with subplans" 
This reverts commit 8b6d641227.
 2017-12-17 22:34:31 +01:00
Marco Slot 74bd33d0cc Revert "Plan CTEs when subquery pushdown is on" 
This reverts commit e3b953b8e3.
 2017-12-17 22:34:20 +01:00
Marco Slot aca5f35ab9 Revert "Show distributed subplan ID in EXPLAIN output" 
This reverts commit 686b079272.
 2017-12-17 22:34:04 +01:00
Marco Slot 8b6d641227 Add EXPLAIN regression test with subplans 2017-12-17 22:00:25 +01:00
Marco Slot e3b953b8e3 Plan CTEs when subquery pushdown is on 2017-12-17 21:49:36 +01:00
Marco Slot 686b079272 Show distributed subplan ID in EXPLAIN output 2017-12-16 11:32:01 +01:00
Marco Slot 36f049bdc5
Merge pull request #1866 from citusdata/count_distinct_subquery 
Allow count(distinct) in queries with a subquery
 2017-12-15 16:05:25 +01:00
Marco Slot ea6b98fda4 Allow count(distinct) in queries with a subquery 2017-12-15 15:24:26 +01:00
Marco Slot fbb7d9c894
Merge pull request #1873 from citusdata/fix_partition_lock 
Do not take extra access exclusive lock on partitioned tables
 2017-12-15 13:32:42 +01:00
Marco Slot 9ee0e68882 Do not take extra access exclusive lock partitioned tables 2017-12-15 13:02:31 +01:00
Marco Slot cf7dda3892
Merge pull request #1871 from citusdata/relax_from_sublink_checks 
Relax checks on recurring tuples in FROM with sublinks
 2017-12-15 12:13:19 +01:00
Marco Slot 5a69fc1b17 Relax checks on recurring tuples in FROM with sublinks 2017-12-15 11:56:06 +01:00
Marco Slot a64f0060ba Reduce the frequency of FinishConnectionIO calls during COPY (#1864) 2017-12-14 13:21:59 -05:00
Marco Slot a811aad264 Deparallelise multi_modifying_xacts tests 2017-12-14 10:27:17 +01:00
Marco Slot c19c3ef4a1
Merge pull request #1853 from citusdata/ctes 
Add support for CTEs in distributed queries
 2017-12-14 10:26:47 +01:00
mehmet furkan şahin 5851f71bfb Add CTE regression tests 2017-12-14 09:32:55 +01:00
Marco Slot fa73abe6d4 Regression test output changes after CTE support 2017-12-14 09:32:55 +01:00
Marco Slot 2e2b4e81fa Add support for CTEs in distributed queries 2017-12-14 09:32:55 +01:00
Marco Slot d0335ec818 Send BEGIN for SELECTs in the router executor 2017-12-14 09:32:55 +01:00
Marco Slot cbbd418af2 Add citus.copy_format OIDs to metadata cache 2017-12-14 09:32:55 +01:00
Marco Slot 66f9f1d6cd Make some intermediate results functions public 2017-12-14 09:32:55 +01:00
Marco Slot 36ee21c323 Make CanUseBinaryCopyFormatForType public 2017-12-14 09:32:55 +01:00
Marco Slot 7d1191954d Add DistributedSubPlan node 2017-12-14 09:32:55 +01:00
Önder Kalacı b5784ca03a
Merge pull request #1852 from citusdata/group_by_on_function 
Treat recurring tuples as reference tables for GROUP BY checks
 2017-12-13 16:37:31 +03:00
Onder Kalaci 86b2d9420c Treat recurring tuples as reference table for GROUP BY checks 
read_intermediate_results() and immutable functions are implemented.
Empty join trees seems not applicable here.
 2017-12-13 14:55:42 +02:00
Marco Slot f0851257fa
Merge pull request #1867 from citusdata/fix_analyze_block 
Fix issue with multiple ANALYZE in transaction block
 2017-12-12 10:54:27 +01:00
Marco Slot d1a470a52e Fix issue with multiple ANALYZE in transaction block 2017-12-12 10:28:48 +01:00
Mehmet Furkan ŞAHİN 84957fe6e7
Merge pull request #1861 from citusdata/new_guc_to_allow_task_executor_swap 
New guc to allow automated task executor swap
 2017-12-11 09:48:42 +03:00
mehmet furkan şahin 3c941aedf1 adds citus.enable_repartition_joins GUC 
The new GUC allows Citus to switch between task executors
when necessary
 2017-12-11 09:36:37 +03:00
Marco Slot 7544e91c87
Merge pull request #1860 from citusdata/needs_distributed_planning 
Allow queries with local tables in NeedsDistributedPlanning
 2017-12-08 10:11:07 +01:00
Marco Slot 5895c88552 Add materialized view regression tests 2017-12-07 16:20:23 +01:00
Marco Slot 60a1e31671 Allow queries with local tables in NeedsDistributedPlanning 2017-12-07 16:20:23 +01:00
Marco Slot d71d519672
Merge pull request #1857 from citusdata/fix_intermediate_result 
Use proper schema in read_intermediate_result signature
 2017-12-07 14:20:10 +01:00
Marco Slot f8550b8c85 Fix issues with read_intermediate_result signature 2017-12-07 13:47:56 +01:00