external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,067 Commits
636 Branches
162 Tags
99 MiB
Commit Graph

2067 Commits (aa16512a815d2ac263cb99a182f4d887f11f7ff3)

Author SHA1 Message Date
Onder Kalaci 0d5a4b9c72 Recursively plan subqueries that are not safe to pushdown 
With this commit, Citus recursively plans subqueries that
are not safe to pushdown, in other words, requires a merge
step.

The algorithm is simple: Recursively traverse the query from bottom
up (i.e., bottom meaning the leaf queries). On each level, check
whether the query is safe to pushdown (or a single repartition
subquery). If the answer is yes, do not touch that subquery. If the
answer is no, plan the subquery seperately (i.e., create a subPlan
for it) and replace the subquery with a call to
`read_intermediate_results(planId, subPlanId)`. During the the
execution, run the subPlans first, and make them avaliable to the
next query executions.

Some of the queries hat this change allows us:

   * Subqueries with LIMIT
   * Subqueries with GROUP BY/DISTINCT on non-partition keys
   * Subqueries involving re-partition joins, router queries
   * Mixed usage of subqueries and CTEs (i.e., use CTEs in
     subqueries as well). Nested subqueries as long as we
     support the subquery inside the nested subquery.
   * Subqueries with local tables (i.e., those subqueries
     has the limitation that they have to be leaf subqueries)

   * VIEWs on the distributed tables just works (i.e., the
     limitations mentioned below still applies to views)

Some of the queries that is still NOT supported:

  * Corrolated subqueries that are not safe to pushdown
  * Window function on non-partition keys
  * Recursively planned subqueries or CTEs on the outer
    side of an outer join
  * Only recursively planned subqueries and CTEs in the FROM
    (i.e., not any distributed tables in the FROM) and subqueries
    in WHERE clause
  * Subquery joins that are not on the partition columns (i.e., each
    subquery is individually joined on partition keys but not the upper
    level subquery.)
  * Any limitation that logical planner applies such as aggregate
    distincts (except for count) when GROUP BY is on non-partition key,
    or array_agg with ORDER BY
 2017-12-21 08:37:40 +02:00
Onder Kalaci e12ea914b9 Refactor ErrorIfQueryNotSupported to defer errors 2017-12-20 09:03:49 +02:00
Onder Kalaci 71ce42b936 Refactor RecursivelyPlanSubqueriesAndCTEs() to make it ready 
to work with subqueries
 2017-12-20 09:03:47 +02:00
Marco Slot 393e625cb2
Merge pull request #1888 from citusdata/subplan_explain 
Show distributed subplan ID in EXPLAIN output
 2017-12-19 22:13:11 +03:00
Marco Slot 6a6e986c2b Add EXPLAIN regression test with subplans 2017-12-19 16:34:56 +01:00
Marco Slot 5e0539efa3 Plan CTEs when subquery pushdown is on 2017-12-19 16:34:56 +01:00
Marco Slot 44a1ea631a Show distributed subplan ID in EXPLAIN output 2017-12-19 16:34:56 +01:00
Marco Slot 7a25ebe257
Merge pull request #1889 from citusdata/fix_my_backend_data 
Do not reinitialise MyBackendData on cache invalidations
 2017-12-19 18:09:38 +03:00
Marco Slot 35dbacdb69 Do not reinitialise MyBackendData 2017-12-19 15:56:26 +01:00
Marco Slot 5c5bd80afc
Merge pull request #1879 from citusdata/result_in_parallel_worker 
Allow intermediate results to be used in parallel workers
 2017-12-19 11:45:48 +03:00
Marco Slot 9b520ae194 Add test for using transaction ID in parallel worker 2017-12-19 09:30:29 +01:00
Marco Slot af201a2f6d Allow intermediate results to be used in parallel workers 2017-12-18 19:05:08 +01:00
Marco Slot 704828b237
Merge pull request #1869 from citusdata/result_cost 
Set cost estimates for read_intermediate_result
 2017-12-18 16:55:28 +01:00
Marco Slot 7dab078e67 Set cost estimates for read_intermediate_result 2017-12-18 16:23:44 +01:00
Marco Slot e49254f876 Revert "Add EXPLAIN regression test with subplans" 
This reverts commit 8b6d641227.
 2017-12-17 22:34:31 +01:00
Marco Slot 74bd33d0cc Revert "Plan CTEs when subquery pushdown is on" 
This reverts commit e3b953b8e3.
 2017-12-17 22:34:20 +01:00
Marco Slot aca5f35ab9 Revert "Show distributed subplan ID in EXPLAIN output" 
This reverts commit 686b079272.
 2017-12-17 22:34:04 +01:00
Marco Slot 8b6d641227 Add EXPLAIN regression test with subplans 2017-12-17 22:00:25 +01:00
Marco Slot e3b953b8e3 Plan CTEs when subquery pushdown is on 2017-12-17 21:49:36 +01:00
Marco Slot 686b079272 Show distributed subplan ID in EXPLAIN output 2017-12-16 11:32:01 +01:00
Marco Slot 36f049bdc5
Merge pull request #1866 from citusdata/count_distinct_subquery 
Allow count(distinct) in queries with a subquery
 2017-12-15 16:05:25 +01:00
Marco Slot ea6b98fda4 Allow count(distinct) in queries with a subquery 2017-12-15 15:24:26 +01:00
Marco Slot fbb7d9c894
Merge pull request #1873 from citusdata/fix_partition_lock 
Do not take extra access exclusive lock on partitioned tables
 2017-12-15 13:32:42 +01:00
Marco Slot 9ee0e68882 Do not take extra access exclusive lock partitioned tables 2017-12-15 13:02:31 +01:00
Marco Slot cf7dda3892
Merge pull request #1871 from citusdata/relax_from_sublink_checks 
Relax checks on recurring tuples in FROM with sublinks
 2017-12-15 12:13:19 +01:00
Marco Slot 5a69fc1b17 Relax checks on recurring tuples in FROM with sublinks 2017-12-15 11:56:06 +01:00
Marco Slot a64f0060ba Reduce the frequency of FinishConnectionIO calls during COPY (#1864) 2017-12-14 13:21:59 -05:00
Marco Slot a811aad264 Deparallelise multi_modifying_xacts tests 2017-12-14 10:27:17 +01:00
Marco Slot c19c3ef4a1
Merge pull request #1853 from citusdata/ctes 
Add support for CTEs in distributed queries
 2017-12-14 10:26:47 +01:00
mehmet furkan şahin 5851f71bfb Add CTE regression tests 2017-12-14 09:32:55 +01:00
Marco Slot fa73abe6d4 Regression test output changes after CTE support 2017-12-14 09:32:55 +01:00
Marco Slot 2e2b4e81fa Add support for CTEs in distributed queries 2017-12-14 09:32:55 +01:00
Marco Slot d0335ec818 Send BEGIN for SELECTs in the router executor 2017-12-14 09:32:55 +01:00
Marco Slot cbbd418af2 Add citus.copy_format OIDs to metadata cache 2017-12-14 09:32:55 +01:00
Marco Slot 66f9f1d6cd Make some intermediate results functions public 2017-12-14 09:32:55 +01:00
Marco Slot 36ee21c323 Make CanUseBinaryCopyFormatForType public 2017-12-14 09:32:55 +01:00
Marco Slot 7d1191954d Add DistributedSubPlan node 2017-12-14 09:32:55 +01:00
Önder Kalacı b5784ca03a
Merge pull request #1852 from citusdata/group_by_on_function 
Treat recurring tuples as reference tables for GROUP BY checks
 2017-12-13 16:37:31 +03:00
Onder Kalaci 86b2d9420c Treat recurring tuples as reference table for GROUP BY checks 
read_intermediate_results() and immutable functions are implemented.
Empty join trees seems not applicable here.
 2017-12-13 14:55:42 +02:00
Marco Slot f0851257fa
Merge pull request #1867 from citusdata/fix_analyze_block 
Fix issue with multiple ANALYZE in transaction block
 2017-12-12 10:54:27 +01:00
Marco Slot d1a470a52e Fix issue with multiple ANALYZE in transaction block 2017-12-12 10:28:48 +01:00
Mehmet Furkan ŞAHİN 84957fe6e7
Merge pull request #1861 from citusdata/new_guc_to_allow_task_executor_swap 
New guc to allow automated task executor swap
 2017-12-11 09:48:42 +03:00
mehmet furkan şahin 3c941aedf1 adds citus.enable_repartition_joins GUC 
The new GUC allows Citus to switch between task executors
when necessary
 2017-12-11 09:36:37 +03:00
Marco Slot 7544e91c87
Merge pull request #1860 from citusdata/needs_distributed_planning 
Allow queries with local tables in NeedsDistributedPlanning
 2017-12-08 10:11:07 +01:00
Marco Slot 5895c88552 Add materialized view regression tests 2017-12-07 16:20:23 +01:00
Marco Slot 60a1e31671 Allow queries with local tables in NeedsDistributedPlanning 2017-12-07 16:20:23 +01:00
Marco Slot d71d519672
Merge pull request #1857 from citusdata/fix_intermediate_result 
Use proper schema in read_intermediate_result signature
 2017-12-07 14:20:10 +01:00
Marco Slot f8550b8c85 Fix issues with read_intermediate_result signature 2017-12-07 13:47:56 +01:00
Marco Slot d8fea4efb8 Revert "Allow queries with local tables in NeedsDistributedPlanning" 
This reverts commit d2bac081e8.
 2017-12-07 11:19:11 +01:00
Marco Slot d2bac081e8 Allow queries with local tables in NeedsDistributedPlanning 2017-12-07 11:02:16 +01:00