5c4c9304ba
RemoveDuplicateJoinRestrictions() function was introduced with the aim of decrasing the overall planning times by eliminating the duplicate JOIN restriction entries (#1989). However, it turns out that the function itself is so CPU intensive with a very high algorithmic complexity, it hurts a lot more than it helps. The function is a clear example of premature optimization. The table below shows the difference clearly: "distributed query planning time master" RemoveDuplicateJoinRestrictions() execution time on master "Remove the function RemoveDuplicateJoinRestrictions() this PR" 5 table INNER JOIN 9 msec 2msec 7 msec 10 table INNER JOIN 227 msec 194 msec 29 msec 20 table INNER JOIN 1 sec 235 msec 1 sec 139 msec 90 msecs 50 table INNER JOIN 24 seconds 21 seconds 1.5 seconds 100 table INNER JOIN 2 minutes 16 secods 1 minute 53 seconds 23 seconds 250 table INNER JOIN Bottleneck on JoinClauseList 18 minutes 52 seconds Bottleneck on JoinClauseList 5 table INNER JOIN in subquery 9 msec 0 msec 6 msec 10 table INNER JOIN subquery 33 msec 10 msec 32 msec 20 table INNER JOIN subquery 132 msec 67 msec 123 msec 50 table INNER JOIN subquery 1.2 seconds 900 msec 500 msec 100 table INNER JOIN subquery 6 seconds 5 seconds 2 seconds 250 table INNER JOIN subquery 54 seconds 37 seconds 20 seconds 5 table LEFT JOIN 5 msec 0 msec 5 msec 10 table LEFT JOIN 11 msec 0 msec 13 msec 20 table LEFT JOIN 26 msec 2 msec 30 msec 50 table LEFT JOIN 150 msec 15 msec 193 msec 100 table LEFT JOIN 757 msec 71 msec 722 msec 250 table LEFT JOIN 8 seconds 600 msec 8 seconds 5 JOINs among 2 table JOINs 37 msec 11 msec 25 msec 10 JOINs among 2 table JOINs 536 msec 306 msec 352 msec 20 JOINs among 2 table JOINs 794 msec 181 msec 640 msec 50 JOINs among 2 table JOINs 25 seconds 2 seconds 22 seconds 100 JOINs among 2 table JOINs Bottleneck on JoinClauseList 9 seconds Bottleneck on JoinClauseList 150 JOINs among 2 table JOINs Bottleneck on JoinClauseList 46 seconds Bottleneck on JoinClauseList On top of the performance penalty, the function had a critical bug #4255, and with #4254 we hit one more important bug. It should be fixed by adding the followig check to the ContextCoversJoinRestriction(): ``` static bool JoinRelIdsSame(JoinRestriction *leftRestriction, JoinRestriction *rightRestriction) { Relids leftInnerRelIds = leftRestriction->innerrel->relids; Relids rightInnerRelIds = rightRestriction->innerrel->relids; if (!bms_equal(leftInnerRelIds, rightInnerRelIds)) { return false; } Relids leftOuterRelIds = leftRestriction->outerrel->relids; Relids rightOuterRelIds = rightRestriction->outerrel->relids; if (!bms_equal(leftOuterRelIds, rightOuterRelIds)) { return false; } return true; } ``` However, adding this eliminates all the benefits tha RemoveDuplicateJoinRestrictions() brings. I've used the commands here to generate the JOINs mentioned in the PR: https://gist.github.com/onderkalaci/fe8654f9df5916c7af4c7c5eb892561e#file-gistfile1-txt Inner and outer JOINs behave roughly the same, to simplify the table only added INNER joins. |
||
|---|---|---|
| .circleci | ||
| .github | ||
| ci | ||
| config | ||
| src | ||
| vendor | ||
| .codecov.yml | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .ignore | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| Makefile | ||
| Makefile.global.in | ||
| README.md | ||
| aclocal.m4 | ||
| autogen.sh | ||
| configure | ||
| configure.in | ||
| github-banner.png | ||
| prep_buildtree | ||
README.md
What is Citus?
- Open-source PostgreSQL extension (not a fork)
- Built to scale out across multiple nodes
- Distributed engine for query parallelization
- Database designed to scale out multi-tenant applications, real-time analytics dashboards, and high-throughput transactional workloads
Citus is an open source extension to Postgres that distributes your data and your queries across multiple nodes. Because Citus is an extension to Postgres, and not a fork, Citus gives developers and enterprises a scale-out database while keeping the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases, and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.
Citus serves many use cases. Three common ones are:
-
Multi-tenant & SaaS applications: Most B2B applications already have the notion of a tenant / customer / account built into their data model. Citus allows you to scale out your transactional relational database to 100K+ tenants with minimal changes to your application.
-
Real-time analytics: Citus enables ingesting large volumes of data and running analytical queries on that data in human real-time. Example applications include analytic dashboards with sub-second response times and exploratory queries on unfolding events.
-
High-throughput transactional workloads: By distributing your workload across a database cluster, Citus ensures low latency and high performance even with a large number of concurrent users and high volumes of transactions.
To learn more, visit citusdata.com and join the Citus slack to stay on top of the latest developments.
Getting started with Citus
The fastest way to get up and running is to deploy Citus in the cloud. You can also setup a local Citus database cluster with Docker.
Hyperscale (Citus) on Azure Database for PostgreSQL
Hyperscale (Citus) is a deployment option on Azure Database for PostgreSQL, a fully-managed database as a service. Hyperscale (Citus) employs the Citus open source extension so you can scale out across multiple nodes. To get started with Hyperscale (Citus), learn more on the Citus website or use the Hyperscale (Citus) Quickstart in the Azure docs.
Citus Cloud
Citus Cloud runs on top of AWS as a fully managed database as a service. You can provision a Citus Cloud account at https://console.citusdata.com and get started with just a few clicks.
Local Citus Cluster
If you're looking to get started locally, you can follow the following steps to get up and running.
- Install Docker Community Edition and Docker Compose
- Mac:
- Download and install Docker.
- Start Docker by clicking on the application’s icon.
- Linux:
The above version of Docker Compose is sufficient for running Citus, or you can install the latest version.curl -sSL https://get.docker.com/ | sh sudo usermod -aG docker $USER && exec sg docker newgrp `id -gn` sudo systemctl start docker sudo curl -sSL https://github.com/docker/compose/releases/download/1.11.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose
- Pull and start the Docker images
curl -sSLO https://raw.githubusercontent.com/citusdata/docker/master/docker-compose.yml
docker-compose -p citus up -d
- Connect to the master database
docker exec -it citus_master psql -U postgres
- Follow the first tutorial instructions
- To shut the cluster down, run
docker-compose -p citus down
Talk to Contributors and Learn More
| Documentation | Try the Citus
tutorial for a hands-on introduction or the documentation for a more comprehensive reference. |
| Slack | Chat with us in our community Slack channel. |
| Github Issues | We track specific bug reports and feature requests on our project issues. |
| Follow @citusdata for general updates and PostgreSQL scaling tips. | |
| Citus Blog | Read our Citus Data Blog for posts on Postgres, Citus, and scaling your database. |
Contributing
Citus is built on and of open source, and we welcome your contributions. The CONTRIBUTING.md file explains how to get started developing the Citus extension itself and our code quality guidelines.
Who is Using Citus?
Citus is deployed in production by many customers, ranging from technology start-ups to large enterprises. Here are some examples:
- Algolia uses Citus to provide real-time analytics for over 1B searches per day. For faster insights, they also use TopN and HLL extensions. User Story
- Heap uses Citus to run dynamic funnel, segmentation, and cohort queries across billions of users and has more than 700B events in their Citus database cluster. Watch Video
- Pex uses Citus to ingest 80B data points per day and analyze that data in real-time. They use a 20+ node cluster on Google Cloud. User Story
- MixRank uses Citus to efficiently collect and analyze vast amounts of data to allow inside B2B sales teams to find new customers. User Story
- Agari uses Citus to secure more than 85 percent of U.S. consumer emails on two 6-8 TB clusters. User Story
- Copper (formerly ProsperWorks) powers a cloud CRM service with Citus. User Story
You can read more user stories about how they employ Citus to scale Postgres for both multi-tenant SaaS applications as well as real-time analytics dashboards here.
Copyright © Citus Data, Inc.