d399f395f7
So far citus used postgres' predicate proofing logic for shard
pruning, except for INSERT and COPY which were already optimized for
speed. That turns out to be too slow:
* Shard pruning for SELECTs is currently O(#shards), because
PruneShardList calls predicate_refuted_by() for every
shard. Obviously using an O(N) type algorithm for general pruning
isn't good.
* predicate_refuted_by() is quite expensive on its own right. That's
primarily because it's optimized for doing a single refutation
proof, rather than performing the same proof over and over.
* predicate_refuted_by() does not keep persistent state (see 2.) for
function calls, which means that a lot of syscache lookups will be
performed. That's particularly bad if the partitioning key is a
composite key, because without a persistent FunctionCallInfo
record_cmp() has to repeatedly look-up the type definition of the
composite key. That's quite expensive.
Thus replace this with custom-code that works in two phases:
1) Search restrictions for constraints that can be pruned upon
2) Use those restrictions to search for matching shards in the most
efficient manner available:
a) Binary search / Hash Lookup in case of hash partitioned tables
b) Binary search for equal clauses in case of range or append
tables without overlapping shards.
c) Binary search for inequality clauses, searching for both lower
and upper boundaries, again in case of range or append
tables without overlapping shards.
d) exhaustive search testing each ShardInterval
My measurements suggest that we are considerably, often orders of
magnitude, faster than the previous solution, even if we have to fall
back to exhaustive pruning.
|
||
|---|---|---|
| src | ||
| .codecov.yml | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .travis.yml | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| Makefile | ||
| Makefile.global.in | ||
| README.md | ||
| autogen.sh | ||
| configure | ||
| configure.in | ||
| github-banner.png | ||
| prep_buildtree | ||
README.md
What is Citus?
- Open-source PostgreSQL extension (not a fork)
- Scalable across multiple machines through sharding and replication
- Distributed engine for query parallelization
- Database designed to scale multi-tenant applications
Citus is a distributed database that scales across commodity servers using transparent sharding and replication. Citus extends the underlying database rather than forking it, giving developers and enterprises the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases, and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.
Citus serves many use cases. Two common ones are:
-
Multi-tenant database: Most B2B applications already have the notion of a tenant / customer / account built into their data model. Citus allows you to scale out your transactional relational database to 100K+ tenants with minimal changes to your application.
-
Real-time analytics: Citus enables ingesting large volumes of data and running analytical queries on that data in human real-time. Example applications include analytic dashboards with subsecond response times and exploratory queries on unfolding events.
To learn more, visit citusdata.com and join the mailing list to stay on top of the latest developments.
Getting started with Citus
The fastest way to get up and running is to create a Citus Cloud account. You can also setup a local Citus cluster with Docker.
Citus Cloud
Citus Cloud runs on top of AWS as a fully managed database as a service and has development plans available for getting started. You can provision a Citus Cloud account at https://console.citusdata.com and get started with just a few clicks.
Local Citus Cluster
If you're looking to get started locally, you can follow the following steps to get up and running.
- Install Docker Community Edition and Docker Compose
- Mac:
- Download and install Docker.
- Start Docker by clicking on the application’s icon.
- Linux:
The above version of Docker Compose is sufficient for running Citus, or you can install the latest version.curl -sSL https://get.docker.com/ | sh sudo usermod -aG docker $USER && exec sg docker newgrp `id -gn` sudo systemctl start docker sudo curl -sSL https://github.com/docker/compose/releases/download/1.11.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose
- Pull and start the Docker images
curl -sSLO https://raw.githubusercontent.com/citusdata/docker/master/docker-compose.yml
docker-compose -p citus up -d
- Connect to the master database
docker exec -it citus_master psql -U postgres
- Follow the first tutorial instructions
- To shut the cluster down, run
docker-compose -p citus down
Talk to Contributors and Learn More
| Documentation | Try the Citus
tutorial for a hands-on introduction or the documentation for a more comprehensive reference. |
| Google Groups | The Citus Google Group is our place for detailed questions and discussions. |
| Slack | Chat with us in our community Slack channel. |
| Github Issues | We track specific bug reports and feature requests on our project issues. |
| Follow @citusdata for general updates and PostgreSQL scaling tips. |
Contributing
Citus is built on and of open source, and we welcome your contributions. The CONTRIBUTING.md file explains how to get started developing the Citus extension itself and our code quality guidelines.
Who is Using Citus?
Citus is deployed in production by many customers, ranging from technology start-ups to large enterprises. Here are some examples:
- CloudFlare uses Citus to provide real-time analytics on 100 TBs of data from over 4 million customer websites. Case Study
- MixRank uses Citus to efficiently collect and analyze vast amounts of data to allow inside B2B sales teams to find new customers. Case Study
- Neustar builds and maintains scalable ad-tech infrastructure that counts billions of events per day using Citus and HyperLogLog.
- Agari uses Citus to secure more than 85 percent of U.S. consumer emails on two 6-8 TB clusters. Case Study
- Heap uses Citus to run dynamic funnel, segmentation, and cohort queries across billions of users and tens of billions of events. Watch Video
Copyright © 2012–2017 Citus Data, Inc.