Commit Graph

118 Commits (2d10d4975dfd10ff9fb6762a69359c31eb477cb5)

Author SHA1 Message Date
Murat Tuncer e86b4b397c Refactor multi_planner to create router plan directly
If router plan creation fails, it falls back to normal planner
2016-06-21 12:50:21 +03:00
Burak Yucesoy 2da5ae240e Fix master_append_table_to_shard to work with schemas
Fixes #78

With this change, it is possible to append a table in any schema to shard. The function
master_append_table_to_shard now supports schema names.
2016-06-17 04:35:00 +03:00
Andres Freund acb36b4505 Store ShardInterval instead of shardId in RangeTableFragments.
For CITUS_RTE_RELATION type fragments, reloading shardIntervals from the
database is rather expensive. So store a pointer to the full shard
interval, instead of just the shard id.  There's no new memory lifetime
hazards here, because we already passed a pointer to the shardInterval's
->shardId field around.

The plan time for the query in issue #607 goes from 2889 ms to 106 ms.
with this change.
2016-06-16 17:31:35 -07:00
Andres Freund 1e07a94435 Use cached comparator in ShardIntervalsOverlap().
By far the most expensive part of ShardIntervalsOverlap() is computing
the function to use to determine overlap. Luckily we already have that
computed and cached.

The plan time for the query in issue #607 goes from 8764 ms to 2889 ms
with this change.
2016-06-16 17:21:19 -07:00
Marco Slot f15ec5554c Do not copy outer join clauses into WHERE 2016-06-16 16:42:32 -07:00
Metin Doslu 7ede3db4f5 Drop function from public and create in pg_catalog
Fixes #600
2016-06-16 14:08:40 -07:00
Burak Yucesoy bdf9ca2466 Append shardId before escaping the table name
Fixes #550, fixes #545

If table name contains special characters, it needs to be escaped. However in some cases,
we escape table name before appending shardId, which causes syntax error in the queries
sent to worker nodes. With this change we now append shardId before escaping table names.
2016-06-15 04:15:40 +03:00
Eren ae5687e726 Eliminate compile time warnings in multi_logical_optimizer.c
This change removes some issues about mixed declarations
and code in TablePartitioningSupportsDistinct() and
WorkerExtendedOpNode() functions.
2016-06-10 12:27:12 +03:00
Murat Tuncer 0de0e7c3d1 Refactor task tracker cleanup to enable workers receive cleanup jobs
Long sleep is replaced by multiple small sleeps. Maximum timeout
is also increased since we do not have to wait for that long most
of the cases.
2016-06-09 17:03:54 +03:00
Murat Tuncer 315b7f3e4c Fix crash in count distinct with filters in repartition subqueries
now copies all column references in count distinct aggreagete
to worker target list and group by. Master target list is
also updated to reflect changes in attribute order.

Fixes 569
2016-06-09 11:47:24 +03:00
Jason Petersen 369ab7664c Minor formatting/comment fixes 2016-06-08 10:34:07 -06:00
Amos Bird b58dfd93ae Add overflow checks. 2016-06-08 10:30:03 +08:00
Amos Bird ad42423c24 Eliminates the possibilities of counter overflows.
This patch uses scanint8 instead of pg_atoi to make sure the affected
tuples counter never gets overflow.
2016-06-08 10:30:03 +08:00
Burak Yücesoy 54c0d827d8 Fix wrong storage type for foreign tables
Fixes #496

Previously we do not check whether table is foreign or not while creating empty
shards, and set storage type to 't'(Standard table) or 'c'(Columnar table). Now
if the table is foreign table(but not CStore foreign table) we set storage
type to 'f'(Foreign table). If it is CStore foreign table, we set its storage
type to 'c', i.e. columnar table have priority over foreign table.

Please note that 'c' is only used for CStore tables not for other possible
columnar stores at the moment. Possible improvement could be checking for other
columnar stores, though I am not sure if there is a way to check it for all
other columnar stores.
2016-06-08 04:12:01 +03:00
Jason Petersen 281d93b3c6 Make ReportRemoteError's CONTEXT style-compliant
There's not a ton of documentation about what CONTEXT lines should look
like, but this seems like the most dominant pattern. Similarly, users
should expect lowercase, non-period strings.
2016-06-07 12:47:16 -06:00
Jason Petersen 8efb504d1a Refactor ReportRemoteError to remove boolean arg
Broke it into two explicitly-named functions instead: WarnRemoteError
and ReraiseRemoteError.
2016-06-07 12:38:32 -06:00
Metin Doslu dfc7dd8d87 Fail fast on constraint violations in router executor 2016-06-07 18:11:17 +03:00
Metin Doslu 6195535906 Update ereport format 2016-06-07 15:58:32 +03:00
Metin Doslu 5eb2e76296 Update only shard length on statistics update for hash-partitioned
Update only the shard length on master_update_shard_statistics() call for
hash-partitioned tables.

Fixes #519.
2016-06-07 15:04:29 +03:00
Murat Tuncer fcd4248f6a Add enable_ddl_propagation flag to control automatic ddl propagation 2016-06-06 13:42:46 +03:00
Murat Tuncer 41096f2076 Change equality operator check for operator expressions 2016-06-06 12:34:16 +03:00
Burak Yucesoy 15f55cb675 Remove ONLY clause from worker queries
Fixes #475

With this change we prevent addition of ONLY clause to queries prepared for
worker nodes. When we add ONLY clause we may miss the inherited tables in
worker nodes created by users manually.
2016-06-03 11:42:43 +03:00
Andres Freund ee5bb2297b Rely less on remote_task_check_interval.
When executing queries with citus.task_executor = 'real-time', query
execution could, so far, spend a significant amount of time
sleeping. That's because we were
a) sleeping after several phases of query execution, even if we're not
   waiting for network IO
b) sleeping for a fixed amount of time when waiting for network IO;
   often a lot longer than actually required.
Just reducing the amount of time slept isn't a real solution, because
that just increases CPU usage.

Instead have the real-time executor's ManageTaskExecution return whether
a task is currently being processed, waiting for reads or writes, or
failed. When all tasks are waiting for IO use poll() to wait for IO
readyness.

That requires to slightly redefine how connection timeouts are handled:
before we counted the number of times ManageTaskExecution() was called,
and compared that with the timeout divided by the task check
interval. That, if processing of tasks took a while, could significantly
increase the time till a timeout occurred. Because it was based on the
ManageTaskExecution() being called on a constant interval, this approach
isn't feasible anymore.  Instead measure the actual time since
connection establishment was started. That could in theory, if task
processing takes a very long time, lead to few passes over
PQconnectPoll().

The problem of sleeping too much also exists for the 'task-tracker'
executor, but is generally less problematic there, as processing the
individual tasks usually will take longer. That said, for e.g. the
regression tests it'd be helpful to use a similar approach.
2016-06-02 12:11:16 -06:00
Metin Doslu c094104d9e Move master_update_shard_statistics() to pg_catalog
Fixes #546
2016-06-02 10:52:47 +03:00
Jason Petersen cc46222e35 Fix formatting
Checking in citus_indent output.
2016-05-27 15:13:28 -06:00
Amos Bird ed0002f28e Remove redundant implementations of error funcs.
This patch does some basic cleaning jobs. It removes duplicated
implementations of ReportRemoteError() and related ones and adjusts
regression tests.
2016-05-27 15:12:59 -06:00
Jason Petersen f9f17cd1ba Merge branch credativ:reproducible
cr: @jasonmp85
2016-05-27 12:45:55 -06:00
Matthew Seaman 62bf21de5d Add inet includes for htonl and htons funtions
Needed to fix FreeBSD builds.
2016-05-27 12:36:12 -06:00
Murat Tuncer 9167373f54 Add complex distinct count support for repartitioned subqueries
Single table repartition subqueries now support count(distinct column)
and count(distinct (case when ...)) expressions. Repartition query
extracts column used in aggregate expression and adds them to target
list and group by list, master query stays the same (count (distinct ...))
but attribute numbers inside the aggregate expression is modified to
reflect changes in repartition query.
2016-05-27 15:43:05 +03:00
Metin Doslu a82efa6613 Make master_create_empty_shard() aware of the shard placement policy
Now, master_create_empty_shard() will create shards according to the
value of citus.shard_placement_policy which also makes default round-robin
instead of random.
2016-05-27 15:05:53 +03:00
eren 793cb2d004 ADD master_modify_multiple_shards UDF
Fixes #10

This change creates a new UDF: master_modify_multiple_shards
Parameters:
  modify_query: A simple DELETE or UPDATE query as a string.

The UDF is similar to the existing master_apply_delete_command UDF.
Basically, given the modify query, it prunes the shard list, re-constructs
the query for each shard and sends the query to the placements.

Depending on the value of citus.multi_shard_commit_protocol, the commit
can be done in one-phase or two-phase manner.

Limitations:
* It cannot be called inside a transaction block
* It only be called with simple operator expressions (like Single Shard Modify)

Sample Usage:
```
SELECT master_modify_multiple_shards(
  'DELETE FROM customer_delete_protocol WHERE c_custkey > 500 AND c_custkey < 500');
```
2016-05-26 17:30:35 +03:00
Burak Yucesoy 31b0423f1f Fix #469
This change renames one of the ReceiveRegularFile functions with
more descriptive name.
2016-05-26 12:03:36 +03:00
Christoph Berg 2d56be6983 Sort list of objects in src/backend/distributed/Makefile
Make's $(wildcard) does not sort the glob result, but returns filenames
in filesystem ordering. This makes the build result vary and hence
unreproducible on the binary level. Fix by adding $(sort).

Spotted by Debian's reproducible builds project.
2016-05-18 10:42:20 +02:00
Jason Petersen 60b7cdfa7c Add gitignore rules for latest install files
Got tired of dirty git tree.
2016-05-10 11:57:11 -06:00
Marco Slot d333c49280 Add JSON/XML validation to EXPLAIN regression tests and fix issues 2016-05-06 11:30:07 +02:00
Lukas Fittl 19e71b5271 Distributed EXPLAIN: Generate valid JSON output.
This modifies the EXPLAIN output functions to actually generate
valid JSON output when (FORMAT JSON) is being used.

Fixes #494.
2016-05-05 12:48:01 +02:00
Andres Freund 812a930f6c Stamp 5.1 release. 2016-05-04 18:05:41 -07:00
Andres Freund e28ce607d2 Generate extension versions from the previous one. 2016-05-04 18:05:41 -07:00
Onder Kalaci 38a1092687 Fix compile time warning
This change fixes a compile time warning related to definition/declaration order
of the code.
2016-05-04 09:42:10 +03:00
Metin Doslu fb6b6daf9d Add COPY support on worker nodes for append partitioned relations
Now, we can copy to an append-partitioned distributed relation from
any worker node by providing master options such as;

COPY relation_name FROM file_path WITH (delimiter '|', master_host 'localhost', master_port 5432);

where master_port is optional and default is 5432.
2016-05-03 16:00:00 +03:00
Brian Cloutier 5962c9b7c8 Query Planning Performance Improvments (#474)
- Only look at pruned shards when determining AnchorTable
- Use cached shardIntervalCompareFunction during copartition check
2016-05-03 10:48:46 +03:00
Jason Petersen 37103eb92f Force bad connections in tests by closing sockets
Based on Andres' suggestion, I removed SetConnectionStatus, moving its
functionality directly into set_connection_status_bad, which now simply
shuts down the socket underlying a particular connection.

This keeps the functionality as-is while removing our questionable use
of internal libpq headers.
2016-04-29 15:56:04 -07:00
Marco Slot cfbdbe29a9 Add EXPLAIN for simple distributed queries 2016-04-30 00:11:02 +02:00
eren 9dc6f6b2e2 FIX "mixed declarations and code" Warning in multi_physical_planner.c
Fixes #477

This change fixes the compile time warning message in BuildMapMergeJob in
multi_physical_planner.c about mixed declarations and code. Basically, the
problematic declaration is moved up so that no expression is before it.
2016-04-29 11:18:04 +03:00
Brian Cloutier 38fdb01b91 Allow references to columns in UPDATE statements (#472)
Allow references to columns in UPDATE statements

Queries like "UPDATE tbl SET column = column + 1" are now allowed, so long as you don't use any IMMUTABLE functions.
2016-04-28 05:45:16 -07:00
eren 888457bb7f Rename copy_transaction_manager
This change renames the distributed transaction manager parameter from
citus.copy_transaction_manager to citus.multi_shard_commit_protocol.

Distributed transaction manager has been used only by the COPY on hash
partitioned tables but it can be used by upcoming features so, we needed
to rename so that its name do not contain a reference to COPY.

The change also includes renames like transaction_manager_options to
commit_protocol_options and TRANSACTION_MANAGER_1PC to COMMIT_PROTOCOL_1PC.

With this change, declaration of MultiShardCommitProtocol (was
CopyTransactionManager) is moved from multi_copy.c to multi_transaction.c.
2016-04-28 15:12:50 +03:00
Andres Freund a9d7f62cad Perform permission checks on operations re-implemented by citus.
Currently that's just COPY FROM.  There's other places where we could
check for permissions earlier (to fail less verbosely), but since
there's other pending changes in the whole DDL area, which is affected
by this, I'm just adding a note to those places.
2016-04-27 10:28:36 -07:00
Andres Freund 63998786ba Create new shards as owned the distributed table's owner.
That's important because ownership of relations implies special
privileges. Without this change, a distributed table can be accessible
by a table's owner, but a shard created by another user might not.
2016-04-27 10:28:33 -07:00
Andres Freund c45b94e88a Add ReplicateGrantStmt().
This is the basis for coordinating GRANT/REVOKE across nodes.
2016-04-27 10:28:25 -07:00
Andres Freund ee6ef363c0 Add pg_get_table_grants() function and support extending GRANTs. 2016-04-27 10:28:25 -07:00