Commit Graph

886 Commits (747e4396018511a5210da6ab6fd2e1103782ca1e)

Author SHA1 Message Date
Hadi Moshayedi 747e439601 Limit number of stats collection retries to once a day. 2017-10-31 21:51:43 -04:00
Hadi Moshayedi 78a2cd9052 Check for Citus updates.
Sends a request to /v1/releases/latest?flavor=$CITUS_EDITION once a day,
which returns a response similar to {"version": "7.1.0", "major": 7,
"minor": 1, "patch": 0}. Then compares it with current Citus version,
and if the latest release is newer, logs a LOG message.
2017-10-31 21:51:43 -04:00
Hadi Moshayedi 34f3ec0961 Call FlushDistTableCache() before stats collection. 2017-10-31 21:51:43 -04:00
Hadi Moshayedi c18c6625d9 Lock relations before calling citus_table_size().
This is to make sure they don't get dropped.
2017-10-31 21:51:43 -04:00
Hadi Moshayedi 97d544b75c Follow the patterns used in Deadlock Detection in Stats Collection.
This includes:

(1) Wrap everything inside a StartTransactionCommand()/CommitTransactionCommand().
This is so we can access the database. This also switches to a new memory context
and releases it, so we don't have to do our own memory management.

(2) LockCitusExtension() so the extension cannot be dropped or created concurrently.

(3) Check CitusHasBeenLoaded() && CheckCitusVersion() before doing any work.

(4) Do not PG_TRY() inside a loop.
2017-10-31 21:51:43 -04:00
Marco Slot 100aaeb3f5 Fix typo in distributed deadlock error message 2017-10-31 19:39:32 +01:00
metdos 8c356b2bc8 Don't try to add restrictions for reference tables in insert into select 2017-10-31 19:44:10 +02:00
mehmet furkan şahin 32fb19911c Add Constraint %s Add Primary Key Using index %s support
This commit makes a change in relay_event_utility.c to check if the
Alter Table command adds a constraint using index. If this is the
case, it appends the shard id to the index name.
2017-10-31 16:03:56 +03:00
Marco Slot 7e34348334 Add shard transfer mode parameter to shard copy functions 2017-10-31 13:30:48 +01:00
Marco Slot 2bb46bb5ee Reset connectionReady flag after moving a connection in WaitForAllConnections 2017-10-31 12:06:53 +01:00
Marco Slot e6e6897499 Defer initial PQflush to main loop in WaitForAllConnections 2017-10-31 12:06:53 +01:00
Marco Slot d6dadb1b25 Use correct index for ModifyWaitEvent in WaitForAllConnections 2017-10-31 12:06:53 +01:00
Furkan Sahin 2b39c52f0b Replica identity on create_distributed_table
By this commit, citus minds the replica identity of the table when
we distribute the table. So the shards of the distributed table
have the same replica identity with the local table.
2017-10-31 13:08:36 +03:00
Marco Slot 7f68f78ee9 Omit public schema from shard_name output 2017-10-31 00:22:07 +01:00
Murat Tuncer e16805215d
Support count(distinct) for non-partition columns (#1692)
Expands count distinct coverage by allowing more cases. We used to support
count distinct only if we can push down distinct aggregate to worker query
i.e. the count distinct clause was on the partition column of the table,
or there was a grouping on the partition column.

Now we can support
- non-partition columns, with or without grouping on partition column
- partition, and non partition column in the same query
- having clause
- single table subqueries
- insert into select queries
- join queries where count distinct is on partition, or non-partition column
- filters on count distinct clauses (extends existing support)

We first try to push down aggregate to worker query (original case), if we
can't then we modify worker query to return distinct columns to coordinator
node. We do that by adding distinct column targets to group by clauses. Then
we perform count distinct operation on the coordinator node.

This work should reduce the cases where HLL is used as it can address anything
that HLL can. However, if we start having performance issues due to very large
number rows, then we can recommend hll use.
2017-10-30 13:12:24 +02:00
Marco Slot be46661bf7 Block only 2PCs instead of all writes in citus_create_restore_point 2017-10-27 00:07:32 +02:00
mehmet furkan şahin 83ac84d594 order by and unnest are added to multi_colocation_utils tests 2017-10-26 13:44:28 +03:00
mehmet furkan şahin 61ae33dc7f ALTER TABLE .. REPLICA IDENTITY support is implemented 2017-10-26 13:44:28 +03:00
Brian Cloutier 4a17d12d74 Replace uint with uint32 2017-10-25 19:32:12 -07:00
velioglu 0b5db5d826 Support multi shard update/delete queries 2017-10-25 15:52:38 +03:00
Marco Slot 4bde83e1d2 Relay error message if DML fails on worker 2017-10-25 14:23:21 +02:00
Hadi Moshayedi 9a04b78980 Send server_id for statistics reports. (#1698)
This change introduces the `pg_dist_node_metadata` which has a single jsonb value. When creating
the extension, a random server id is generated and stored in there. Everything in the metadata table
is added as a nested objected to the json payload that is sent to the reports server.
2017-10-18 21:20:32 -04:00
Hadi Moshayedi 86bcd93a4a Don't collect stats when there is a version mismatch. (#1712)
The following scenario can cause an Assert() crash if we don't do this:
- Install Citus v7.0-15
- Restart server & run a query to start maintenanced.
- Install Citus v7.1
- Restart server & run a query. This will tell user to upgrade.
- Type "UPDATE EXTENSION c" & press tab. maintenanced will start and crash
  with Assert(CitusHasBeenLoaded() && CheckCitusVersion(WARNING));

This change checks Citus version before calling metadata functions so the
crash doesn't happen.
2017-10-17 14:01:14 -04:00
Jason Petersen f2c593b25c
Add CITUS_NAME and CITUS_EDITION
Unambiguous places to check whether we're running simply Citus or Citus
Enterprise, or to check for 'community' or 'enterprise'.
2017-10-16 18:09:29 -06:00
Jason Petersen 8544878c4b
Add citus_version(), analogous to PG's version()
This will provide the full project name (i.e. Citus/Citus Enterprise),
and the host system, compiler, and architecture word size.

I wanted to limit the number of copied files in 'config', so I added
only config.guess and call it manually, rather than using the macro
AC_CANONICAL_HOST, which requires several other files.
2017-10-16 18:09:29 -06:00
Brian Cloutier 91ff8cd2d5 {*,}create_distributed_table doesn't emit OID (#1710) 2017-10-16 18:08:51 -06:00
Brian Cloutier ebcb2b65e9 Add master_move_node function 2017-10-16 10:51:28 -07:00
Brian Cloutier 58cf15ceca DistributedTableSize doesn't emit oid when erring out 2017-10-14 02:42:57 +03:00
Hadi Moshayedi 2aec6eda49 Properly use #ifdef HAVE_LIBCURL. 2017-10-13 12:04:36 -06:00
Jason Petersen 01353cb7cb Use header define rather than -D flag
Eclipse apparently doesn't scan build output looking for -D flags, so
having the value actually appear in a header is nicer for those of us
using IDEs.
2017-10-13 11:00:09 -04:00
Hadi Moshayedi 946659aebe Delete StatsCollection memory context after we are done with stats reporting.
Previously we left the memory context untouched, which overtime leaked memory.
2017-10-13 11:00:09 -04:00
Hadi Moshayedi 873fd1e7ff Fix compiling --without-libcurl.
Previously <curl/curl.h> was included even if compiled --without-libcurl.
This can fail when libcurl headers are not there. This commit guards this
include by checks for HAVE_LIBCURL.
2017-10-13 11:00:09 -04:00
Murat Tuncer 4832abc7cb Make multi_master_planner.c coding convention compliant
Changed order of function definitions and added
declarations in the beginning of the file
2017-10-13 14:59:48 +03:00
Murat Tuncer f7ab901766 Add select distinct, and distinct on support
Distinct, and distinct on() clauses are supported
in simple selects, joins, subqueries, and insert into select
queries.
2017-10-13 14:59:48 +03:00
Hadi Moshayedi 6879f92e23 Fix out of bound memeory access when getting HTTP response code. (#1699) 2017-10-12 12:51:42 -04:00
Hadi Moshayedi a1387f4aa8 Basic usage statistics collection. (#1656)
Adds ```citus.enable_statistics_collection``` GUC variable, which ```true``` by default, unless built without libcurl. If statistics collection is enabled, sends basic usage data to Citus servers every 24 hours.

The data that is collected consists of:
- Citus version
- OS name & release
- Hardware Id
- Number of tables, rounded to next power of 2
- Size of data, rounded to next power of 2
- Number of workers
2017-10-11 09:55:15 -04:00
Onder Kalaci 498ac80d8b Add window function support for SUBQUERY PUSHDOWN and INSERT INTO SELECT
This commit provides the support for window functions in subquery and insert
into select queries. Note that our support for window functions is still limited
because it must have a partition by clause on the distribution key. This commit
makes changes in the files insert_select_planner and multi_logical_planner. The
required tests are also added with files multi_subquery_window_functions.out
and multi_insert_select_window.out.
2017-10-04 15:33:07 +03:00
Marco Slot 24915779d1 Remove separate citus.binary_worker_copy_format regression tests 2017-10-03 17:44:50 +02:00
Marco Slot 9e516513fc Use local group ID when querying for prepared transactions 2017-10-03 16:36:53 +02:00
Hadi Moshayedi 11adb9b034 Push down LIMIT and HAVING when grouped by partition key. (#1641)
We can do this because all rows belonging to a group are in the same shard when grouping by distribution column on a range/hash distributed table.
2017-10-02 20:17:51 -04:00
Marco Slot 394918f9d0 Invalidate worker and group ID cache in maintenance daemon 2017-10-02 18:14:29 +02:00
Marco Slot bb50fc9cb5 Add multi-user re-partitioning regression tests 2017-09-28 15:27:26 +02:00
Marco Slot 43d5e79eaa Execute transmit commands as superuser during task-tracker queries 2017-09-28 15:27:25 +02:00
Marco Slot 306c58d59b Check for absolute paths in COPY with format transmit 2017-09-28 15:27:11 +02:00
Marco Slot cb6b0e820c Allow read-only users to run task-tracker queries 2017-09-28 13:52:36 +02:00
Marco Slot da6b42a3e2 Use unique constraint index for transaction record deletion 2017-09-28 12:04:56 +02:00
Onder Kalaci 68ca8cb7f0 Skip relation extension locks
We should skip if the process blocked on the relation
extension since those locks are hold for a short duration
while the relation is actually extended on the disk and
released as soon as the extension is done. Thus, recording
such waits on our lock graphs could yield detecting wrong
distributed deadlocks.
2017-09-28 10:09:09 +03:00
Murat Tuncer 4676c4f7a5 Prevent crash when remote transaction start fails (#1662)
We sent multiple commands to worker when starting a transaction.
Previously we only checked the result of the first command that
is transaction 'BEGIN' which always succeeds. Any failure on
following commands were not checked.

With this commit, we make sure all command results are checked.
If there is any error we report the first error found.
2017-09-26 17:25:46 -07:00
Jason Petersen b4d53423fa
Add adapter functions for OpenFile changes 2017-09-25 17:20:24 -07:00
Jason Petersen d686123dae
Omit now-public Explain methods from PG11 build
This copy-pasted code is no longer needed in PG11.
2017-09-25 17:20:24 -07:00