Commit Graph

3702 Commits (e61ced53e3480bb3040b47a8ed6cbb3d0108ea43)

Author SHA1 Message Date
Jelte Fennema f4791fcb10
Remove SwallowErrors by using PathNameDeleteTemporaryDir (#3893)
This is a different version of #3634. It also removes SwallowErrors, but
instead of modifying our own functions to not throw errors, it uses the
postgres built in `PathNameDeleteTemporaryDir` function. This function
does not throw errors.

Since this change is for a bugfix, I tried to minimize the changes.

PRs with the following changes would be good to do separately from this
PR:
1. Use PathName(Create|Open|Delete)Temporary(File|Dir) to open and
   remove all files/dirs instead of our own custom file functions.
2. Prefix our outmost files/directories with `PG_TEMP_FILE_PREFIX` so
   that they are identified by Postgres as temporary files, which will be
   removed at postmaster start. This way we do not have to do this cleanup
   ourselves.
3. Store the files in the temporary table space if it exists.

Fixes #3634
Fixes #3618
2020-06-10 17:04:07 +02:00
Hanefi Onaldi 6e9324e99d
Merge pull request #3841 from citusdata/copy_max_adaptive_executor_pool_size 2020-06-10 17:25:12 +03:00
Onder Kalaci 640717bea2
Copy doesn't use more than MaxAdaptiveExecutor
Co-authored-by: Hanefi Önaldı <Hanefi.Onaldi@Microsoft.com>
2020-06-10 16:46:21 +03:00
Jelte Fennema b87bae71bb
Error out when using different users in the same transaction (#3869)
Fixes #3867

As described in the issue above we return incorrect results when
changing user within a transaction. This causes us to error out instead.
2020-06-10 14:07:40 +02:00
Marco Slot 02a70df656
Merge pull request #3889 from citusdata/fix/stage_generates_utility_commands
Execute shard creation as utility tasks
2020-06-10 11:51:24 +02:00
Marco Slot 1243b6a948 Execute shard creation as utility tasks 2020-06-10 11:29:49 +02:00
Önder Kalacı 34554a2957
Merge pull request #3886 from citusdata/fix_coercion
Coerce types properly for distribution keys when necessary
2020-06-10 10:50:47 +02:00
Onder Kalaci 06461ca55f Coerce types properly for INSERT
Also, unify similar code-paths to rely on more accurate function.
2020-06-10 10:40:28 +02:00
Hadi Moshayedi 08d2b9b40b
Merge pull request #3881 from citusdata/explain_analyze_udfs
Implement EXPLAIN ANALYZE udfs.
2020-06-09 10:12:21 -07:00
Hadi Moshayedi 5cdfa9f571 Implement EXPLAIN ANALYZE udfs.
Implements worker_save_query_explain_analyze and worker_last_saved_explain_analyze.

worker_save_query_explain_analyze executes and returns results of query while
saving its EXPLAIN ANALYZE to be fetched later.

worker_last_saved_explain_analyze returns the saved EXPLAIN ANALYZE result.
2020-06-09 10:02:05 -07:00
Onur Tirtir a4f1c41391
Implement GetQueryLockMode helper (#3860)
If we want to get necessary lockmode for a relation RangeVar within
a query, we can get the lockmode easily from the RangeVar itself (if
pg version >= 12).

However, if we want to decide the lockmode appropriate for the
"query", we can derive this information by using GetQueryLockMode
according to the code comment from RangeTblEntry->rellockmode.
2020-06-09 13:08:44 +03:00
Hadi Moshayedi 5781aaf6c7
Merge pull request #3883 from citusdata/circ_deps
typedef TupleDestination only once
2020-06-09 01:36:41 -07:00
Hadi Moshayedi 198d5d8b0f typedef TupleDestination once 2020-06-08 20:38:28 -07:00
Hadi Moshayedi 6869d7bfb2
Merge pull request #3878 from citusdata/explain_analyze_initial_cleanup
Explain Analyze tests & cleanup
2020-06-07 00:06:11 -07:00
Hadi Moshayedi 45a41e249f Test EXPLAIN ANALYZE doesn't show repartition join tasks 2020-06-06 23:24:45 -07:00
Hadi Moshayedi 02cff1a7c6 Test that EXPLAIN ANALYZE is not supported for some forms of INSERT/SELECT 2020-06-06 23:24:45 -07:00
Hadi Moshayedi f54a8e53c0 Remove unused consts from multi_explain.c 2020-06-06 23:24:45 -07:00
Hadi Moshayedi 797405f3d1
Merge pull request #3871 from citusdata/tupledest
Implement TupleDestination to allow custom processing of task results.
2020-06-06 10:50:02 -07:00
Hadi Moshayedi 0bfd39ea52 Implement TupleDestination intereface.
Implements a new `TupleDestination` interface to allow custom tuple processing per task.

This can be specially useful if a task contains multiple queries. An example of this EXPLAIN
ANALYZE, where it needs to add some UDF calls to the query to fetch the explain output
from worker after fetching the actual query results.
2020-06-05 17:47:40 -07:00
SaitTalhaNisanci d0f47eb338
Check the removeType in IsDropCitusStmt (#3859)
We should check the remove type in IsDropCitusStmt because if the remove
type is not OBJECT_EXTENSION then the stored objects in
dropStmt->objects may not be of type Value. This was crashing PG-13.

Also rename the method as IsDropCitusExtensionStmt.
2020-06-05 20:49:54 +03:00
Onur Tirtir f7224a12f2
Implement PushOverrideEmptySearchPath (#3874)
To reduce code duplication, implement function that pushes search_path
to be NIL and sets addCatalog to true so that all objects outside of
pg_catalog will be schema-prefixed.
2020-06-05 19:23:59 +03:00
Onur Tirtir 8b39d12846
Append IF NOT EXISTS to deparsed CREATE SERVER commands (#3875)
Append IF NOT EXISTS to CREATE SERVER commands generated by
pg_get_serverdef_string function when deparsing an existing server
object that a foreign table depends.
2020-06-05 18:04:33 +03:00
Onur Tirtir 741e808049
Merge pull request #3873 from citusdata/refactor/implement-explicit-index
Implement IndexIsImpliedByAConstraint
2020-06-05 16:05:08 +03:00
Onur Tirtir f3f711e097 Implement IndexIsImpliedByAConstraint 2020-06-05 15:33:54 +03:00
Philip Dubé 4f63443b49
Merge pull request #3861 from citusdata/remove-null-plannerrestrictioncontext-check
multi_router_planner: Remove NULL check which would've segfaulted earlier
2020-06-02 13:18:30 +00:00
Philip Dubé 25f86bca3f multi_router_planner: Remove NULL check which would've segfaulted earlier 2020-06-02 13:08:38 +00:00
Philip Dubé 1b0776f98e
Merge pull request #3863 from citusdata/remove-getupdateordeleterte
multi_router_planner: replace GetUpdateOrDeleteRTE with ExtractResultRelationRTE
2020-06-02 12:58:42 +00:00
Philip Dubé 2623aefe38 multi_router_planner: replace GetUpdateOrDeleteRTE with ExtractResultRelationRTE 2020-06-02 00:22:30 +00:00
Onur Tirtir 738a4ddb58
Merge pull request #3858 from citusdata/copy_paste_pg_get_triggerdef
* Error out if creating a citus table from a table having triggers already.
* Error out for CREATE TRIGGER commands that are run on citus tables.
* Introduce the ability to deparse CREATE TRIGGER commands needed to recreate triggers on a table.
2020-06-01 10:35:45 +03:00
Onur Tirtir dfcc18468c Error out for unsupported trigger objects
Error out if creating a citus table from a table having triggers.
Error out for CREATE TRIGGER commands that are run on citus tables.
2020-05-31 23:10:01 +03:00
Onur Tirtir 6e6bc155a9 Implement methods to process & recreate triggers on citus tables 2020-05-31 15:28:17 +03:00
Onur Tirtir 5af64084ea Copy & paste pg_get_triggerdef_worker from Postgres 2020-05-31 15:25:07 +03:00
Philip Dubé dca09e998f
Merge pull request #3857 from citusdata/use_RelationGetPartitionDesc
use RelationGetPartitionDesc to be more safe
2020-05-29 15:12:57 +00:00
Sait Talha Nisanci dec2b28d49 use RelationGetPartitionDesc to be more safe
For getting the partition desc, we should use RelationGetPartitionDesc
method so that even if it is NULL, it will be created in the method.
2020-05-29 10:55:52 +03:00
Philip Dubé a3c470b2b8
Merge pull request #3834 from citusdata/prep-routing-modifying-ctes
Prep routing modifying ctes
2020-05-20 17:35:36 +00:00
Philip Dubé c0515dcd67 This prepares for routing modifying CTEs, where modLevel should not be used to infer whether a plan is a select or not
SELECT_TASK is renamed to READ_TASK as a SELECT with modifying CTEs will be a MODIFYING_TASK

RouterInsertJob: Assert originalQuery->commandType == CMD_INSERT
CreateModifyPlan: Assert originalQuery->commandType != CMD_SELECT

Remove unused function IsModifyDistributedPlan

DistributedExecution, ExecutionParams, DistributedPlan: Rename hasReturning to expectResults
SELECTs set expectResults to true

Rename CreateSingleTaskRouterPlan to CreateSingleTaskRouterSelectPlan
2020-05-20 17:26:12 +00:00
Onur Tirtir 780c72bdc8
Merge pull request #3846 from citusdata/implement-table-constraints-internal
Refactor the methods accessing to pg_constraint
2020-05-20 17:45:59 +03:00
Onur Tirtir 98a660d0b7 Don't release lock on pg_constraint until the xact ends
Do not release AccessShareLock when closing pg_constraint to prevent
modifications to be done on pg_constraint to make sure that caller
will process valid foreign key constraints through the transaction.
2020-05-20 17:27:17 +03:00
Onur Tirtir 79a688ffe0 Refactor the methods accessing to pg_constraint
Implement internal functions to accces to pg_contraint
and utilize them in existing foreign key checks.
2020-05-20 17:27:17 +03:00
SaitTalhaNisanci 80e34382cf
Rename AppropriateReplicationModel -> DecideReplicationModel (#3842) 2020-05-17 10:24:14 +03:00
Onur Tirtir 8f9ef63e8a
Implement get_relation_constraint_oid_compat helper (#3836) 2020-05-15 17:36:59 +03:00
Philip Dubé 4ad10adbc4
Merge pull request #3832 from citusdata/fix/varchar-type-propagation
Fix composite types lacking typemod when distributed
2020-05-15 13:25:19 +00:00
MoYi 9e1f198155 Fix composite create type deparsing to preserve typmod 2020-05-15 13:12:54 +00:00
Onur Tirtir 249550b815
Refactor EnsureLocalTableEmptyIfNecessary (#3830) 2020-05-15 14:20:33 +03:00
Onur Tirtir 8f3373c702
Remove unused parameter from RecordDistributedRelationDependencies (#3831) 2020-05-15 10:34:35 +03:00
SaitTalhaNisanci 625f0d034c
Merge pull request #3773 from citusdata/enh/hammerdbJob
Add optional ch_benchmark and tpcc_benchmark job
2020-05-14 16:08:43 +03:00
Sait Talha Nisanci 41fceb7849 Add optional ch_benchmark and tpcc_benchmark job
With this commit:
You can trigger two types of hammerdb benchmark jobs:
-ch_benchmark (analytical and transactional queries)
-tpcc_benchmark (only transactional queries)

Your branch will be run against `master` branch.

In order to trigger the jobs prepend `ch_benchmark/` or `tpcc_benchmark/` to your branch and push it.

For example if you were running on a feature/improvement branch with name `improve/adaptive_executor`. In order to trigger a tpcc benchmark, you can do the following:

```bash
git checkout improve/adaptive_executor
git checkout -b tpcc_benchmark/improve/adaptive_executor
git push origin tpcc_benchmark/improve/adaptive_executor # the tpcc benchmark job will be triggered.
```

You will see the results in a branch in [https://github.com/citusdata/release-test-results](https://github.com/citusdata/release-test-results).

The branch name will be something like: `delete_me/citusbot_tpcc_benchmark_rg/<date>/<date>`.

The resource groups will be deleted automatically but if the benchmark fails, they won't be deleted(If you don't see the results after a reasonable time, it might mean it failed, you can check the resource usage from portal, if it is almost 0 and you didn't see the results, it means it probably failed). In that case, you will need to delete the resource groups manually from portal, the resource groups are `citusbot_ch_benchmark_rg` and `citusbot_tpcc_benchmark_rg`.
2020-05-14 16:01:48 +03:00
SaitTalhaNisanci cf98b9d6d5
not wait forever for metadata sync in tests (#3760)
We shouldn't wait forever for metada sync in tests, otherwise when a
test gets stuck, we don't know which line causes the problem.
2020-05-14 10:51:24 +03:00
Onur Tirtir a024389ab6
Use exactly matching tag in citus_version output (#3828)
Instead of using the git tag that is reachable from the HEAD commit,
use the exactly matching tag in citus_version().

For the other branches (including the master), just use name of the
current branch that we are installing citus and sha of the HEAD.
This is because, we do not tag the branches except the release ones.

That way, we won't see v9.2.3-DO-NOT-INSTALL tag anymore in the output
of the citus_version();
2020-05-13 15:05:07 +03:00
Onur Tirtir ac1ec40bfb
Add changelog entry for 9.3.0 (#3823) 2020-05-07 16:04:07 +03:00