external
/
citus
mirror of https://github.com/citusdata/citus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,538 Commits
637 Branches
162 Tags
99 MiB
Commit Graph

2244 Commits (a7254db8fda08e823767ce37df5757b45a7a9fa0)

Author SHA1 Message Date
Onder Kalaci a7254db8fd Fix rebase conflict 2020-04-08 10:13:42 +02:00
Onder Kalaci 597651add4 Add assert back 2020-04-08 10:13:42 +02:00
Onder Kalaci 310a0cd442 Effectively do not remove any entries 2020-04-08 10:13:42 +02:00
Onder Kalaci ffd7dcc88f Fix rebase conflict 2020-04-08 10:13:42 +02:00
Onder Kalaci ab345ab4fb Style fix 2020-04-08 10:13:42 +02:00
Onder Kalaci 9311b4b953 WIP handle node operations 2020-04-08 10:13:42 +02:00
Onder Kalaci b6b47370dc WIP on update/delete 2020-04-08 10:13:42 +02:00
Onder Kalaci 7cad050b62 Improve comment 2020-04-08 10:13:42 +02:00
Onder Kalaci ef49e8ba04 Improve readability 2020-04-08 10:13:42 +02:00
Onder Kalaci dd36dd2530 Remove useless check 2020-04-08 10:13:42 +02:00
Onder Kalaci 7fbbb28ee1 Rebase to master - the version files 2020-04-08 10:13:42 +02:00
Onder Kalaci 6b6d00ef38 Properly add the external function 2020-04-08 10:13:42 +02:00
Onder Kalaci dfe2354378 Some indentation/style fixes 2020-04-08 10:13:42 +02:00
Onder Kalaci 156d12483d do not forget to rethrow 2020-04-08 10:13:42 +02:00
Onder Kalaci 217b6520cb Guard against any issues on StartConnectionEstablishment 2020-04-08 10:13:42 +02:00
Onder Kalaci 5083f058f8 Improve comments 2020-04-08 10:13:42 +02:00
Onder Kalaci e09811cfe0 When force_max_query_parallelization is enabled, do not wait 2020-04-08 10:13:42 +02:00
Onder Kalaci 7f578c9c68 citus_inden 2020-04-08 10:13:42 +02:00
Onder Kalaci cf32de7620 Disable failing tests 2020-04-08 10:13:42 +02:00
Onder Kalaci b780b42c93 Rely on condition variable again 2020-04-08 10:13:42 +02:00
Onder Kalaci 8a123fb8e8 Add debug message, shorten sleep 2020-04-08 10:13:42 +02:00
Onder Kalaci 75fb6556b5 Defer pool start to first session 2020-04-08 10:13:42 +02:00
Onder Kalaci 3961f4517b Revert "Use condition variable instead of sleep" 
This reverts commit bc282e8ee53b78cb4c88c6e10ccc00b5da11898a.
 2020-04-08 10:13:42 +02:00
Onder Kalaci 96613c6b2b Use condition variable instead of sleep 2020-04-08 10:13:42 +02:00
Onder Kalaci cd4b9fd004 Minor refactoring 2020-04-08 10:13:42 +02:00
Onder Kalaci 7bff12571d Implement master_update_node 2020-04-08 10:13:42 +02:00
Onder Kalaci 531de5d848 Introduce CanUseOptionalConnection() to consider cached connections 2020-04-08 10:13:42 +02:00
Onder Kalaci 275d9d83f8 Improve assert 2020-04-08 10:13:42 +02:00
Onder Kalaci 452d0b1dcf Add more tests 2020-04-08 10:13:42 +02:00
Onder Kalaci 72a3e1df29 Add the first set of regression tests 2020-04-08 10:13:42 +02:00
Onder Kalaci 3d242ff5f9 Use LwLocks again 2020-04-08 10:13:42 +02:00
Onder Kalaci 593a3be1a3 Properly implement retry timeout 2020-04-08 10:13:42 +02:00
Onder Kalaci bed023c03d Add the GUC for MaxSharedPoolSize 2020-04-08 10:13:42 +02:00
Onder Kalaci 34dec931f4 Remove MaxTrackedWorkerNodes use MaxWorkerNodesTracked 2020-04-08 10:13:42 +02:00
Onder Kalaci 188e571d3c Replace LwLock with spinLock 2020-04-08 10:13:42 +02:00
Onder Kalaci ebb0ebc3c8 Naive implementation for Optional connection concept 2020-04-08 10:13:42 +02:00
Onder Kalaci a385c44f15 Naive implementation for Optional connection concept 2020-04-08 10:13:42 +02:00
Onder Kalaci 66e7b822aa Basic implementation for TryIncrementSharedConnectionCounter 2020-04-08 10:13:42 +02:00
Onder Kalaci 22c0f2a8cd Add citus_remote_connection_stats() function 
This function is intended to be used for monitoring
the remote connections.
 2020-04-08 10:13:42 +02:00
Onder Kalaci 4d00dc94f8 Add the necessary shared memory infrastructure 
- The hashmap in the shared memory
- The lock to access the hashmap
- The GUC to control the size
 2020-04-08 10:13:42 +02:00
Philip Dubé 26797bfb94 Verify trigger relation before reading old/new tuples 
master_dist_placement_cache_invalidate: bail when triggering on pg_dist_shard_placement
 2020-04-07 15:39:31 +00:00
Önder Kalacı 70012dfd33 Do not error when an intermediate file does not exit (#3707) 
When the file does not exist, it could mean two different things.
First -- and a lot more common -- case is that a failure happened
in a concurrent backend on the same distributed transaction. And,
one of the backends in that transaction has already been roll
backed, which has already removed the file. If we throw an error
here, the user might see this error instead of the actual error
message. Instead, we prefer to WARN the user and pretend that the
file has no data in it. In the end, the user would see the actual
error message for the failure.

Second, in case of any bugs in intermediate result broadcasts,
we could try to read a non-existing file. That is most likely
to happen during development. Thus, when asserts enabled, we throw
an error instead of WARNING so that the developers cannot miss.
 2020-04-07 17:06:55 +02:00
Onder Kalaci a695b44ce9 Add new regression tests 2020-04-07 17:06:55 +02:00
Onder Kalaci 4b3d17f466 Make sure that tests are not failing randomly 2020-04-07 17:06:55 +02:00
Onder Kalaci 4f7c902c6c Move connection establishment for intermediate results after query execution 
When we have a query like the following:

```SQL
WITH a AS (SELECT * FROM foo LIMIT 10) SELECT max(x) FROM a JOIN bar 2 USING (y);
```

Citus currently opens side channels for doing the
	`COPY "1_1"` FROM STDIN (format 'result')

before starting the execution of
	`SELECT * FROM foo LIMIT 10`

Since we need at least 1 connection per worker to do
	`SELECT * FROM foo LIMIT 10`
We need to have 2 connections to worker in order to broadcast the results.

However, we don't actually send a single row over the side channel until the
execution of `SELECT * FROM foo LIMIT 10` is completely done (and connections
unclaimed) and the results are written to a tuple store. We could actually
reuse the same connection for doing the `COPY "1_1"` FROM STDIN (format 'result').

This also fixes the issue that Citus doesn't obey `citus.max_adaptive_executor_pool_size`
when the query includes an intermediate result.
 2020-04-07 17:06:55 +02:00
Onder Kalaci 721daec9a5 Move the logic that initilize connections/local files into a function 2020-04-07 17:06:55 +02:00
Onder Kalaci 9b29a32d7a Remove all references for side channel connections 
We don't need any side channel connections. That is actually
problematic in the sense that it creates extra connections.
Say, citus.max_adaptive_executor_pool_size equals to 1, Citus
ends up using one extra connection for the intermediate results.
Thus, not obeying citus.max_adaptive_executor_pool_size.

In this PR, we remove the following entities from the codebase
to allow further commits to implement not requiring extra connection
for the intermediate results:

- The connection flag REQUIRE_SIDECHANNEL
- The function GivePurposeToConnection
- The ConnectionPurpose struct and related fields
 2020-04-07 17:06:55 +02:00
Hanefi Onaldi 1d22d0c2ff
Remove metadata locks from size functions 2020-04-07 17:37:15 +03:00
SaitTalhaNisanci 0430b568be
explicitly return false if transaction connected to local node (#3715) 
* explicitly return false if transaction connected to local node

* not set TransactionConnectedToLocalGroup if we are writing to a file

We use TransactionConnectedToLocalGroup to prevent local execution from
happening as that might cause visibility problems. As files are visible
to all transactions, we shouldn't set this variable if we are writing to
a file.
 2020-04-07 17:30:34 +03:00
Marco Slot 2632343f64 Fix intermediate result pruning for INSERT..SELECT 2020-04-07 11:07:49 +02:00