21be1af515
Improve citus_shard_sizes performance in release-11.3 ( #7051 )
...
This PR combines the relevant parts of #7003 , #7018 and the PR that
backports those changes to 11.3, #7050 to merge into the 11.3 release
branch.
This PR backports the changes to 11.3 release branch, there is another
PR, #7062 , for 12.0 release branch and one, #7050 , for main branch.
The commits are:
613cced1ae772d194357
2023-07-14 17:19:30 +03:00
48f7d6c279
Show local managed tables in citus_tables and hide tables owned by extensions ( #6321 )
...
Co-authored-by: Marco Slot <marco.slot@gmail.com>
2022-09-12 17:49:17 +03:00
4c3934272f
Improve performance of citus_shards ( #5036 )
...
We were effectively joining on a calculated column because of our calls
to `shard_name`. This caused a really bad plan to be generated. In my
specific case it was taking ~18 seconds to show the output of
citus_shards. It had this explain plan:
```
QUERY PLAN
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Subquery Scan on citus_shards (cost=18369.74..18437.34 rows=5408 width=124) (actual time=18277.461..18278.509 rows=5408 loops=1)
-> Sort (cost=18369.74..18383.26 rows=5408 width=156) (actual time=18277.457..18277.726 rows=5408 loops=1)
Sort Key: ((pg_dist_shard.logicalrelid)::text), pg_dist_shard.shardid
Sort Method: quicksort Memory: 1629kB
CTE shard_sizes
-> Function Scan on citus_shard_sizes (cost=0.00..10.00 rows=1000 width=40) (actual time=71.137..71.934 rows=5413 loops=1)
-> Hash Join (cost=177.62..18024.42 rows=5408 width=156) (actual time=77.985..18257.237 rows=5408 loops=1)
Hash Cond: ((pg_dist_shard.logicalrelid)::oid = (pg_dist_partition.logicalrelid)::oid)
-> Hash Join (cost=169.81..371.98 rows=5408 width=48) (actual time=1.415..13.166 rows=5408 loops=1)
Hash Cond: (pg_dist_placement.groupid = pg_dist_node.groupid)
-> Hash Join (cost=168.68..296.49 rows=5408 width=16) (actual time=1.403..10.011 rows=5408 loops=1)
Hash Cond: (pg_dist_placement.shardid = pg_dist_shard.shardid)
-> Seq Scan on pg_dist_placement (cost=0.00..113.60 rows=5408 width=12) (actual time=0.004..3.684 rows=5408 loops=1)
Filter: (shardstate = 1)
-> Hash (cost=101.08..101.08 rows=5408 width=12) (actual time=1.385..1.386 rows=5408 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 318kB
-> Seq Scan on pg_dist_shard (cost=0.00..101.08 rows=5408 width=12) (actual time=0.003..0.688 rows=5408 loops=1)
-> Hash (cost=1.06..1.06 rows=6 width=40) (actual time=0.007..0.007 rows=6 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on pg_dist_node (cost=0.00..1.06 rows=6 width=40) (actual time=0.004..0.005 rows=6 loops=1)
-> Hash (cost=5.69..5.69 rows=169 width=130) (actual time=0.070..0.071 rows=169 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 36kB
-> Seq Scan on pg_dist_partition (cost=0.00..5.69 rows=169 width=130) (actual time=0.009..0.041 rows=169 loops=1)
SubPlan 2
-> Limit (cost=0.00..3.25 rows=1 width=8) (actual time=3.370..3.370 rows=1 loops=5408)
-> CTE Scan on shard_sizes (cost=0.00..32.50 rows=10 width=8) (actual time=3.369..3.369 rows=1 loops=5408)
Filter: ((shard_name(pg_dist_shard.logicalrelid, pg_dist_shard.shardid) = table_name) OR (('public.'::text || shard_name(pg_dist_shard.logicalrelid, pg_dist_shard.shardid)) = table_name))
Rows Removed by Filter: 2707
Planning Time: 0.705 ms
Execution Time: 18278.877 ms
```
With the changes it only takes 180ms to show the same output:
```
QUERY PLAN
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sort (cost=904.59..918.11 rows=5408 width=156) (actual time=182.508..182.960 rows=5408 loops=1)
Sort Key: ((pg_dist_shard.logicalrelid)::text), pg_dist_shard.shardid
Sort Method: quicksort Memory: 1629kB
-> Hash Join (cost=418.03..569.27 rows=5408 width=156) (actual time=136.333..146.591 rows=5408 loops=1)
Hash Cond: ((pg_dist_shard.logicalrelid)::oid = (pg_dist_partition.logicalrelid)::oid)
-> Hash Join (cost=410.22..492.83 rows=5408 width=56) (actual time=136.231..140.132 rows=5408 loops=1)
Hash Cond: (pg_dist_placement.groupid = pg_dist_node.groupid)
-> Hash Right Join (cost=409.09..417.34 rows=5408 width=24) (actual time=136.218..138.890 rows=5408 loops=1)
Hash Cond: ((((regexp_matches(citus_shard_sizes.table_name, '_(\d+)$'::text))[1])::integer) = pg_dist_shard.shardid)
-> HashAggregate (cost=45.00..48.50 rows=200 width=12) (actual time=131.609..132.481 rows=5408 loops=1)
Group Key: ((regexp_matches(citus_shard_sizes.table_name, '_(\d+)$'::text))[1])::integer
Batches: 1 Memory Usage: 737kB
-> Result (cost=0.00..40.00 rows=1000 width=12) (actual time=107.786..129.831 rows=5408 loops=1)
-> ProjectSet (cost=0.00..22.50 rows=1000 width=40) (actual time=107.780..128.492 rows=5408 loops=1)
-> Function Scan on citus_shard_sizes (cost=0.00..10.00 rows=1000 width=40) (actual time=107.746..108.107 rows=5414 loops=1)
-> Hash (cost=296.49..296.49 rows=5408 width=16) (actual time=4.595..4.598 rows=5408 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 339kB
-> Hash Join (cost=168.68..296.49 rows=5408 width=16) (actual time=1.702..3.783 rows=5408 loops=1)
Hash Cond: (pg_dist_placement.shardid = pg_dist_shard.shardid)
-> Seq Scan on pg_dist_placement (cost=0.00..113.60 rows=5408 width=12) (actual time=0.004..0.837 rows=5408 loops=1)
Filter: (shardstate = 1)
-> Hash (cost=101.08..101.08 rows=5408 width=12) (actual time=1.683..1.685 rows=5408 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 318kB
-> Seq Scan on pg_dist_shard (cost=0.00..101.08 rows=5408 width=12) (actual time=0.004..0.824 rows=5408 loops=1)
-> Hash (cost=1.06..1.06 rows=6 width=40) (actual time=0.007..0.008 rows=6 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on pg_dist_node (cost=0.00..1.06 rows=6 width=40) (actual time=0.004..0.006 rows=6 loops=1)
-> Hash (cost=5.69..5.69 rows=169 width=130) (actual time=0.079..0.079 rows=169 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 36kB
-> Seq Scan on pg_dist_partition (cost=0.00..5.69 rows=169 width=130) (actual time=0.011..0.046 rows=169 loops=1)
Planning Time: 0.789 ms
Execution Time: 184.095 ms
```
2021-06-14 13:32:30 +02:00
a20cc3b36a
Only consider shard state 1 in citus shards ( #4970 )
2021-05-28 11:33:48 +03:00
e96da4886f
Sort results in citus_shards and give raw size ( #4649 )
...
* Sort results in citus_shards and give raw size
Sort results so that it is consistent and also similar to citus_tables.
Use raw size in the output so that doing operations on the size is
easier.
* Change column ordering
2021-02-08 15:29:42 +03:00
724d56f949
Add citus shard helper view ( #4361 )
...
With citus shard helper view, we can easily see:
- where each shard is, which node, which port
- what kind of table it belongs to
- its size
With such a view, we can see shards that have a size bigger than some
value, which could be useful. Also debugging can be easier in production
as well with this view.
Fetch shards in one go per node
The previous implementation was slow because it would do a lot of round
trips, one per shard to be exact. Hence it is improved so that we fetch
all the shard_name, shard-size pairs per node in one go.
Construct shards_names, sizes query on coordinator
2021-01-13 13:58:47 +03:00
Halil Ozan Akgül
Marco Slot
Jelte Fennema
SaitTalhaNisanci