citus/src/backend/distributed/operations
Jelte Fennema fd1427de2c
Change by_disk_size rebalance strategy to have a base size (#7035)
One problem with rebalancing by disk size is that shards in newly
created collocation groups are considered extremely small. This can
easily result in bad balances if there are some other collocation groups
that do have some data. One extremely bad example of this is:
1. You have 2 workers
2. Both contain about 100GB of data, but there's a 70MB difference.
3. You create 100 new distributed schemas with a few empty tables in
   them
4. You run the rebalancer
5. Now all new distributed schemas are placed on the node with that had
   70MB less.
6. You start loading some data in these shards and quickly the balance
   is completely off

To address this edge case, this PR changes the by_disk_size rebalance
strategy to add a a base size of 100MB to the actual size of each
shard group. This can still result in a bad balance when shard groups
are empty, but it solves some of the worst cases.
2023-06-27 16:37:09 +02:00
..
citus_create_restore_point.c Move CheckCitusVersion to the top of each function 2021-06-01 17:43:46 +02:00
citus_split_shard_by_split_points.c Add non-blocking variant of create_distributed_table (#6087) 2022-08-30 15:35:40 +03:00
citus_tools.c Include gpid in all internal application names (#6431) 2022-11-25 11:16:33 +01:00
create_shards.c Call null-shard-key tables as single-shard distributed tables in code 2023-05-03 17:02:43 +03:00
delete_protocol.c Drop SHARD_STATE_TO_DELETE (#6494) 2023-01-03 14:38:16 +03:00
health_check.c Make pg_version_compat.h and listutils.c dependency-free. (#5548) 2022-01-04 23:02:08 -08:00
isolate_shards.c Fix bug preventing isolate_tenant_to_new_shard with text column (#6320) 2022-09-12 16:29:57 +02:00
modify_multiple_shards.c Deprecate citus.replicate_reference_tables_on_activate, make it always off (#6474) 2022-11-04 16:21:10 +01:00
node_protocol.c Identity column implementation refactorings (#6738) 2023-03-30 10:41:01 +03:00
partitioning.c Move CheckCitusVersion to the top of each function 2021-06-01 17:43:46 +02:00
shard_cleaner.c Drop SHARD_STATE_TO_DELETE (#6494) 2023-01-03 14:38:16 +03:00
shard_rebalancer.c Change by_disk_size rebalance strategy to have a base size (#7035) 2023-06-27 16:37:09 +02:00
shard_split.c When creating a HTAB we need to use HASH_COMPARE flag in order to set a user defined comparison function. (#6845) 2023-04-11 21:24:33 +03:00
shard_transfer.c Refactor shard transfers (#6631) 2023-04-03 10:43:54 +03:00
stage_protocol.c Changes citus_shard_sizes view's Shard Name Column to Shard Id (#7003) 2023-06-13 16:36:35 +03:00
worker_copy_table_to_node_udf.c Exclude-Generated-Columns-In-Copy (#6721) 2023-03-07 18:15:50 +03:00
worker_node_manager.c Drop PG13 Support Phase 2 - Remove PG13 specific paths/tests (#7007) 2023-06-21 14:18:23 +03:00
worker_shard_copy.c Drop PG13 Support Phase 2 - Remove PG13 specific paths/tests (#7007) 2023-06-21 14:18:23 +03:00
worker_split_copy_udf.c Exclude-Generated-Columns-In-Copy (#6721) 2023-03-07 18:15:50 +03:00
worker_split_shard_release_dsm_udf.c Add worker_split_shard_release_dsm udf to release dynamic shared memory (#6248) 2022-08-26 18:27:32 +05:30
worker_split_shard_replication_setup_udf.c Unique names for replication artifacts (#6529) 2022-12-06 15:48:16 +03:00