mirror of https://github.com/citusdata/citus.git
3fcb456381
This change creates a slightly higher abstraction of the `PartitionedResultDestReceiver` where it decouples the partitioning from writing it to a file. This allows for easier reuse for other `DestReceiver`'s that would like to route different tuples to different `DestReceiver`'s. Originally there was a lot of state kept in `PartitionedResultDestReceiver` to be able to lazily create `FileDestReceivers` when the first tuple arrived for that target. This convoluted the implementation of the processing of tuples with where they should go. This refactor changes that where it makes the `PartitionedResultDestReceiver` completely agnostic of what kind of Receivers it is writing to. When constructed you pass it a list of `DestReceiver` compatible pointers with the length of `partitionCount`. Internally the `PartitionedResultDestReceiver` keeps track of which `DestReceiver`'s have been started or not, and start them when they first receive a tuple. Alternatively, if the instantiating code of the `PartitionedResultDestReceiver` wants, the startup can be turned from lazily to eagerly. When the startup is eager (not lazy) all `rStartup` functions on the list of `DestReceiver`'s are called during the startup of the `PartitionedResultDestReceiver` and marked as such. A downside of this approach is the following. On highly partitioned destinations we now need to allocate a `FileDestReceiver` for every target, _always_. When the data passed into the `PartitionedResultDestReceiver` is highly skewed to a small set of `FileDestReceiver`'s this will waste some memory. Given the small size of a `FileDestReceiver`, and the fact that actual file handles are only created during the processing of the startup of the `FileDestReceiver` I think this memory waste is not a problem. If this would become a problem we could refactor the source list into some kind of generator object which can generate the `DestReceiver`'s on the fly. |
||
|---|---|---|
| .. | ||
| adaptive_executor.c | ||
| citus_custom_scan.c | ||
| directed_acyclic_graph_execution.c | ||
| distributed_execution_locks.c | ||
| distributed_intermediate_results.c | ||
| insert_select_executor.c | ||
| intermediate_results.c | ||
| local_executor.c | ||
| multi_client_executor.c | ||
| multi_executor.c | ||
| multi_server_executor.c | ||
| partitioned_intermediate_results.c | ||
| placement_access.c | ||
| query_stats.c | ||
| repartition_join_execution.c | ||
| subplan_execution.c | ||
| tuple_destination.c | ||