smallpond.logical.node.ConsolidateNode#
- class smallpond.logical.node.ConsolidateNode(ctx: Context, input_dep: Node, dimensions: List[str])#
Consolidate partitions into larger ones.
- __init__(ctx: Context, input_dep: Node, dimensions: List[str])#
Effectively reduces the number of partitions without shuffling the data across the network.
Parameters#
- dimensions
Partitions would be grouped by these dimensions and consolidated into larger partitions.
Methods
__init__(ctx, input_dep, dimensions)Effectively reduces the number of partitions without shuffling the data across the network.
add_perf_metrics(name, value)create_task(*args, **kwargs)get_perf_stats(name)slim_copy()task_factory(task_builder)Attributes
enable_resource_boostnum_partitions