smallpond.dataframe.DataFrame.map_batches

DataFrame.map_batches(func: Callable[[Table], Table], *, batch_size: int = 122880, **kwargs) → DataFrame

Apply the given function to batches of data.

Parameters

func

A function or a callable class to apply to each batch of data. It should take a arrow.Table as input and returns a arrow.Table.

batch_size, optional

The number of rows in each batch. Defaults to 122880.