smallpond.logical.dataset.FileSet#

class smallpond.logical.dataset.FileSet(paths: str | List[str], root_dir: str | None = '', recursive=False, columns: List[str] | None = None, union_by_name=False)#

A set of files.

__init__(paths: str | List[str], root_dir: str | None = '', recursive=False, columns: List[str] | None = None, union_by_name=False) → None#

Construct a dataset from a list of paths.

Parameters#

paths: A path or a list of paths or path patterns. e.g. [‘data/100.parquet’, ‘/datasetA/*.parquet’].
root_dir, optional: Relative paths in paths would be resolved under root_dir if specified.
recursive, optional: Resolve path patterns recursively if true.
columns, optional: Only load the specified columns if not None.
union_by_name, optional: Unify the columns of different files by name (see https://duckdb.org/docs/data/multiple_files/combining_schemas#union-by-name).

Methods

`__init__`(paths[, root_dir, recursive, ...])	Construct a dataset from a list of paths.
`log`([num_rows])	Log the dataset to the logger.
`merge`(datasets)	Merge multiple datasets into a single dataset.
`partition_by_files`(npartition[, random_shuffle])	Evenly split into npartition datasets by files.
`reset`([paths, root_dir, recursive])	Reset the dataset with new paths, root_dir, and recursive flag.
`sql_query_fragment`([filesystem, conn])	Return a sql fragment that represents the dataset.
`to_arrow_table`([max_workers, filesystem, conn])	Load the dataset to an arrow table.
`to_batch_reader`([batch_size, filesystem, conn])	Return an arrow record batch reader to read the dataset.
`to_pandas`()	Convert the dataset to a pandas dataframe.

Attributes

`paths`	The paths to the dataset files.
`root_dir`	The root directory of paths.
`recursive`	Whether to resolve path patterns recursively.
`columns`	The columns to load from the dataset files.
`absolute_paths`	An ordered list of absolute paths of the given file patterns.
`empty`	Whether the dataset is empty.
`num_files`	The number of files in the dataset.
`num_rows`	The number of rows in the dataset.
`resolved_paths`	An ordered list of absolute paths of files.
`udfs`
`union_by_name`	Whether to unify the columns of different files by name.