smallpond.logical.dataset.SqlQueryDataSet#

class smallpond.logical.dataset.SqlQueryDataSet(sql_query: str, query_builder: Callable[[DuckDBPyConnection, AbstractFileSystem], str] | None = None)#

The result of a sql query.

__init__(sql_query: str, query_builder: Callable[[DuckDBPyConnection, AbstractFileSystem], str] | None = None) → None#

Construct a dataset from a list of paths.

Parameters#

paths: A path or a list of paths or path patterns. e.g. [‘data/100.parquet’, ‘/datasetA/*.parquet’].
root_dir, optional: Relative paths in paths would be resolved under root_dir if specified.
recursive, optional: Resolve path patterns recursively if true.
columns, optional: Only load the specified columns if not None.
union_by_name, optional: Unify the columns of different files by name (see https://duckdb.org/docs/data/multiple_files/combining_schemas#union-by-name).

Methods

`__init__`(sql_query[, query_builder])	Construct a dataset from a list of paths.
`log`([num_rows])	Log the dataset to the logger.
`merge`(datasets)	Merge multiple datasets into a single dataset.
`partition_by_files`(npartition[, random_shuffle])	Evenly split into npartition datasets by files.
`reset`([paths, root_dir, recursive])	Reset the dataset with new paths, root_dir, and recursive flag.
`sql_query_fragment`([filesystem, conn])	Return a sql fragment that represents the dataset.
`to_arrow_table`([max_workers, filesystem, conn])	Load the dataset to an arrow table.
`to_batch_reader`([batch_size, filesystem, conn])	Return an arrow record batch reader to read the dataset.
`to_pandas`()	Convert the dataset to a pandas dataframe.

Attributes

`sql_query`
`query_builder`
`absolute_paths`	An ordered list of absolute paths of the given file patterns.
`columns`	The columns to load from the dataset files.
`empty`	Whether the dataset is empty.
`num_files`	The number of files in the dataset.
`num_rows`	The number of rows in the dataset.
`paths`	The paths to the dataset files.
`recursive`	Whether to resolve path patterns recursively.
`resolved_paths`	An ordered list of absolute paths of files.
`root_dir`	The root directory of paths.
`udfs`
`union_by_name`	Whether to unify the columns of different files by name.