#
Daft

Daft is a high-performance data engine providing simple and reliable data processing for any modality and scale. Daft has native support for reading from and writing to Hugging Face datasets.

##
Getting Started

To get started, pip install `daft` with the `huggingface` feature:

    pip install 'daft[huggingface]'

##
Read

Daft is able to read datasets directly from the Hugging Face Hub using the `daft.read_huggingface()` function or via the `hf://datasets/` protocol.

###
Reading an Entire Dataset

Using `daft.read_huggingface()`, you can easily load a dataset.

    import daft

    df = daft.read_huggingface("username/dataset_name")

This will read the entire dataset into a DataFrame.

###
Reading Specific Files

Not only can you read entire datasets, but you can also read individual files from a dataset repository. Using a read function that takes in a path (such as `daft.read_parquet()`, `daft.read_csv()`, or `daft.read_json()`), specify a Hugging Face dataset path via the `hf://datasets/` prefix:

    import daft

    # read a specific Parquet file
    df = daft.read_parquet("hf://datasets/username/dataset_name/file_name.parquet")

    # or a csv file
    df = daft.read_csv("hf://datasets/username/dataset_name/file_name.csv")

    # or a set of Parquet files using a glob pattern
    df = daft.read_parquet("hf://datasets/username/dataset_name/**/*.parquet")

##
Write

Daft is able to write Parquet files to a Hugging Face dataset repository using `daft.DataFrame.write_huggingface`. Daft supports Content-Defined Chunking and Xet for faster, deduplicated writes.

Basic usage:

    import daft

    df: daft.DataFrame = ...

    df.write_huggingface("username/dataset_name")

See the `DataFrame.write_huggingface` API page for more info.

##
Authentication

The `token` parameter in `daft.io.HuggingFaceConfig` can be used to specify a Hugging Face access token for requests that require authentication (e.g. reading private dataset repositories or writing to a dataset repository).

Example of loading a dataset with a specified token:

    from daft.io import IOConfig, HuggingFaceConfig

    io_config = IOConfig(hf=HuggingFaceConfig(token="your_token"))
    df = daft.read_parquet("hf://datasets/username/dataset_name", io_config=io_config)

< > Update on GitHub