Loading huggingface Datasets from Local Paths

One of the key features of Hugging Face datasets is its ability to load datasets from local paths, enabling users to leverage their existing data assets without having to upload them to external repositories. Here’s a step-by-step guide on how to load datasets from local paths using Hugging Face datasets:

Method 1: Using load_dataset with Local Files

Step 1: Install Hugging Face datasets: Begin by installing the Hugging Face datasets library using pip:

pip install datasets

Step 2: Prepare your dataset: Ensure that your dataset is stored locally in a compatible format supported by Hugging Face datasets, such as CSV, JSON, or Parquet. If your dataset is in a different format, you may need to preprocess it accordingly to convert it into a compatible format.

Step 3: Load the dataset: Use the load_dataset function provided by Hugging Face datasets to load your dataset from the local path. Here’s an example of how to load a dataset from a CSV file:

Step 4: Accessing the dataset: Once loaded, you can access the dataset using dictionary-like syntax. For example, to access the first few examples in the dataset:

Output:

1, 3, 4, 5, 6, 6

This will print the first 5 examples in the ‘train’ split of your dataset.

Method 2: Using load_from_disk

If you have previously saved a dataset using the save_to_disk method, you can load it back using load_from_disk.

Example

First, save your dataset to disk:

from datasets import load_dataset

dataset = load_dataset("Dahoas/rm-static")
dataset.save_to_disk("/path/to/save")

Later, you can load it from the saved location:

from datasets import load_from_disk
dataset = load_from_disk("/path/to/save")

This method is useful for reusing datasets without needing to reprocess or redownload them.

Method 3: Using a Local Dataset Script

If your dataset requires a custom processing script, you can place the script in the same directory as your data files and use load_dataset to load it.

Example:

Assume you have the following structure,

/dataset/squad
|- squad.py
|- data
   |- train.json
   |- test.json

To load this dataset, use:

from datasets import load_dataset
dataset = load_dataset("/dataset/squad")

The squad.py script should define how to load and process the dataset. This method is particularly useful for complex datasets that require custom loading logic.

Hugging Face datasets – a powerful library that simplifies the process of loading and managing datasets for machine learning tasks. Loading a Hugging Face dataset from a local path can be done using several methods, depending on the structure and format of your dataset. In this comprehensive guide, we’ll explore how to leverage Hugging Face datasets to load data from local paths, empowering data scientists and machine learning practitioners to harness the full potential of their local data.