Datasets

Overview

datasets.big_bench_hard
datasets.trec
datasets.hotpot_qa
datasets.types

---

class BigBenchHard(task_name: Literal['object_counting'] = 'object_counting', root: str = None, split: Literal['train', 'val', 'test'] = 'train', *args, **kwargs)

Bases: Dataset

Big Bench Hard dataset for object counting task.

You can find the task name from the following link: https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/bbh

Data will be saved to ~/.adalflow/cache_datasets/BBH_object_counting/{split}.csv if root is not specified.

Size for each split: - train: 50 examples - val: 100 examples - test: 100 examples

Parameters:

• task_name (str) – The name of the task. “{task_name}” is the task name in the dataset.

• root (str, optional) – Root directory of the dataset to save the data. Defaults to ~/.adalflow/cache_datasets/task_name.

• split (str, optional) – The dataset split, supports "train" (default), "val" and "test".

static get_default_task_instruction()

class HotPotQA(only_hard_examples=True, root: str = None, split: Literal['train', 'val', 'test'] = 'train', keep_details: Literal['all', 'dev_titles', 'none'] = 'dev_titles', size: int = None, **kwargs)

Bases: Dataset

class Example(id: str = '9df2c18a-470c-4c27-ae44-e237ce1236d2', question: str = None, answer: str = None)

Bases: DataClass

A common dataclass for representing examples in a dataset.

id: str = '9df2c18a-470c-4c27-ae44-e237ce1236d2'

question: str = None

answer: str = None

class HotPotQAData(id: str = '9df2c18a-470c-4c27-ae44-e237ce1236d2', question: str = None, answer: str = None, gold_titles: set = None, context: Dict[str, object] = None)

Bases: Example

A dataclass for representing examples in the HotPotQA dataset.

gold_titles: set = None

context: Dict[str, object] = None

class TrecDataset(root: str = None, split: Literal['train', 'test'] = 'train')

Bases: Dataset

Trec dataset for question classification.

Here we only load a small subset of the dataset for training and evaluation.

In default: train: 600, 100 per class, val: 36, test: 144 All class-balanced.

Reference: - https://huggingface.co/datasets/trec labels: https://huggingface.co/datasets/trec/blob/main/trec.py

class TrecData(id: str = 'e8f1eed0-1cd4-486e-b43c-2cf5927a7fc2', question: str = None, class_name: str = None, class_index: int = -1)

Bases: BaseData

A dataclass for representing examples in the TREC dataset.

question: str = None

class_name: str = None

class_index: int = -1

class GSM8KData(id: str = '9df2c18a-470c-4c27-ae44-e237ce1236d2', question: str = None, answer: str = None, gold_reasoning: str = None, reasoning: str = None)

Bases: Example

A dataclass for representing examples in the GSM8K dataset.

You can reset the output fields:

GSM8KData.set_output_fields(["answer"])

gold_reasoning: str = None

reasoning: str = None

class GSM8K(root: str = None, split: Literal['train', 'val', 'test'] = 'train', size: int = None, **kwargs)

Bases: Dataset

Use huggingface datasets to load GSM8K dataset.

official_train: 7473 official_test: 1319

Our train split: 3736/2 Our val split: 3736/2 Our test split: 1319

You can use size to limit the number of examples to load.

Example:

dataset = GSM8K(split="train", size=10)

print(f"example: {dataset[0]}")

The output will be:

GSM8KData(id='8fc791e6-ea1d-472c-a882-d00d0600d423',
question="The result from the 40-item Statistics exam Marion and Ella took already came out.
Ella got 4 incorrect answers while Marion got 6 more than half the score of Ella.
  What is Marion's score?",
  answer='24',
  gold_reasoning="Ella's score is 40 items - 4 items = <<40-4=36>>36 items.
  Half of Ella's score is 36 items / 2 = <<36/2=18>>18 items.
  So, Marion's score is 18 items + 6 items = <<18+6=24>>24 items.",
  reasoning=None)