big_bench_hard#

Classes

BigBenchHard([task_name, root, split])

Big Bench Hard dataset for object counting task.

class BigBenchHard(task_name: Literal['BBH_object_counting'] = 'BBH_object_counting', root: str = None, split: Literal['train', 'val', 'test'] = 'train', *args, **kwargs)[source]#

Bases: Dataset

Big Bench Hard dataset for object counting task.

You can find the task name from the following link: suzgunmirac/BIG-Bench-Hard

Data will be saved to ~/.adalflow/cache_datasets/BBH_object_counting/{split}.csv if root is not specified.

Size for each split: - train: 50 examples - val: 50 examples - test: 100 examples

Parameters:

task_name (str) – The name of the task. “BHH_{task_name}” is the task name in the dataset.
root (str, optional) – Root directory of the dataset to save the data. Defaults to ~/.adalflow/cache_datasets/task_name.
split (str, optional) – The dataset split, supports "train" (default), "val" and "test".

static get_default_task_instruction()[source]#