bootstrap_optimizer#

Adapted and optimized boostrap fewshot optimizer:

DSPy: Com-piling declarative language model calls into state-of-the-art pipelines.

Classes

BootstrapFewShot(params[, raw_shots, ...])

BootstrapFewShot performs few-shot sampling used in few-shot ICL.

class BootstrapFewShot(params: List[Parameter], raw_shots: int | None = None, bootstrap_shots: int | None = None, dataset: List[DataClass] | None = None, weighted: bool = True, exclude_input_fields_from_bootstrap_demos: bool = False)[source]#

Bases: DemoOptimizer

BootstrapFewShot performs few-shot sampling used in few-shot ICL.

It will be used to optimize paramters of demos. Based on research from AdalFlow team and DsPy library.

Compared with Dspy’s version:

we added weighted sampling for both the raw and augmented demos to prioritize failed demos but successful in augmented demos based on the evaluation score while we backpropagate the demo samples.
In default, we exclude the input fields from the augmented demos. Our reserch finds that using the reasoning demostrations from teacher model can be more effective in some cases than taking both inputs and output samples and be more token efficient.

Reference: - DsPy: Com-piling declarative language model calls into state-of-the-art pipelines.

add_scores(ids: List[str], scores: List[float], is_teacher: bool = True)[source]#

config_shots(raw_shots: int, bootstrap_shots: int)[source]#: Initialize the samples for each parameter.

config_dataset(dataset: List[DataClass])[source]#

property num_shots: int#

sample(augmented_demos: Dict[str, DataClass], demos: Dict[str, DataClass], dataset: List[DataClass], raw_shots: int, bootstrap_shots: int, weighted: bool = True)[source]#: Performs weighted sampling, ensure the score is in range [0, 1]. The higher score means better accuracy.

static samples_to_str(samples: List[DataClass], augmented: bool = False, exclude_inputs: bool = False) → str[source]#

propose()[source]#: Proposing a value while keeping previous value saved on parameter.

revert()[source]#: Revert to the previous value when the evaluation is worse.

step()[source]#: Discard the previous value and keep the proposed value.