trainer#

Ready to use trainer for LLM task pipeline

Classes

Trainer(adaltask[, optimization_order, ...])

Ready to use trainer for LLM task pipeline to optimize all types of parameters.
class Trainer(adaltask: AdalComponent, optimization_order: Literal['sequential', 'mix'] = 'sequential', strategy: Literal['random', 'constrained'] = 'constrained', max_steps: int = 1000, train_batch_size: int | None = 4, num_workers: int = 4, ckpt_path: str = None, batch_val_score_threshold: float | None = 1.0, max_error_samples: int | None = 4, max_correct_samples: int | None = 4, max_proposals_per_step: int = 5, train_loader: Any | None = None, train_dataset: Any | None = None, val_dataset: Any | None = None, test_dataset: Any | None = None, raw_shots: int | None = None, bootstrap_shots: int | None = None, weighted_sampling: bool = False, exclude_input_fields_from_bootstrap_demos: bool = False, debug: bool = False, save_traces: bool = False, *args, **kwargs)[source]#

Bases: Component

Ready to use trainer for LLM task pipeline to optimize all types of parameters.

Training set: can be used for passing initial proposed prompt or for few-shot sampling. Validation set: Will be used to select the final prompt or samples. Test set: Will be used to evaluate the final prompt or samples.

Parameters:

  • adaltask – AdalComponent: AdalComponent instance

  • strategy – Literal[“random”, “constrained”]: Strategy to use for the optimizer

  • max_steps – int: Maximum number of steps to run the optimizer

  • num_workers – int: Number of workers to use for parallel processing

  • ckpt_path – str: Path to save the checkpoint files, default to ~/.adalflow/ckpt.

  • batch_val_score_threshold – Optional[float]: Threshold for skipping a batch

  • max_error_samples – Optional[int]: Maximum number of error samples to keep

  • max_correct_samples – Optional[int]: Maximum number of correct samples to keep

  • max_proposals_per_step – int: Maximum number of proposals to generate per step

  • train_loader – Any: DataLoader instance for training

  • train_dataset – Any: Training dataset

  • val_dataset – Any: Validation dataset

  • test_dataset – Any: Test dataset

  • few_shots_config – Optional[FewShotConfig]: Few shot configuration

  • save_traces – bool: Save traces for for synthetic data generation or debugging

optimizer: Optimizer = None#
ckpt_file: str | None = None#
optimization_order: Literal['sequential', 'mix'] = 'sequential'#
strategy: Literal['random', 'constrained']#
max_steps: int#
ckpt_path: str | None = None#
adaltask: AdalComponent#
num_workers: int = 4#
train_loader: Any#
val_dataset = None#
test_dataset = None#
batch_val_score_threshold: float | None = 1.0#
max_error_samples: int | None = 8#
max_correct_samples: int | None = 8#
max_proposals_per_step: int = 5#
demo_optimizers: List[DemoOptimizer]#
text_optimizers: List[TextOptimizer]#
train_batch_size: int | None = 4#
debug: bool = False#
diagnose(dataset: Any, split: str = 'train')[source]#

Run an evaluation on the trainset to track all error response, and its raw response using AdaplComponent’s default configure_callbacks :param dataset: Any: Dataset to evaluate :param split: str: Split name, default to train and it is also used as set the directory name for saving the logs

Example:

trainset, valset, testset = load_datasets(max_samples=10)
adaltask = TGDWithEvalFnLoss(
    task_model_config=llama3_model,
    backward_engine_model_config=llama3_model,
    optimizer_model_config=llama3_model,
)

trainer = Trainer(adaltask=adaltask)
diagnose = trainer.diagnose(dataset=trainset)
print(diagnose)
fit(*, adaltask: AdalComponent | None = None, train_loader: Any | None = None, train_dataset: Any | None = None, val_dataset: Any | None = None, test_dataset: Any | None = None, debug: bool = False, save_traces: bool = False, raw_shots: int | None = None, bootstrap_shots: int | None = None, resume_from_ckpt: str | None = None)[source]#

train_loader: An iterable or collection of iterables specifying training samples.

initial_validation(val_dataset: Any, test_dataset: Any)[source]#
gather_trainer_states()[source]#
prep_ckpt_file_path(trainer_state: Dict[str, Any] = None)[source]#

Prepare the checkpoint root path: ~/.adalflow/ckpt/task_name/.

It also generates a unique checkpoint file name based on the strategy, max_steps, and a unique hash key. For multiple runs but with the same adalcomponent + trainer setup, the run number will be incremented.

training: bool#
On this page