trainer

Ready to use trainer for LLM task pipeline

Classes

Trainer(adaltask[, optimization_order, ...])

Ready to use trainer for LLM task pipeline to optimize all types of parameters.

class Trainer(adaltask: AdalComponent, optimization_order: Literal['sequential', 'mix'] = 'sequential', strategy: Literal['random', 'constrained'] = 'constrained', max_steps: int = 1000, train_batch_size: int | None = 4, num_workers: int = 4, ckpt_path: str = None, batch_val_score_threshold: float | None = 1.0, correct_val_score_threshold: float | None = 0.5, max_error_samples: int | None = 2, max_correct_samples: int | None = 2, max_proposals_per_step: int = 5, train_loader: Any | None = None, train_dataset: Any | None = None, val_dataset: Any | None = None, test_dataset: Any | None = None, raw_shots: int | None = None, bootstrap_shots: int | None = None, weighted_sampling: bool = False, exclude_input_fields_from_bootstrap_demos: bool = False, debug: bool = False, save_traces: bool = False, sequential_order: List[str] = ['text', 'demo'], skip_subset_val: bool = False, disable_backward_gradients: bool = False, disable_backward: bool = False, text_optimizers_config_kwargs: Dict[str, Any] | None = {}, *args, **kwargs)

Bases: Component

Ready to use trainer for LLM task pipeline to optimize all types of parameters.

Training set: can be used for passing initial proposed prompt or for few-shot sampling. Validation set: Will be used to select the final prompt or samples. Test set: Will be used to evaluate the final prompt or samples.

Parameters:

• adaltask – AdalComponent: AdalComponent instance

• strategy – Literal[“random”, “constrained”]: Strategy to use for the optimizer

• max_steps – int: Maximum number of steps to run the optimizer

• num_workers – int: Number of workers to use for parallel processing

• ckpt_path – str: Path to save the checkpoint files, default to ~/.adalflow/ckpt.

• batch_val_score_threshold – Optional[float]: Threshold for skipping a batch

• max_error_samples – Optional[int]: Maximum number of error samples to keep

• max_correct_samples – Optional[int]: Maximum number of correct samples to keep

• max_proposals_per_step – int: Maximum number of proposals to generate per step

• train_loader – Any: DataLoader instance for training

• train_dataset – Any: Training dataset

• val_dataset – Any: Validation dataset

• test_dataset – Any: Test dataset

• few_shots_config – Optional[FewShotConfig]: Few shot configuration

• save_traces – bool: Save traces for for synthetic data generation or debugging

• debug (and for demo) – bool: Debug mode to run the trainer in debug mode. If debug is True, for text debug, the graph will be under /ckpt/YourAdalComponentName/debug_text_grads for prompt parameter,

• debug

• parameters. (the graph will be under /ckpt/YourAdalComponentName/debug_demos for demo)

Note

When you are in the debug mode, you can use get_logger api to show more detailed log on your own.

Example:

from adalflow.utils import get_logger

get_logger(level=”DEBUG”)

optimizer: Optimizer = None

ckpt_file: str | None = None

random_seed: int = None

optimization_order: Literal['sequential', 'mix'] = 'sequential'

strategy: Literal['random', 'constrained']

max_steps: int

ckpt_path: str | None = None

adaltask: AdalComponent

num_workers: int = 4

train_loader: Any

val_dataset = None

test_dataset = None

batch_val_score_threshold: float | None = 1.0

correct_val_score_threshold: float | None = 0.5

max_error_samples: int | None = 2

max_correct_samples: int | None = 2

max_proposals_per_step: int = 5

demo_optimizers: List[DemoOptimizer]

text_optimizers: List[TextOptimizer]

train_batch_size: int | None = 4

debug: bool = False

sequential_order: List[str] = ['text', 'demo']

skip_subset_val: bool = False

disable_backward_gradients: bool = False

disable_backward: bool = False

text_optimizers_config_kwargs: Dict[str, Any] | None = {}

set_random_seed(seed: int)

diagnose(dataset: Any, split: str = 'train', resume_from_ckpt: str = None)

Run an evaluation on the trainset to track all error response, and its raw response using AdaplComponent’s default configure_callbacks :param dataset: Any: Dataset to evaluate :param split: str: Split name, default to train and it is also used as set the directory name for saving the logs

Example:

trainset, valset, testset = load_datasets(max_samples=10)
adaltask = TGDWithEvalFnLoss(
    task_model_config=llama3_model,
    backward_engine_model_config=llama3_model,
    optimizer_model_config=llama3_model,
)

trainer = Trainer(adaltask=adaltask)
diagnose = trainer.diagnose(dataset=trainset)
print(diagnose)

diagnose_report(split: str, acc_score: float | None = None, stats_list: List[Dict] | None = None, log_paths: Dict[str, List[str]] | None = None)

debug_report(text_grad_debug_path: Dict[str, object] | None = None, few_shot_demo_debug_path: Dict[str, object] | None = None)

resume_params_from_ckpt(ckpt_file: str)

Resume the parameters from the checkpoint file

fit(*, adaltask: AdalComponent | None = None, train_loader: Any | None = None, train_dataset: Any | None = None, val_dataset: Any | None = None, test_dataset: Any | None = None, debug: bool = False, save_traces: bool = False, raw_shots: int | None = None, bootstrap_shots: int | None = None, resume_from_ckpt: str | None = None, backward_pass_setup: BackwardPassSetup | None = None) → Tuple[str, TrainerResult]

train_loader: An iterable or collection of iterables specifying training samples.

Returns:

Checkpoint file and the TrainerResult object

Return type:

Tuple[str, TrainerResult]

initial_validation(val_dataset: Any, test_dataset: Any)

gather_trainer_states()

prep_ckpt_file_path(trainer_state: Dict[str, Any] = None)

Prepare the checkpoint root path: ~/.adalflow/ckpt/task_name/.

It also generates a unique checkpoint file name based on the strategy, max_steps, and a unique hash key. For multiple runs but with the same adalcomponent + trainer setup, the run number will be incremented.

training: bool

teacher_mode: bool

tracing: bool