Callbacks

Early Stopping

class torchflare.callbacks.EarlyStopping(mode: str, monitor: str, patience: int = 5, min_delta: float = 1e-07)[source]

Implementation of Early Stopping Callback.

Parameters

  • monitor – The quantity to be monitored. (Default : val_loss) If you want to monitor other metric just pass in the name of the metric.

  • patience – Number of epochs with no improvement after which training will be stopped.

  • mode – One of {“min”, “max”}. In min mode, training will stop when the quantity monitored has stopped decreasing. In “max” mode it will stop when the quantity monitored has stopped increasing.

  • min_delta – Minimum change in the monitored quantity to qualify as an improvement.

Note

EarlyStopping will only use the values of metrics/loss obtained on validation set.

Raises

ValueError if monitor does not start with prefix val_ or train_. –

Example

import torchflare.callbacks as cbs
early_stop = cbs.EarlyStopping(monitor="val_accuracy", patience=5, mode="max")

Model Checkpoint

class torchflare.callbacks.ModelCheckpoint(mode: str, monitor: str, save_dir: str = './', file_name: str = 'model.bin')[source]

Callback for Checkpointing your model.

Parameters

  • mode – One of {“min”, “max”}. In min mode, training will stop when the quantity monitored has stopped decreasing in “max” mode it will stop when the quantity monitored has stopped increasing.

  • monitor – The quantity to be monitored. (Default : val_loss) If you want to monitor other metric just pass in the name of the metric.

  • save_dir – The directory where you want to save the model files.

  • file_name – The name of file. Default : model.bin

Note

ModelCheckpoint will save state_dictionaries for model , optimizer , scheduler and the value of epoch with following key values:

  1. ‘model_state_dict’ : The state dictionary of model

  2. ‘optimizer_state_dict’ : The state dictionary of optimizer

Model checkpoint will be saved based on the values of metrics/loss obtained from validation set.

Raises

ValueError if monitor does not start with prefix val_ or train_. –

Example

import torchflare.callbacks as cbs
model_ckpt = cbs.ModelCheckpoint(monitor="val_accuracy", mode="max")
checkpoint(model, optimizer)[source]

Method to save the state dictionaries of model, optimizer,etc.

Load Checkpoint

class torchflare.callbacks.LoadCheckpoint(path_to_model: Optional[str] = None)[source]

Class to load checkpoint.

static unpack_ckpt(nn_obj, ckpt)[source]

Method to unpack checkpoint.

Parameters

  • nn_obj – The nn.Module object.

  • ckpt – The corresponding state_dict for the object.

Learning Rate Scheduler Callbacks

Callbacks for auto adjust the learning rate based on the number of epochs or other metrics measurements. These callbacks are wrappers of native PyTorch torch.optim.lr_scheduler.

StepLR

class torchflare.callbacks.StepLR(step_size: int, gamma: float = 0.1, last_epoch: int = - 1, step_on_batch: bool = False)[source]

Multiply learning rate by a given factor with a given period.

Parameters

  • step_size (int) – Period of learning rate update in epochs.

  • gamma (float, optional) – The multiplicative factor. Defaults to 0.1.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to False.

LambdaLR

class torchflare.callbacks.LambdaLR(lr_lambda: Union[Callable[[int], float], List[Callable[[int], float]]], last_epoch: int = - 1, step_on_batch: bool = False)[source]

Multiply learning rate by a factor computed with a given function. The function should take int value number of epochs as the only argument.

Parameters

  • lr_lambda (function or list of functions) – Lambda function for the learning rate factor computation.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to False.

MultiStepLR

class torchflare.callbacks.MultiStepLR(milestones: Iterable[int], gamma: float = 0.1, last_epoch: int = - 1, step_on_batch: bool = False)[source]

Multiply learning rate by a given factor on each epoch from a given list.

Parameters

  • milestones (list of int) – List of epochs number to perform lr step.

  • gamma (float, optional) – The multiplicative factor. Defaults to 0.1.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to False.

ExponentialLR

class torchflare.callbacks.ExponentialLR(gamma: float, last_epoch: int = - 1, step_on_batch: bool = False)[source]

Multiply learning rate by a given factor on each epoch.

Parameters

  • gamma (float, optional) – The multiplicative factor. Defaults to 0.1.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to False.

CosineAnnealingLR

class torchflare.callbacks.CosineAnnealingLR(T_max: int, eta_min: float = 0, last_epoch: int = - 1, step_on_batch: bool = True)[source]

Set the learning rate of each parameter group using a cosine annealing schedule.

Parameters

  • T_max (int) – Max number of epochs or iterations.

  • eta_min (float, optional) – Min learning rate. Defaults to 0.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to True.

ReduceLROnPlateau

class torchflare.callbacks.ReduceLROnPlateau(mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)[source]

Reduce learning rate when a metric has stopped improving.

Parameters

  • mode – One of {“min”, “max”}. In min mode, training will stop when the quantity monitored has stopped decreasing.In “max” mode it will stop when the quantity monitored has stopped increasing.

  • factor (float, optional) – The multiplicative factor. Defaults to 0.1.

  • patience (int, optional) – Number of training epochs without the metric improvement to update the learning rate. Defaults to 10.

  • verbose (bool, optional) – Print info on each update to stdout. Defaults to False.

  • threshold (float, optional) – Threshold for considering the changes significant. Defaults to 1e-4.

  • threshold_mode (str, optional) – Should be ‘rel’, ‘abs’. Defaults to ‘rel’.

  • cooldown (int, optional) – Number of epochs to wait before resuming normal operation after lr has been updated. Defaults to 0.

  • min_lr (float or list of float, optional) – Min learning rate. Defaults to 0.

  • eps (float, optional) – Min significant learning rate update. Defaults to 1e-8.

CyclicLR

class torchflare.callbacks.CyclicLR(base_lr: float, max_lr: float, step_size_up: int = 2000, step_size_down: Optional[int] = None, mode: str = 'triangular', gamma: float = 1.0, scale_fn: Optional[Callable[[float], float]] = None, scale_mode: str = 'cycle', cycle_momentum: bool = True, base_momentum: float = 0.8, max_momentum: float = 0.9, last_epoch: int = - 1, step_on_batch: bool = True)[source]

Sets the learning rate of each parameter group according to cyclical learning rate policy.

Parameters

  • base_lr (float or list of float) – Initial learning rate.

  • max_lr (float or list of float) – Max learning rate.

  • step_size_up (int, optional) – Increase phase duration in epochs or iterations. Defaults to 2000.

  • step_size_down (int, optional) – Decrease phase duration in epochs or iterations. Defaults to None.

  • mode (str, optional) – Should be ‘triangular’, ‘triangular2’ or ‘exp_range’. Defaults to ‘triangular’.

  • gamma (float, optional) – Constant for the ‘exp_range’ policy. Defaults to 1.

  • scale_fn (function, optional) – Custom scaling policy function. Defaults to None.

  • scale_mode (str, optional) – Should be ‘cycle’ or ‘iterations’. Defaults to ‘cycle’.

  • cycle_momentum (bool, optional) – Momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Defaults to True.

  • base_momentum (float or list of float, optional) – Lower momentum boundaries in the cycle for each parameter group. Defaults to 0.8.

  • max_momentum (float or list of float, optional) – Upper momentum boundaries in the cycle for each parameter group. Defaults to 0.9.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to True.

CosineAnnealingWarmRestarts

class torchflare.callbacks.CosineAnnealingWarmRestarts(T_0: int, T_mult: int = 1, eta_min: int = 0, last_epoch: int = - 1, step_on_batch: bool = True)[source]

Set the learning rate of each parameter group using a cosine annealing schedule with a warm restart.

Parameters

  • T_0 (int) – Number of epochs or iterations for the first restart.

  • T_mult (int) – T increase factor after a restart.

  • eta_min (float, optional) – Min learning rate. Defaults to 0.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to True.

MultiplicativeLR

class torchflare.callbacks.MultiplicativeLR(lr_lambda: Union[Callable[[int], float], List[Callable[[int], float]]], last_epoch: int = - 1, step_on_batch: bool = False)[source]

Multiply the learning rate of each parameter group by the factor given in the specified function.

Parameters

  • lr_lambda (function or list of functions) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in an optimizer.param_groups.

  • last_epoch (int) – The index of last epoch. Default: -1.

  • step_on_batch (bool) – Step on each training iteration rather than each epoch. Defaults to False.

OneCycleLR

class torchflare.callbacks.OneCycleLR(max_lr: Union[float, List[float]], total_steps: Optional[int] = None, epochs: Optional[int] = None, steps_per_epoch: Optional[int] = None, pct_start: float = 0.3, anneal_strategy: str = 'cos', cycle_momentum: bool = True, base_momentum: Union[float, List[float]] = 0.85, max_momentum: Union[float, List[float]] = 0.95, div_factor: float = 25.0, final_div_factor: float = 10000.0, last_epoch: int = - 1)[source]

Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate.

Parameters

  • max_lr (float or list of float) – Upper learning rate boundaries in the cycle for each parameter group.

  • total_steps (int) – The total number of steps in the cycle. Note that if a value is not provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Defaults to None.

  • epochs (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Defaults to None.

  • steps_per_epoch (int) – The number of steps per an epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Defaults to None.

  • pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Defaults to 0.3.

  • anneal_strategy (str) – {‘cos’, ‘linear’} Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Defaults to ‘cos’.

  • cycle_momentum (bool) – If True, momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Defaults to True.

  • base_momentum (float or list of float) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Defaults to 0.85.

  • max_momentum (float or list of float) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Defaults to 0.95.

  • div_factor (float) – Determines the initial learning rate via initial_lr = max_lr/div_factor Defaults to 25.

  • final_div_factor (float) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Defaults to 1e4.

  • last_epoch (int) – The index of last epoch. Default: -1.

Logging Callbacks

Logging module integrates various logging services like neptune, comet, etc for logging the progress of experiments.

Comet Logger

class torchflare.callbacks.CometLogger(api_token: str, params: dict, project_name: str, workspace: str, tags: List[str])[source]

Callback to log your metrics and loss values to Comet to track your experiments. For more information about Comet look at [Comet.ml](https://www.comet.ml/site/)

Parameters

  • api_token – Your API key obtained from comet.ml

  • params – The hyperparameters for your model and experiment as a dictionary

  • project_name – Send your experiment to a specific project. Otherwise, will be sent to Uncategorized Experiments.

  • workspace – Attach an experiment to a project that belongs to this workspace

  • tags – List of strings.

Examples

from torchflare.callbacks import CometLogger

params = {"bs": 16, "lr": 0.3}

logger = CometLogger(
    project_name="experiment_10",
    workspace="username",
    params=params,
    tags=["Experiment", "fold_0"],
    api_token="your_secret_api_token",
)

Neptune Logger

class torchflare.callbacks.NeptuneLogger(project_dir: str, api_token: str, params: Optional[dict] = None, experiment_name: Optional[str] = None, tags: Optional[List[str]] = None)[source]

Callback to log your metrics and loss values to Neptune to track your experiments.

For more information about Neptune take a look at [Neptune](https://neptune.ai/)

Parameters

  • project_dir – The qualified name of a project in a form of namespace/project_name

  • params – The hyperparameters for your model and experiment as a dictionary

  • experiment_name – The name of the experiment

  • api_token – User’s API token

  • tags – List of strings.

Examples

from torchflare.callbacks import NeptuneLogger

params = {"bs": 16, "lr": 0.3}

logger = NeptuneLogger(
    project_dir="username/Experiments",
    params=params,
    experiment_name="Experiment_10",
    tags=["Experiment", "fold_0"],
    api_token="your_secret_api_token",
)

Wandb Logger

class torchflare.callbacks.WandbLogger(project: str, entity: str, name: Optional[str] = None, config: Optional[Dict] = None, tags: Optional[List[str]] = None, notes: Optional[str] = None, directory: Optional[str] = None)[source]

Callback to log your metrics and loss values to wandb platform.

For more information about wandb take a look at [Weights and Biases](https://wandb.ai/)

Parameters

  • project – The name of the project where you’re sending the new run

  • entity – An entity is a username or team name where you’re sending runs.

  • name – A short display name for this run

  • config – The hyperparameters for your model and experiment as a dictionary

  • tags – List of strings.

  • directory – where to save wandb local run directory. If set to None it will use experiments save_dir argument.

  • notes – A longer description of the run, like a -m commit message in git

Note

set os.environ[‘WANDB_SILENT’] = True to silence wandb log statements. If this is set all logs will be written to WANDB_DIR/debug.log

Examples

from torchflare.callbacks import WandbLogger

params = {"bs": 16, "lr": 0.3}

logger = WandbLogger(
    project="Experiment",
    entity="username",
    name="Experiment_10",
    config=params,
    tags=["Experiment", "fold_0"])

Tensorboard Logger

class torchflare.callbacks.TensorboardLogger(log_dir: str)[source]

Callback to use Tensorboard to log your metrics and losses.

Parameters

log_dir – The directory where you want to save your experiments.

Message Notifiers

Message notifiers send training progress per epoch to your personal slack and discord channels.

Discord Notifier Callback

class torchflare.callbacks.DiscordNotifierCallback(exp_name: str, webhook_url: str)[source]

Class to Dispatch Training progress.

Parameters

  • exp_name – The name of your experiment bot. (Can be anything)

  • webhook_url – The webhook url of your discord server/channel.

Examples

import torchflare.callbacks as cbs

discord_notif = cbs.DiscordNotifierCallback(
    webhook_url="YOUR_SECRET_URL", exp_name="MODEL_RUN"
)

Slack Notifier Callback

class torchflare.callbacks.SlackNotifierCallback(webhook_url: str)[source]

Class to Dispatch Training progress to your Slack channel.

Parameters

webhook_url – Slack webhook url.

Examples

import torchflare.callbacks as cbs
slack_notif = cbs.SlackNotifierCallback(webhook_url="YOUR_SECRET_URL")

Callback Decorators

The main callback decorators simply take function , order as inputs and bind it to a callback point, returning the result.

on_experiment_start

torchflare.callbacks.on_experiment_start(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_experiment_start is used to initialise a class with method Callbacks.on_experiment_start.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_experiment_start, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_experiment_start(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Experiment started")
#This callback will print "Experiment started" at the start of experiment.
Returns

Initialised callback with method Callbacks.on_experiment_start.

on_epoch_start

torchflare.callbacks.on_epoch_start(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_epoch_start is used to initialise a class with method Callbacks.on_epoch_start.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_epoch_start, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_epoch_start(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Epoch started")

#This callback will print "Epoch started" at the start of epoch.
Returns

Initialised callback with method Callbacks.on_epoch_start.

on_loader_start

torchflare.callbacks.on_loader_start(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_loader_start is used to initialise a class with method Callbacks.on_loader_start.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_loader_start, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_loader_start(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Loader started")

#This callback will print "Loader started" at the start of loader.
Returns

Initialised callback with method Callbacks.on_loader_start.

on_batch_start

torchflare.callbacks.on_batch_start(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_batch_start is used to initialise a class with method Callbacks.on_batch_start.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_batch_start, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_batch_start(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Batch started")

#This callback will print "Batch started" at the start of batch.
Returns

Initialised callback with method Callbacks.on_batch_start.

on_experiment_end

torchflare.callbacks.on_experiment_end(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_experiment_end is used to initialise a class with method Callbacks.on_experiment_end.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_experiment_end, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_experiment_end(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Experiment Ended")

#This callback will print "Experiment Ended" at the end of experiment.
Returns

Initialised callback with method Callbacks.on_experiment_end.

on_epoch_end

torchflare.callbacks.on_epoch_end(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_epoch_end is used to initialise a class with method Callbacks.on_epoch_end.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_experiment_end, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_epoch_end(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Epoch Ended")

#This callback will print "Epoch Ended" at the end of epoch.
Returns

Initialised callback with method Callbacks.on_epoch_end.

on_loader_end

torchflare.callbacks.on_loader_end(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_loader_end is used to initialise a class with method Callbacks.on_loader_end.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_loader_end, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_loader_end(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("loader Ended")

#This callback will print "loader Ended" at the end of loader.
Returns

Initialised callback with method Callbacks.on_loader_end.

on_batch_end

torchflare.callbacks.on_batch_end(order: torchflare.callbacks.states.CallbackOrder)[source]

The method on_batch_end is used to initialise a class with method Callbacks.on_batch_end.

Parameters

order – The callback order.

Example

from torchflare.callbacks import on_batch_end, CallbackOrder
from torchflare.experiments import Experiment

#Creating a custom callback using the decorator.

@on_batch_end(order=CallbackOrder.MODEL_INIT)
def print_on_start(experiment : "Experiment"):
    print("Batch Ended")

#This callback will print "Batch Ended" at the end of batch.
Returns

Initialised callback with method Callbacks.on_batch_end.