Ignite Simple¶

ignite_simple.analarams¶

Describes parameters relevant for analysis. The parameters are specified for the output, and what’s required to produce that output is calculated from that.

class ignite_simple.analarams.AnalysisSettings(hparam_selection_specifics: bool, hparam_selection_specific_imgs: bool, training_metric_imgs: bool, suppress_extra_images: bool, typical_run_pca3dvis: bool, typical_run_pca3dvis_draft: bool)[source]¶

Describes the analysis settings. Note that, where relevant and equivalent, parallel computations are described as if they were performed sequentially.

Variables

hparam_selection_specifics (bool) – True if the output should include a text explanation about how specifically we came about selecting the learning rate and batch size, by going through the numbers, False otherwise.
hparam_selection_specific_imgs (bool) – True if the output should include figures with captions for the hyper paramater selection process as it occurred for this model specifically, False otherwise.
training_metric_imgs (bool) – True if the output should include figures with captions for the performance of the model through training, i.e., accuracy vs epoch and loss vs epoch figures.
suppress_extra_images (bool) – Suppresses generating images which are not displayed in the html report.
typical_run_pca3dvis (bool) – True if the output should include a video from pca3dvis for the points as they moved through the network, False otherwise.
typical_run_pca3dvis_draft (bool) – If typical_run_pca3dvis is True, then the value of typical_run_pca3dvis_draft corresponds to if the video should have draft settings. If typical_run_pca3dvis is False, this has no effect.

ignite_simple.analarams.animations() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text, images, and animations. Animations are videos which are under 15 seconds for the purpose of this module.

Returns: the animations preset for analysis settings, which produces text, images, and animations in the output.
Return type: AnalysisSettings

ignite_simple.analarams.animations_draft() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text, images, and animations but the animations are given draft settings (i.e. low-fps and low resolution) to speed up output. For the purposes of this module, animations are simply videos under 15 seconds.

Returns: the animations-draft preset for analysis settings, which produces text, image, and draft-quality animations in the output.
Return type: AnalysisSettings

ignite_simple.analarams.get_settings(preset: Union[str, ignite_simple.analarams.AnalysisSettings]) → ignite_simple.analarams.AnalysisSettings[source]¶

Gets the analysis settings from the given preset name or analysis settings.

Parameters: preset – either a str name of a preset or the settings to return
Returns: the corresponding preset or just the argument if its already AnalysisSettings

ignite_simple.analarams.images() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text and images only

Returns: the images preset for analysis settings, which produces text and image output.
Return type: AnalysisSettings

ignite_simple.analarams.images_minimum() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text and images only, and does not produce images which do not make it into the text report.

Returns: the images_minimum preset for analysis settings, which produces text and enough images for a pretty text output.
Return type: AnalysisSettings

ignite_simple.analarams.none() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis settings that produce absolutely nothing

Returns: the none preset for analysis settings, which produces no output
Return type: AnalysisSettings

ignite_simple.analarams.text() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text only

Returns: the text preset for analysis settings, which produces text output.
Return type: AnalysisSettings

ignite_simple.analarams.video() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text, images, animations, and video

Returns: the video preset for analysis settings, which produces text, images, animations, and videos in the output.
Return type: AnalysisSettings

ignite_simple.analarams.video_draft() → ignite_simple.analarams.AnalysisSettings[source]¶

Analysis output that uses text, images, animations, and video but the animations and video(s) are given draft settings (i.e. low-fps and low-resolution) to speed up output

Returns: the video-draft preset for analysis settings, which produces text, images, draft-quality animations, and draft-quality videos in the output.
Return type: AnalysisSettings

ignite_simple.hyperparams¶

This module is used to describe the hyperparameter tuning settings and presets used for training.

class ignite_simple.hyperparams.HyperparameterSettings(lr_start: float, lr_end: float, lr_min_inits: int, batch_start: int, batch_end: int, batch_rn_min_inits: int, batch_pts: int, batch_pt_min_inits: int, rescan_lr_after_bs: bool)[source]¶

Describes settings for tuning hyperparameters

Variables

lr_start (float) – the smallest learning rate that is checked
lr_end (float) – the largest learning rate that is checked
lr_min_inits (int) – the minimum number of model initializations that are averaged together and then smoothed to get the lr-vs-accuracy plot. Note that when multiple physical cores are available they will be utilized since this process is well-suited to parallelization
batch_start (int) – the smallest batch size that is checked during the initial reasonableness sweep (a single pass)
batch_end (int) – the largest batch size that is checked during the initial reasonableness sweep (a single pass)
batch_rn_min_inits (int) – the minimum number of model initializations that are averaged together then smoothed to get the batch-vs-accuracy plot.
batch_pts (int) – the number of different batch sizes which are checked, points sampled from a distribution weighted toward a higher first derivative of performance. Must be either 0 or greater than 1. If 0, the batch size corresponding to the greatest increase in accuracy during the reasonableness sweep is used.
batch_pt_min_inits (int) – the minimum number of model initializations that are combined together via LogSumExp. We use LogSumExp instead of mean because we care more about the best performance than the most consistent performance when selecting batch size. If you want more motivation, we prefer a final accuracy of [0, 1, 1] over 3 trials to [2/3, 2/3, 2/3] even though the mean is the same
rescan_lr_after_bs (bool) – if True, the learning rate is scanned once more after we tweak the batch size. otherwise, we use the same ratio of learning rate to batch size as we found in the first sweep.

ignite_simple.hyperparams.fast() → ignite_simple.hyperparams.HyperparameterSettings[source]¶: Returns a reasonably fast (in time spent tuning parameters) preset

ignite_simple.hyperparams.fastest() → ignite_simple.hyperparams.HyperparameterSettings[source]¶: Returns the fastest (in time spent tuning parameters) preset

ignite_simple.hyperparams.get_settings(preset: Union[str, ignite_simple.hyperparams.HyperparameterSettings])[source]¶

Gets the corresponding preset if the argument is a name of one, returns the argument directly if the argument is already a settings object, and raises an exception in all other circumstances.

Parameters: preset – the name for a preset or the complete settings object
Returns: the corresponding preset or the settings object passed in

ignite_simple.hyperparams.slow() → ignite_simple.hyperparams.HyperparameterSettings[source]¶: Returns a somewhat slow (in time spent tuning parameters) preset

ignite_simple.hyperparams.slowest() → ignite_simple.hyperparams.HyperparameterSettings[source]¶: Returns the slowest (in time spent tuning parameters) preset

ignite_simple.model_manager¶

This module is meant to be responsible for all of the training performed on an identical model, dataset, and loss. Specifically, it decides on the folder structure, collates results for analysis, and handles archiving old data.

ignite_simple.model_manager.train(model_loader: Tuple[str, str, tuple, dict], dataset_loader: Tuple[str, str, tuple, dict], loss_loader: Tuple[str, str, tuple, dict], folder: str, hyperparameters: Union[str, ignite_simple.hyperparams.HyperparameterSettings], analysis: Union[str, ignite_simple.analarams.AnalysisSettings], allow_later_analysis_up_to: Union[str, ignite_simple.analarams.AnalysisSettings], accuracy_style: str, trials: int, is_continuation: bool, history_folder: str, cores: Union[str, int], trials_strict: bool = False) → None[source]¶

Trains the given model on the given dataset with the given loss by finding and saving or loading the hyperparameters with the given settings, performing the given analysis but gathering sufficient data to later analyze up to the specified amount.

The folder structure is as follows:

folder/
    hparams/
        the result from tuner.tune
    trials/
        i/  (where i=0,1,...)
            result.json
                note that for classification tasks perf is accuracy,
                and in other tasks it is inverse loss. Note that perf
                is always available and higher is better, whereas for
                loss lower is better.

                {'loss_train': float, 'loss_val': float,
                 'perf_train': float, 'perf_val': float}
            model_init.pt
                the initial random initialization of the model, saved
                with torch.save
            model.pt
                the model after training, saved with torch.save
            throughtime.npz
                this is only stored if storing training_metric_imgs.

                settings:
                    shape (1,), where the values are:
                        - number of points randomly selected from the
                          corresponding dataset to calculate metrics

                epochs:
                    the partial epoch number for the samples

                losses_train:
                    the loss for the corresponding epoch, same shape as
                    epochs, for the training dataset

                losses_val:
                    the loss for the corresponding epoch, same shape as
                    epochs, for the validation dataset

                perfs_train:
                    in classification tasks this is fractional accuracy
                    in other tasks, this is inverse loss

                    the performance at the corresponding epoch, same
                    shape as epochs, for the training dataset

                perfs_val:
                    in classification tasks this is fractional accuracy
                    in other tasks, this is inverse loss

                    the performance at the corresponding epoch, same
                    shape as epochs, for the validation dataset
    results.npz
        The trials/result.json except concatenated together for easier
        loading

        final_loss_train:
            shape (trials,)

        final_loss_val:
            shape (trials,)

        final_perf_train:
            shape (trials,)

        final_perf_val:
            shape (trials,)

    throughtimes.npz
        the trials/throughtime.npz, if available, stacked for easier
        loading

        settings:
            shape (1,) the number of points used for gathering metrics
            at each sample

        epochs:
            shape (samples,) the epoch that corresponds to each sample
            during training. this is a float, since we may sample
            multiple times per epoch

        losses_train:
            shape (trials, samples)

        losses_train_smoothed:
            shape (trials, samples)

        losses_val:
            shape (trials, samples)

        losses_val_smoothed:
            shape (trials, samples)

        perfs_train:
            shape (trials, samples)

        perfs_train_smoothed:
            shape (trials, samples)

        perfs_val:
            shape (trials, samples)

        perfs_val_smoothed:
            shape (trials, samples)

Parameters

model_loader – (module, attrname, args, kwargs) defines where the callable which returns a torch.nn.Module can be found, and what arguments to pass to the callable to get the module. The callable should return a random initialization of the model.
dataset_loader – (module, attrname, args, kwargs) defines where the callable which returns (train_set, val_set) can be found, and what arguments to pass to the callable to get the datasets
loss_loader – (module, attrname, args, kwargs) defines where the callable which returns the torch.nn.Module that computes a scalar value which ought to be minimized, and what arguments to pass the callable for the loss.
folder – the folder where the output should be stored
hyperparameters – the hyperparameter settings or a name of a preset (one of fastest, fast, slow, and slowest)
analysis – the analysis settings or a name of a preset (typically one of none, text, images, animations, videos), for a complete list see ignite_simple.analarams. It is always equivalent to set this value to none and then call analysis.reanalyze with the desired analysis
allow_later_analysis_up_to – this is also an analysis settings or name of a preset, except this analysis isn’t produced but instead we ensure that sufficient data is collected to perform this analysis if desired later. it must be at least as high as analysis
accuracy_style – one of classification, multiclass, and inv-loss to describe how performance is measured. classification assumes one-hot labels for the output, multiclass assumes potentially multiple ones in the labels, and inv-loss uses 1/loss as the performance metric instead.
trials – the number of trials which should be formed with the found settings
is_continuation – if True then if folder already exists then it is assumed to have been the result of this function called with the same parameters except possible trials, and the result will be the sum of the existing trials plus the new trials to perform. If this is False and the folder already exists, it will be moved into history_folder where the name is the current timestamp.
history_folder – where to store the old folders if they are found when is_continuation is False.
cores – either an integer for the number of physical cores that are available for training, or the string ‘all’ for the number of cores to be auto-detected and used.
trials_strict – if False, then this will use all available resources to compute trials such that this completes in approximately the minimum amount of time to produce the required number of trials. This may result in more than the specified number of trials being run. If True, exactly trial runs will be performed regardless of the amount of available computational resources (i.e., available cores may be unused)

ignite_simple.trainer¶

This module manages preparing and running the training environment for given settings.

class ignite_simple.trainer.TrainSettings(accuracy_style: str, model_loader: Tuple[str, str, tuple, dict], loss_loader: Tuple[str, str, tuple, dict], task_loader: Tuple[str, str, tuple, dict], handlers: Tuple[Tuple[str, Tuple[str, str, tuple, dict]]], initializer: Optional[Tuple[str, str, tuple, dict]], lr_start: float, lr_end: float, cycle_time_epochs: int, epochs: int)[source]¶

Describes the settings which ultimately go into a training session. This is intended to be trivially serializable, in that all attributes are built-ins that can be json serialized. The train function here runs in the same process, but this strategy allows us to use the same interface design throughout and allows repeating / printing training sessions trivially.

Variables

accuracy_style (str) –
one of the following constants:
- classification
  labels are one-hot encoded classes, outputs are one-hot encoded classes.
- multiclass
  labels are one-hot encoded multi-class labels, outputs are the same
- inv-loss
  accuracy is not measured and inverse loss is used as the performance metric instead. For stability, and legibility of plots,
  
  $\frac{1}{\text{loss} + 1}$
  
  is used.
str, tuple, dict] model_loader (tuple[str,) – the tuple contains the module and corresponding attribute name for a function which returns the nn.Module to train. The module must have the calling convention model(inp) -> out. The next two arguments are the args and keyword args to the callable respectively.
str, tuple, dict] loss_loader (tuple[str,) – the tuple contains the model and corresponding attribute name for a function which returns the loss function to minimize.
str, tuple, dict] task_loader (tuple[str,) – the tuple contains the module and corresponding attribute name for a function which returns (train_set, val_set, train_loader), each as described in TrainState. The next two arguments are the args and keyword args to the callable respectively.
tuple[str, str, tuple, dict]]] handlers (tuple[tuple[str,) –
the event handlers for the engine which will perform training. After the specified positional arguments, the handlers will be passed the Engine that is training the model and the TrainState that is in use. The str associated with each callable is the event that each callable listens to.
str, tuple, dict] initializer (tuple[str,) – this is called with trainer as the next positional argument. May be used to attach additional events to the trainer.
lr_start (float) – the learning rate at the start of each cycle
lr_end (float) – the learning rate at the end of each cycle
cycle_time_epochs (int) – the number of epochs for the learning rate scheduler
epochs (int) – the number of epochs to train for

get_handlers() → Tuple[Tuple[str, Callable]][source]¶: This returns handlers except instead of the function descriptions (module, attribute, args, kwargs), actual callables are provided with the necessary arguments and keyword arguments already bound.

get_initializer() → Callable[source]¶: This returns the initializer; if it is not specified this is a no-op. Otherwise, this is the callable which accepts the trainer and initializes it, with the other arguments and keyword arguments already bound.

get_loss_loader() → Callable[source]¶: Gets the actual loss loader callable, which is defined through loss_loader. This is a callable which returns the torch.nn.Module that goes from the output of the model to a scalar which should be minimized.

get_model_loader() → Callable[source]¶: Gets the actual model loader callable, which is defined through model_loader. This is a callable which returns the torch.nn.Module to train. The resulting callable already has the required arguments and keyword arguments bound.

get_task_loader() → Callable[source]¶: Gets the actual task loader callable, which is defined through task_loader. This is a callable which returns (train_set, val_set, train_loader), each as defined in TrainState. The resulting callable already has the required arguments and keyword arguments bound.

class ignite_simple.trainer.TrainState(model: torch.nn.modules.module.Module, unstripped_model: Optional[torch.nn.modules.module.Module], train_set: torch.utils.data.dataset.Dataset, val_set: torch.utils.data.dataset.Dataset, train_loader: torch.utils.data.dataloader.DataLoader, optimizer: torch.optim.optimizer.Optimizer, cycle_time_epochs: int, lr_scheduler: ignite.contrib.handlers.param_scheduler.CyclicalScheduler, loss: torch.nn.modules.module.Module, evaluator: ignite.engine.engine.Engine)[source]¶

Describes the state which is passed as the second positional argument to each event handler, which contains generic information about the training session that may be useful.

Variables

model (torch.nn.Module) – the model which is being trained
unstripped_model (optional[torch.nn.Module]) – the unstripped model, if there is one, otherwise just the same reference as model
train_set (torch.utils.data.Dataset) – the dataset which is used to train the model
val_set (torch.utils.data.Dataset) – the dataset which is used to validate the models performance on unseen / held out data.
train_loader (torch.utils.data.DataLoader) – the dataloader which is being used to generate batches from the train set to be passed into the model. This incorporates the batch size.
optimizer (torch.optim.Optimizer) – the optimizer which is used to update the parameters of the model.
cycle_time_epochs (int) – the number of epochs in a complete cycle of the learning rate, always even.
lr_scheduler (ignite.contrib.handlers.param_scheduler.CyclicalScheduler) – the parameter scheduler for the learning rate. Its instance values can be used to get the learning rate range and length in batches.
loss (torch.nn.Module) – the loss function, which accepts (input, target) and returns a scalar which is to be minimized.
evaluator (ignite.engine.Engine) – the engine which can be used to gather metrics. Always has a 'loss' and 'perf' metric, but may or may not have an 'accuracy' metric.

ignite_simple.trainer.train(settings: ignite_simple.trainer.TrainSettings) → None[source]¶

Trains a model with the given settings.

Note

In order to store anything you will need to use a handler. For example, a handler for ignite.engine.Event.COMPLETED and stores the model somewhere.

Parameters: settings (TrainSettings) – The settings to use for training

ignite_simple.analysis¶

This module is tasked with generating analysis text, images, animations, and videos using only the output from the model manager, the dataset, the loss metric, and the accuracy style.

ignite_simple.analysis.analyze(dataset_loader: Tuple[str, str, tuple, dict], loss_loader: Tuple[str, str, tuple, dict], folder: str, settings: Union[str, ignite_simple.analarams.AnalysisSettings], accuracy_style: str, cores: int)[source]¶

Performs the requested analysis of the given folder, which is assumed to have been generated from model_manager.train. The output folder structure is as follows:

folder/
    analysis/
        hparams/
            lr/
                i/  (where i=0,1,...)
                    lr_vs_perf.(img)
                    lr_vs_smoothed_perf.(img)
                    lr_vs_perf_deriv.(img)
                    lr_vs_smoothed_perf_deriv.(img)

                lr_vs_perf_*.(img)
                lr_vs_smoothed_perf_*.(img)
                lr_vs_perf_deriv_*.(img)
                lr_vs_smoothed_perf_deriv_*.(img)
                lr_vs_lse_smoothed_perf_then_deriv_*.(img)
            batch/
                i/ (where i=0,1,...)
                    batch_vs_perf.(img)
                    batch_vs_smoothed_perf.(img)
                    batch_vs_perf_deriv.(img)
                    batch_vs_smoothed_perf_deriv.(img)

                batch_vs_perf_*.(img)
                batch_vs_smoothed_perf_*.(img)
                batch_vs_perf_derivs_*.(img)
                batch_vs_smoothed_perf_derivs_*.(img)
                batch_vs_lse_smoothed_perf_then_derivs_*.(img)

            TODO videos & animations
        trials/
            i/  (where i=0,1,...)
                epoch_vs_loss_train.(img) (*)
                epoch_vs_loss_val.(img) (*)
                epoch_vs_perf_train.(img) (*)
                epoch_vs_perf_val.(img) (*)

                pca3dvis_train_draft/
                    Only produced if settings.typical_run_pca3dvis and
                    settings.typical_run_pca3dvis_draft are set, and
                    only done on trial 0
                pca3dvis_train/
                    Only produced if settings.typical_run_pca3dvis and
                    not settings.typical_run_pca3dvis_draft, and only
                    done on trial 0

            epoch_vs_loss_train_*.(img) (*)
            epoch_vs_loss_val_*.(img) (*)
            epoch_vs_smoothed_loss_train_*.(img) (*)
            epoch_vs_smoothed_loss_val_*.(img) (*)
            epoch_vs_perf_train_*.(img) (*)
            epoch_vs_smoothed_perf_train_*.(img) (*)
            epoch_vs_perf_val_*.(img) (*)
            epoch_vs_smoothed_perf_val_*.(img) (*)


            (*)
                Only produced if throughtime.npz is available for
                the trial and settings.training_metric_images is
                set

            TODO more summary of trials
            TODO text & videos & animations

        html/
            See text_analysis.py for details

Parameters

dataset_loader – the module and corresponding attribute that gives a training and validation dataset when invoked with the specified arguments and keyword arguments.
loss_loader – the module and corresponding attribute that gives a loss nn.Module when invoked with the specified arguments and keyword arguments
folder – where the model manager saved the trials to analyze
settings – the settings for analysis, or the name of the preset to use. Common preset names are none, text, images, animations, and videos. For the full list, see the ignite_simple.analarams module.
accuracy_style – how performance was calculated. one of ‘classification’, ‘multiclass’, and ‘inv-loss’. See train for details.
cores – the number of physical cores this can assume are available to speed up analysis.

ignite_simple.text_analysis¶

Produces the html folder for analysis. This is essentially just copying the html/ folder and preprocessing certain ids.

ignite_simple.text_analysis.find_referenced_images(html)[source]¶: Searches the given html file for images which are referenced in the standard way (src attribute on img tag) and returns them as they are written.

ignite_simple.text_analysis.text_analyze(settings: ignite_simple.analarams.AnalysisSettings, folder: str)[source]¶

Analyzes the given folder produced by the model manager, storing the results in folder/analysis/html

Args:: settings (AnalysisSettings): the analysis settings. folder (str): the folder that was passed to the model manager.

ignite_simple.range_finder¶

This module is responsible for finding a good range of values within which a measure is increasing most rapidly. This works best when the y-values have already been smoothed.

ignite_simple.range_finder.autosmooth(arr: numpy.ndarray, axis: int = -1)[source]¶

Causes scipy.signal.savgol_filter on the given data for the given dimension with the default settings.

Parameters

arr – the array to smooth
dim – the smooth dimension

ignite_simple.range_finder.find_with_derivs(xs: numpy.ndarray, derivs: numpy.ndarray) → Tuple[float, float][source]¶

Finds the range in derivs wherein the derivative is always positive. From these intervals, this returns specifically the one with the greatest integral.

Parameters

xs (np.ndarray) – with shape (points,), the x-values corresponding to the derivatives
derivs (np.ndarray) – with shape (points,), the relevant derivatives

Returns

the (min, max) for the best interval in xs that has positive derivative in ys. Where multiple such intervals exist, this is the one with the greatest integral

ignite_simple.range_finder.nonzero_intervals(vec: numpy.ndarray) → numpy.ndarray[source]¶

Find which intervals in the given vector are non-zero.

Adapted from https://stackoverflow.com/a/27642744

Parameters: vec (np.ndarray) – the vector to scan for non-zero intervals
Returns: a list [x1, x2, …, xn] such that [xi, xi+1) is a nonzero interval in vec

ignite_simple.range_finder.smooth_window_size(npts: int) → int[source]¶

Returns the a heuristic for the window size for smoothing an array with the given number of points. Specifically, this is 1/10 the number of points or 101, whichever is smaller.

Parameters: npts (int) – the number of data points you have
Returns: suggested window size for smoothing the data

ignite_simple.range_finder.trim_range_derivs(xs: numpy.ndarray, derivs: numpy.ndarray, x_st_ind: int, x_en_ind: int) → Tuple[int, int][source]¶

Given that we have found a good interval in the xs for which the derivs is positive, it may be possible that the bulk of the integral can be maintained while reducing the width of the integral. This function returns a new interval which is a subset of the given interval for which the bulk of the integral is maintained.

Parameters

xs (np.ndarray) – with shape (points,), the x-values corresponding to the derivatives
derivs (np.ndarray) – with shape (points,), the relevant derivatives
x_st_ind (int) – the index in xs the good interval starts at
x_en_ind (int) – the index in xs the good interval ends at

Returns

(x_st_ind, x_end_ind) which is a subset of the passed interval

ignite_simple.tuner¶

This module is responsible for tuning the learning rate and batch size for training a module.

ignite_simple.tuner.tune(model_loader: Tuple[str, str, tuple, dict], dataset_loader: Tuple[str, str, tuple, dict], loss_loader: Tuple[str, str, tuple, dict], accuracy_style: str, folder: str, cores: int, settings: ignite_simple.hyperparams.HyperparameterSettings, store_up_to: ignite_simple.analarams.AnalysisSettings, logger: logging.Logger = None)[source]¶

Finds the optimal learning rate and batch size for the specified model on the specified dataset trained with the given loss. Stores the following information:

folder/
    final.json
        {'lr_start': float, 'lr_end': float, 'batch_size': float,
         'cycle_size_epochs': int, 'epochs': int}
    misc.json
        Variables that went into the final output. Typically selected
        via heuristics, constants, or come from the hyperparameter
        settings. Some may be deduced from the numpy array files
        directly

        {
            'initial_batch_size': int,
            'initial_cycle_time': int,
            'initial_min_lr': float,
            'initial_max_lr': float,
            'initial_lr_num_to_val': int,
            'initial_lr_num_trials': int,
            'initial_lr_window_size': int,
            'initial_lr_sweep_result_min': float,
            'initial_lr_sweep_result_max': float,
            'second_min_lr': float,
            'second_max_lr': float
        }

    lr_vs_perf.npz
        lrs=np.ndarray[number of batches]
        perfs=np.ndarray[trials, number of batches]
        smoothed_perfs=np.ndarray[trials, number of batches]
        lse_smoothed_perfs=np.ndarray[trials, number of batches]
        perf_derivs=np.ndarray[trials, number_of_batches]
        smoothed_perf_derivs=np.ndarray[trials, number of batches]
        mean_smoothed_perf_derivs=np.ndarray[number of batches]
        lse_smoothed_perf_then_derivs=np.ndarray[number of batches]
            lse = log sum exp. when there are many trials, the mean
            gets overly pessimistic from bad initializations,
            so LSE is more stable. however, we can't do lse on the
            smoothed derivatives because then derivatives will tend
            to be positive everywhere, so we have to smooth first,
            then take lse, then take derivative
        lr_range=np.ndarray[2]
            min, max for the good range of learning rates
    bs_vs_perf.npz  (bs=batch_size)
        Where a single batch is tried multiple times, we take the
        mean over those times to ensure bss contains only unique
        values and hence can be treated like lrs

        bss=np.ndarray[number of batches]
        perfs=np.ndarray[trials, number of batches]
        smoothed_perfs=np.ndarray[number of batches]
        lse_smoothed_perfs=np.ndarray[number of batches]
        perf_derivs=np.ndarray[trials, number_of_batches]
        smoothed_perf_derivs=np.ndarray[trials, number of batches]
        mean_smoothed_perf_derivs=np.ndarray[number of batches]
        lse_smoothed_perf_then_derivs=np.ndarray[number of batches]
        bs_range=np.ndarray[2]
            min, max for the good range of batch sizes
    lr_vs_perf2.npz
        only stored if settings.rescan_lr_after_bs. looks exactly
        like lr_vs_perf.npz, except these runs are performed with
        the newly selected batch size
    bs_sampled.npz
        only stored if settings.batch_pts > 0

        bss=np.ndarray[num bs attempted]
        final=np.ndarray[num bs attempted, trials]
            final performance for batch size i for each trial
        lse_final=np.ndarray[num bs attempted]
            final logsumexp performance for each batch size, argmax
            is the selected batch size. If you want this to nicely
            be below the maximum, subtract log(trials) and note
            this does not effect the argmax

        raw_i=np.ndarray[trials, number of batches]
            only if store_up_to.hparam_selection_specific_imgs,
            same for the *_raw_i

            i is a sampled batch size and raw_i[t, j] is the
            performance of the model after iteration j for
            batch size i on trial t.
        smoothed_raw_i=np.ndarray[trials, number of batches]
        lse_smoothed_raw_i=np.ndarray[number of batches]

Parameters

model_loader –
describes which module and corresponding attribute can be passed what arguments and keyword arguments to produce the nn.Module with a random initialization which can be trained
dataset_loader – describes which module and corresponding attribute can be passed what arguments and keyword arguments to produce the training dataset and validation dataset.
loss_loader – describes which module and corresponding attribute can be passed what arguments and keyword arguments to produce the nn.Module that converts (y_pred, y) to a scalar which should be minimized
folder – where to save the output to
cores – how many cores to use; 1 for just the main process
settings – the settings to use to tune the learning rate and batch size
store_up_to – the information stored should be at least what is required to produce this analysis

ignite_simple.helper¶

This acts as a potential runner for files, or as an import to reduce the amount of boilerplate in a runner. Given a module which has a model() function, dataset() function, and loss() function this uses argparse to fill in the rest of the parameters to train() or reanalyze() as requested.

Example

# in file mymod.py
import ignite_simple.helper
import torch

def model():
    pass # omitted, should return torch.nn.Module

def dataset():
    pass # omitted, return train_set, val_set

accuracy_style = 'multiclass'
loss = torch.nn.MSELoss # any callable that returns a loss works

if __name__ == '__main__':
    ignite_simple.helper(__name__)

> python3 -m mymod --help

ignite_simple.helper.handle(module=None)[source]¶

Uses the given module containing model(), dataset(), loss() and accuracy_style as the module for train() or analyze() with everything else determined by the command line arguments.

Parameters: module – module containing model(), dataset(), loss(), accuracy_style

ignite_simple.helper.reanalyze(module, args)[source]¶

Uses the given arguments from argparse to determine the arguments to ignite_simple.reanalyze

Parameters

module – module containing model(), dataset(), loss(), accuracy_style
args – argparse result

ignite_simple.helper.train(module, args)[source]¶

Uses the given arguments from argparse to determine the arguments to ignite_simple.train

Parameters

module – module containing the model(), dataset(), loss(), and accuracy_style
args – argparse result

ignite_simple.utils¶

Utility functions which are used through ignite-simple but are short enough not to warrant their own module.

ignite_simple.utils.create_partial_loader(dset: torch.utils.data.dataset.Dataset, amt: int, batch_size: int = 256) → torch.utils.data.dataloader.DataLoader[source]¶

Creates a dataloader which loads only a random subset of the specified length from the dataset, using the specified batch size.

Parameters

dset – the dataset to create a loader for a partial subset of
amt – the number of items in the partial subset

Returns

the described dataloader with a reasonable batch size

ignite_simple.utils.fix_imports(loader: Tuple[str, str, tuple, dict])[source]¶

Returns the loader which represents the same callable as the given one, except potentially with the __main__ name replaced with the correct module name.

Parameters: loader – the loader whose imports might need to be cleaned
Returns: the cleaned loader

ignite_simple.utils.invoke(loader: Tuple[str, str, tuple, dict])[source]¶

Invokes the callable which has the given name in the given module, using the given arguments and keyword arguments

Parameters: loader – (module, attrname, args, kwargs) - the callable to invoke
Returns: the result of the callable

ignite_simple.utils.noop(*args, **kwargs)[source]¶: This function does nothing.

ignite_simple.utils.split(full: torch.utils.data.dataset.Dataset, val_perc: float, filen: str = None) → Tuple[torch.utils.data.dataset.Dataset, torch.utils.data.dataset.Dataset][source]¶

Splits the given dataset into two datasets, the first of which has (1 - val_perc) fraction of the data and the other has val_perc fraction of the data, distributed randomly.

Parameters

full (data.Dataset) – the entire dataset to split into two
val_perc (float) – the amount to be broken away from full

Returns

(train_set, val_set)

ignite_simple.utils.task_loader(dataset_loader, batch_size, shuffle, drop_last)[source]¶

Creates a task loader from a dataset loader.

Parameters

dataset_loader – the dataset loader (str, str, tuple, dict)
batch_size – the batch size for the data loader
shuffle – if the training dataset should be shuffled between epochs
drop_last – if the last batch should be dropped if its not the same size as the rest. should only be used if shuffle is True

ignite_simple.figure_utils¶

Some utility functions related to creating good looking figures.

ignite_simple.figure_utils.fig_exists(outfile_wo_ext: str, filter_: Optional[Callable] = None) → bool[source]¶

Returns true if the files from save_fig with the given outfile already exist.

Parameters

outfile_wo_ext (str) – where we expect the file to have been saved
filter (Optional[callable]) – an optional callable which accepts images and returns True if they are relevant and False if they can be ignored

Returns

True if the file(s) exists, False otherwise

Return type

bool

ignite_simple.figure_utils.make_vs_title(x: str, y: str)[source]¶

Creates the axes title for a plot with the given x-axis variable and y-axis variable

Parameters

x (str) – the name for the x-variable
y (str) – the name for the y-variable

Returns

The correct axes title

Return type

str

ignite_simple.figure_utils.save_fig(fig: matplotlib.figure.Figure, ax: matplotlib.axes._axes.Axes, title: str, outfile_wo_ext: str, filter_: Optional[Callable] = None)[source]¶

Saves the given figure with many different commonly used figure sizes, resizing labels and titles as appropriate. This technique works only for a single axis plot, since for other styles different font sizes would be appropriate.

Parameters

fig (Figure) – The figure to save
ax (Axes) – The axis on the figure
title (str) – The title of the plot
outfile_wo_ext (str) – where to save the figure, omitting the file extension (as many files may be saved)
filter (optional[callable]) – if not None, a callable object which accepts a string which is a (possibly relative) path to a file which will only be saved if the result of filter_ is True.

ignite_simple.figure_utils.set_ticklabel_sizes(fig: matplotlib.figure.Figure, ax: matplotlib.axes._axes.Axes, digital: bool)[source]¶

Updates the sizes of the tick labels for the given figure based on its canvas size and canvas dpi.

Parameters

fig (Figure) – The figure to update
ax (Axes) – The specific axes within the figure to update
digital (bool) – True if this is for digital display, False for physical display

ignite_simple.figure_utils.set_title(fig: matplotlib.figure.Figure, ax: matplotlib.axes._axes.Axes, title: str, digital: bool)[source]¶

Sets the title for the given axes to the given value. This is more particular than the default matplotlib Axes.set_title.

Parameters

fig (Figure) – the figure the axes is in
ax (Axes) – the axes which you want to have a title
title (str) – the desired title text
digital (bool) – if True, a large font size is selected. Otherwise, a smaller font size is selected

Returns

TextCollection that was added

ignite_simple.vary_bs_loader¶

A torch dataloader which varies the batch size between two amounts over the course of a specified number of epochs of an underlying dataset. The variation spends more time at lower batch sizes than at higher batch sizes, for convenience of implementation and to account for the higher stochasticity at lower batch sizes

class ignite_simple.vary_bs_loader.BatchSizeVaryingDataLoader(dataset, start_batch_size, end_batch_size, epochs)[source]¶

A dataloader which acts on an underlying dataset, varying the batch size linearly between two specified amounts over a given period of time. Note that this redefines one epoch to be the specified amount of time!

Variables

dataset (data.Dataset) – the underlying dataset from which points and labels are being pulled
start_batch_size (int) – the starting batch size
end_batch_size (int) – the final batch size
epochs (int) – the number of epochs over which the underlying dataset is iterated over
last_iter (iterator) – the last real iterator that was created

dry_iter()[source]¶: Creates a ‘dry’ iterator which does not actually produce anything but has the correct length and updates last_batch_size normally

ignite_simple.dispatcher¶

This package handles accepting a list of jobs and then deciding which of them to run in what order.

This module is intended for use by the analysis module for managing all the jobs it has to do. The analysis package tends to have many jobs are all fairly independent of each other. Things like rq and celery are way over-engineered for this task.

class ignite_simple.dispatcher.GreedyTaskQueue(total_cores: int)[source]¶

Describes a greedy task queue, which prioritizes tasks with the highest number of cores.

Variables

total_cores (int) – if the cores on a task is None, we assume it is going to require this many cores.
list[Task]] tasks_by_core (dict[int,) – a lookup for remaining tasks, where the key is the number of cores the task requires.
sorted_keys (SortedList[int]) – the keys for tasks_by_core in ascending order.
expecting_more_tasks (bool) – if True, we expect that add_task will be called in the (near) future, hence we always return True from have_more_tasks
_len (int) – the length of this queue

add_task(task: ignite_simple.dispatcher.Task) → None[source]¶: Adds the given task to this queue.

add_tasks(tasks: Iterable[ignite_simple.dispatcher.Task]) → None[source]¶: Adds all the specified tasks to this queue.

flatten() → List[ignite_simple.dispatcher.Task][source]¶: Returns this task queue as a list of tasks.

get_next_task(cores: int)[source]¶

Gets the next task to perform given that there are the specified number of cores available. The resulting Task should require fewer than the number of cores available. May return None if there are no tasks which meet the requirements to do.

Parameters: cores (int) – the number of cores available
Returns: the task to start work on, None if no task available yet
Return type: typing.Optional[Task]

have_more_tasks()[source]¶: Returns True if there is at least one more task to do, and returns False if there are no more tasks to do

set_total_cores(total_cores)[source]¶: Clips the maximum number of cores that each task requires to not exceed total_cores.

class ignite_simple.dispatcher.MainToWorkerConnection(proc: multiprocessing.context.Process, jobq: pympanim.zeromqqueue.ZeroMQQueue, ackq: pympanim.zeromqqueue.ZeroMQQueue, sleep_delay: float)[source]¶

Describes a connection from the main process to the worker process

Variables

process (Process) – the actual worker process
jobq (ZeroMQQueue) – the queue that jobs can be sent through to this worker
ackq (ZeroMQQueue) – the queue that the worker sends job completed notifications through
cores (optional[int]) – the number of cores that this worker is currently using, or None if the worker is not doing anything right now
callback (optional[callable]) – a callback function which will be invoked when the current task completes, or None if there is no current task or no callback registered
sleep_delay (float) – the current sleep delay for this worker

close()[source]¶: Shuts down the worker

is_ready() → bool[source]¶: Checks to see if the worker is ready to get a new task.

send(task: ignite_simple.dispatcher.Task)[source]¶

Tells the worker to perform the given task.

Parameters: task – the task to perform

update_sleep_delay(delay: float)[source]¶: Tells the worker to poll for jobs with the given inter-poll delay

class ignite_simple.dispatcher.Task(module: str, attrname: str, args: tuple, kwargs: dict, cores: Optional[int], callback: Optional[Callable] = None)[source]¶

A description of a task.

Variables

module (str) – the name of the module which contains the callable
attrname (str) – the name of the attribute within the module
args (tuple) – the arguments to pass to the callable
kwargs (dict) – the keyword arguments to pass to the callable
cores (optional[int]) – the number of cores this task will use, None for all cores
callback (optional[callable]) – an optional callback for when this task completes, not sent over. The callback is invoked on the main thread

class ignite_simple.dispatcher.TaskQueue[source]¶

An interface for something which is capable of returning the next task to perform based on the number of available cores.

get_next_task(cores: int) → Optional[ignite_simple.dispatcher.Task][source]¶

Gets the next task to perform given that there are the specified number of cores available. The resulting Task should require fewer than the number of cores available. May return None if there are no tasks which meet the requirements to do.

Parameters: cores (int) – the number of cores available
Returns: the task to start work on, None if no task available yet
Return type: typing.Optional[Task]

have_more_tasks() → bool[source]¶: Returns True if there is at least one more task to do, and returns False if there are no more tasks to do

set_total_cores(total_cores: int) → None[source]¶: Clips the maximum number of cores that each task requires to not exceed total_cores.

ignite_simple.dispatcher.dispatch(tasks: Union[ignite_simple.dispatcher.TaskQueue, Iterable[ignite_simple.dispatcher.Task]], total_cores: int, suggested_imports: Tuple[str] = (), wait_function: Callable = None) → None[source]¶

Dispatches the given tasks using greedy selection such that no more than the specified number of cores are in use at once, where possible to do so. This uses a greedy selection of tasks.

Parameters

tasks – an iterable of tasks to dispatch, or a task queue. May be modified, but the logging may be incorrect if modified outside of the wait function.
total_cores – the target number of cores to use at once
suggested_imports – things which are imported in each worker process during the spawning phase, which causes jobs to be processed more smoothly.
wait_function – if not None, this will be invoked when the dispatcher is idling. Expected to be fairly fast, and it is safe to modify tasks within this call.

General Sweeping Module¶

ignite_simple.gen_sweep.param_selectors¶

Contains the interface and some implementations for how parameters are selected for a particular point. Method selection techniques which do not depend on the results of previous points are more easily parallelizable.

class ignite_simple.gen_sweep.param_selectors.FixedSweep(gnt_results: tuple)[source]¶

A sweep which has a fixed order of results from get_next_trials.

Variables

gnt_results (tuple) – the results for get_next_trials in the order they should appear
gnt_results_index (int) – the index in gnt_results for the next invocation of get_next_trials.

get_next_trials()[source]¶

Get the next set of trials to perform. The result will be a tuple of two elements - the values (a tuple) and the number of trials to perform. This may return the same values multiple times, although it may improve performance to batch trials of the same values together.

This may return None only if the ParamSelector is waiting on the result of previous trials before it will continue. After returning None, this function will be invoked after the next on_trials_completed or on_hparams_found call. It may return None multiple times in a row if there were multiple function calls which it’s waiting for.

Note it should return None, not (None, None)

Returns: A tuple of 2 elements, the first being a tuple of the values for the parameters and the second being a nonnegative int for the number of trials to perform. If the int is 0, the hparams will be found if they haven’t already but no trials will be performed. If the result is None, see above.
Return type: typing.Optional[tuple]

has_more_trials()[source]¶: Determines if this param selector will ever want to perform more trials. If False, the sweeper will stop calling get_next_trials and this must return False at least until the sweep completes. If get_next_trials should continue to be called this should return True.

classmethod with_fixed_trials(pts: Iterable[tuple], trials: int) → ignite_simple.gen_sweep.param_selectors.FixedSweep[source]¶

For each of the given points, perform precisely the specified number of trials.

Parameters

pts (iterable[tuple]) – the points to test
trials (int) – the numebr of trials to perform at each point

Returns

the corresponding parameter selector

Return type

FixedSweep

classmethod with_random_trials(ranges: Tuple[Union[int, set, tuple]], points: int, trials: int, r_eps: float = 0.01) → ignite_simple.gen_sweep.param_selectors.FixedSweep[source]¶

For each parameter, ranges contains the valid values for that parameter. If the range is an int, it is assumed to be the arange for that int + 1. If it is a set, it contains the valid values for that parameter. If it’s a tuple, it should contain two elements, the min and the max, where both are integers if the values can only be integers and either is a float if the values can be floats.

Ranges are inclusive.

Parameters

ranges (typing.Union[int, set, tuple]) – the possible values for each parameter, one set of possible values per parameter.
points (int) – the number of unique points to select
trials (int) – the number of trials per point
r_eps – Only used if one of the ranges is specified as an interval (i.e., not a set). Points which would be within (r_eps*interval_width) of each other are rejected during sampling. Should be less than 1/(2 * trials) for the randomness to be meaningfully different from grid. If too large an error may be thrown as not enough trials could be found that weren’t rejected.

class ignite_simple.gen_sweep.param_selectors.ParamSelector[source]¶

The interface for a class which is capable of selecting what values for a given known set of parameters should have models initialized and compared.

get_next_trials() → Optional[tuple][source]¶

Get the next set of trials to perform. The result will be a tuple of two elements - the values (a tuple) and the number of trials to perform. This may return the same values multiple times, although it may improve performance to batch trials of the same values together.

This may return None only if the ParamSelector is waiting on the result of previous trials before it will continue. After returning None, this function will be invoked after the next on_trials_completed or on_hparams_found call. It may return None multiple times in a row if there were multiple function calls which it’s waiting for.

Note it should return None, not (None, None)

Returns: A tuple of 2 elements, the first being a tuple of the values for the parameters and the second being a nonnegative int for the number of trials to perform. If the int is 0, the hparams will be found if they haven’t already but no trials will be performed. If the result is None, see above.
Return type: typing.Optional[tuple]

has_more_trials() → bool[source]¶: Determines if this param selector will ever want to perform more trials. If False, the sweeper will stop calling get_next_trials and this must return False at least until the sweep completes. If get_next_trials should continue to be called this should return True.

class ignite_simple.gen_sweep.param_selectors.SweepListener[source]¶

An interface for something which is listening for the result of a sweep.

on_hparams_found(values: tuple, lr_min: float, lr_max: float, batch_size: int) → None[source]¶

Called when the parameter sweep has found the appropriate hyper parameters for the given set of main parameters.

Parameters

values (tuple) – each element corresponds to the value of the corresponding parameter. For example, if the parameters list is actuallly “nonlinearity”, “hidden size” then values might be “Tanh”, 128
lr_min (float) – the best value for the minimum for the learning rate during one oscillation of learning rate
lr_max (float) – the best value for the maximum for the learning rate during one oscillation of learning rate
batch_size (int) – the best batch size to use

on_trials_completed(values: tuple, perfs_train: numpy.ndarray, losses_train: numpy.ndarray, perfs_val: numpy.ndarray, losses_val: numpy.ndarray)[source]¶

Invoked when we perform some trials. This could be called multiple times of the same point at the decision of the sweeper, but each trial will only be sent to this function once.

Parameters

values (tuple) – the values for each parameter in order
perfs_train (np.ndarray) – the final performance for each trial performed, so has shape (trials,), on the training dataset
losses_train (np.ndarray) – the final loss for each trial performed, so has shape (trials,), on the training dataset
perfs_val (np.ndarray) – the final performance for each trial performed, so has shape (trials,), on the validation dataset
losses_val (np.ndarray) – the final loss for each trial performed, so has shape (trials,), on the validation dataset

ignite_simple.gen_sweep.sweeper¶

Handles performing the model parameter sweep.

class ignite_simple.gen_sweep.sweeper.ParamsTask(params: tuple, trials: int)[source]¶

Describes a task to call model_manager.train for a particular set of model parameters. This isn’t actually a disp.Task, although it can be converted to one.

Variables

params (tuple) – the parameters. this is the instance variable used for equality checks.
trials (int) – the number of trials to perform.

as_task(module_name: str, hparams: ignite_simple.hyperparams.HyperparameterSettings, folder: str, iden: int, trials: int, sweep_cores: int, listeners: Iterable[ignite_simple.gen_sweep.param_selectors.SweepListener], ntrials_disp: int) → ignite_simple.dispatcher.Task[source]¶

Creates the task object which performs the specified number of trials with this tasks parameters, assuming that the directory structure is as follows:

folder/points/

<id>/: <result from model manager>

This function assumes that we always perform the hparam sweep in a separate call to the trial sweep for the purposes of handling the listeners.

class ignite_simple.gen_sweep.sweeper.ParamsTaskQueue(total_cores: int, sweep_cores: int, module: str, hparams: ignite_simple.hyperparams.HyperparameterSettings, folder: str, listeners: List[ignite_simple.gen_sweep.param_selectors.SweepListener])[source]¶

This task queue acts like a greedy task queue, except it only works for tasks which amount to calling model_manager.train for a particular set of models.

The process for selecting tasks is as follows:

If we have a hparameter sweep queued and we have enough cores to do one, do that.

Perform as many trials as possible. Note we can only perform trials on parameters we already know the hyperparameters for

Variables

total_cores (int) – the number of physical cores that are available
sweep_cores (int) – the number of cores to use for sweeping (not greater than the number of total cores)
module (str) – the module which we are getting the model/dataset/etc
hparams (HyperparameterSettings) – strategy for hyperparameters
folder (str) – the folder containing the points folder
listeners (list[SweepListener]) – the sweep listeners. contains self.
in_progress (set[tuple]) – parameters which are currently in progress.
sweeps (list[ParamsTask]) – the sweeps that need to be performed in an arbitrary order
int] params_to_sweeps_ind (dict[tuple,) – a lookup that goes from the parameters of tasks to the index in sweeps if a sweep is still necessary.
trials (SortedList[ParamsTask]) – the trials that need to be performed, in ascending order of the number of trials.
int] params_to_id (dict[tuple,) – dictionary which converts a given set of parameters to the corresponding unique identifier for that set of parameters.
int] params_to_ntrials (dict[tuple,) – dictionary which converts a given set of parameters to the corresponding number of trials that have been dispatched for those parameters
next_id (int) – the next id that should be given out to a set of parameters and then incremented.
ParamsTask] params_to_task (dict[tuple,) – a lookup that goes from params lists to param tasks, where this only contains tasks which have not yet been dispatched, and does not include tasks which are in sweeps
_len (int) – number of actual tasks currently in queue
expecting_more_trials (bool) – True to prevent saying we are out of trials, False otherwise

add_task_by_params(params: tuple, trials: int) → None[source]¶: Adds a task to this queue based on the parameters which should be swept and the number of trials to perform. Regardless of the value for trials, this will ensure that the hyperparameters for the given model parameters have been found.

get_next_task(cores)[source]¶

Gets the next task to perform given that there are the specified number of cores available. The resulting Task should require fewer than the number of cores available. May return None if there are no tasks which meet the requirements to do.

Parameters: cores (int) – the number of cores available
Returns: the task to start work on, None if no task available yet
Return type: typing.Optional[Task]

have_more_tasks()[source]¶: Returns True if there is at least one more task to do, and returns False if there are no more tasks to do

on_hparams_found(values, lr_min, lr_max, batch_size)[source]¶

Called when the parameter sweep has found the appropriate hyper parameters for the given set of main parameters.

Parameters

values (tuple) – each element corresponds to the value of the corresponding parameter. For example, if the parameters list is actuallly “nonlinearity”, “hidden size” then values might be “Tanh”, 128
lr_min (float) – the best value for the minimum for the learning rate during one oscillation of learning rate
lr_max (float) – the best value for the maximum for the learning rate during one oscillation of learning rate
batch_size (int) – the best batch size to use

on_trials_completed(values, perfs_train, losses_train, perfs_val, losses_val)[source]¶

Invoked when we perform some trials. This could be called multiple times of the same point at the decision of the sweeper, but each trial will only be sent to this function once.

Parameters

values (tuple) – the values for each parameter in order
perfs_train (np.ndarray) – the final performance for each trial performed, so has shape (trials,), on the training dataset
losses_train (np.ndarray) – the final loss for each trial performed, so has shape (trials,), on the training dataset
perfs_val (np.ndarray) – the final performance for each trial performed, so has shape (trials,), on the validation dataset
losses_val (np.ndarray) – the final loss for each trial performed, so has shape (trials,), on the validation dataset

set_total_cores(total_cores)[source]¶: Clips the maximum number of cores that each task requires to not exceed total_cores.

class ignite_simple.gen_sweep.sweeper.Sweeper(module_name: str, param_selector: ignite_simple.gen_sweep.param_selectors.ParamSelector, tasks: ignite_simple.gen_sweep.sweeper.ParamsTaskQueue, sweep_cores: int, folder: str, hparams: ignite_simple.hyperparams.HyperparameterSettings)[source]¶

The instance of the class which performs all the individual operations required for sweeping over an arbitrary number of architectural parameters. Everything within this class occurs on the main thread.

Variables

module_name – the module which contains the model, dataset, loss, and accuracy style. The model function has N parameters, where N is the number of parameters the param selector gives us.
param_selector (ParamSelector) – the object which selects which combination of parameters to test
tasks (ParamsTaskQueue) – the tasks that we know we are ready to perform. These are being immediately dispatched to other threads according to the number of cores required.
sweep_cores (int) – the number of actual physical cores that we assign to sweeping. Note that this differs from the argument to sweep.
folder (str) – the path to the folder we are saving things in. we actually store stuff at folder/points for the most part.
hparams (HyperparameterSettings) – the hyperparameter settings used for hyperparameter tuning.

add_tasks(limit=100) → None[source]¶

Adds as many tasks to the TaskQueue as the parameter selector will provide, without causing the tasks list to exceed the given limit. This will also update tasks.expecting_more_tasks.

This must ensure that we don’t enqueue multiple tasks which have the same hyperparameters, however we can update tasks to add more trials.

collate_files() → None[source]¶

Collates the information found in each of the sweeps into locations which are more readily accessible. This only requires that the folder is set correctly and store_params_lookup has been called.

Produces the following files:

folder/points/

funcres.pt

Contains a pickled list of tuples of the following form:

(params, min lr, max lr, bs, perfs_train, losses_train,: perfs_val, losses_val)

store_params_lookup() → None[source]¶

Stores the parameters that corresponded to each of the arbitrarily named folders in folder/points. This only works when we just swept and hence the details are still in memory.

Produces the following files:

folder/points/

params_lookup.pt: Contains a pickled tuple of two dictionaries. The first dictionary goes form parameter tuples to corresponding ids (names of folders). The second goes from ids to parameter tuples.
params.csv: A human-readable variant of params_lookup. A CSV file where the rows are <id>,<params>. Each individual parameter is given its own column. The first row is used for descriptions and should be skipped when parsing.

ignite_simple.gen_sweep.sweeper.sweep(module_name: str, param_selector: ignite_simple.gen_sweep.param_selectors.ParamSelector, sweep_cores: int, hparams: Union[str, ignite_simple.hyperparams.HyperparameterSettings], folder: str) → list[source]¶

Performs the given architectural parameter search on the problem defined in the given module.

Parameters

module_name (str) – the path to the module which contains the model / dataset / loss / accuracy style, where the model accepts the parameters which are being swept (one argument per parameter).
param_selector (ParamSelector) – what decides what parameters to be sent. Note that this is structured so it can utilize partial sweeps to inform further search, although the best performance requires that it can give more than one trial at a time.
sweep_cores (int) –
the number of cores that are used for sweeping parameters. This is important both for being able to replicate the results of the sweep and in terms of performance. If this number exceeds the number of physical cores, it will be simulated in a way that correctly utilizes the available resources.

A higher number of sweep cores gives more accurate and consistent hyperparameters, which is essential for meaningful comparisons between architectures. 4 is a reasonable starting place.
hparams (HyperparameterSettings) – either the hyperparameter settings to use or a name of a preset (‘fastest’, ‘fast’, ‘slow’, ‘slowest’) that is used for sweeping hyperparameters. This value will be modified to ensure we are simulating sweep_cores correctly if necessary.
folder (str) –
where to save the output to. the main output that one will typically want to use is

folder/points/

funcres.pt: contains a list of tuples where each tuple is
of the form (params, lr_min, lr_max, bs, perf_train, loss_train, perf_val, loss_val). The performance and loss are expressed as an array with one element per trial.

params_lookup.pt: contains a tuple of two dictionaries,
where the first is params_to_ids and goes from a tuple of parameters to the corresponding name of the folder in folder/points which is the result of the model_manager train, and the second has the keys/values swapped (ids to params).

This returns the unpickled content in folder/points/funcres.pt, which is a list of tuples of the form

(params: tuple,
lr_min: float,
lr_max: float,
bs: int,
perf_train: np.ndarray #  (shape=(trials,)),
loss_train: np.ndarray #  (shape=(trials,)),
perf_val: np.ndarray #  (shape=(trials,)),
loss_val: np.ndarray #  (shape=(trials,))
)

As is typical, loss is non-negative and lower is better. Performance is between 0 and 1 and higher is better.

Ignite Simple¶

ignite_simple.analarams¶

ignite_simple.hyperparams¶

ignite_simple.model_manager¶

ignite_simple.trainer¶

ignite_simple.analysis¶

ignite_simple.text_analysis¶

ignite_simple.range_finder¶

ignite_simple.tuner¶

ignite_simple.helper¶

ignite_simple.utils¶

ignite_simple.figure_utils¶

ignite_simple.vary_bs_loader¶

ignite_simple.dispatcher¶

General Sweeping Module¶

ignite_simple.gen_sweep.param_selectors¶

ignite_simple.gen_sweep.sweeper¶

Indices and tables¶