neon.optimizers.optimizer.Optimizer

class neon.optimizers.optimizer.Optimizer(name=None)[source]

Bases: neon.NervanaObject

The optimizer class handles the gradient update stage of training a neural network. Given the current parameters \(w\), update parameters \(\Delta w\), and current state \(s\), the optimizer specifies an algorithm for performing the update.

This base class contains to helper functions for scaling the gradients. specifices the abstract method optimize, which subclasses should implement. The optimize method is called at every minibatch to update the layer parameters.

__init__(name=None)[source]

Class constructor.

Methods

__init__([name]) Class constructor.
clip_gradient_norm(param_list, clip_norm) Returns a scaling factor to apply to the gradients.
clip_value(v[, abs_bound]) Element-wise clip a gradient or parameter tensor to between -abs_bound and +abs_bound.
gen_class(pdict)
get_description([skip]) Returns a dict that contains all necessary information needed to serialize this object.
optimize(layer_list, epoch) Update the parameters for a provided list of layers.
recursive_gen(pdict, key) helper method to check whether the definition
be = None
classnm

Returns the class name.

clip_gradient_norm(param_list, clip_norm)[source]

Returns a scaling factor to apply to the gradients.

The scaling factor is computed such that the root mean squared average of the scaled gradients across all layers will be less than or equal to the provided clip_norm value. This factor is always <1, so never scales up the gradients.

Parameters:
  • param_list (list) – List of layer parameters
  • clip_norm (float, optional) – Target norm for the gradients. If not provided the returned scale_factor will equal 1.
Returns:

Computed scale factor.

Return type:

scale_factor (float)

clip_value(v, abs_bound=None)[source]

Element-wise clip a gradient or parameter tensor to between -abs_bound and +abs_bound.

Parameters:
  • v (tensor) – Tensor of gradients or parameters for a single layer
  • abs_bound (float, optional) – Value to element-wise clip gradients or parameters. Defaults to None.
Returns:

Tensor of clipped gradients or parameters.

Return type:

v (tensor)

gen_class(pdict)
get_description(skip=[], **kwargs)

Returns a dict that contains all necessary information needed to serialize this object.

Parameters:skip (list) – Objects to omit from the dictionary.
Returns:Dictionary format for object information.
Return type:(dict)
modulenm

Returns the full module path.

optimize(layer_list, epoch)[source]

Update the parameters for a provided list of layers.

Parameters:
  • layer_list (list) – List of layers to optimize
  • epoch (integer) – Epoch count of training
recursive_gen(pdict, key)

helper method to check whether the definition dictionary is defining a NervanaObject child, if so it will instantiate that object and replace the dictionary element with an instance of that object