ML Operational Layer (MOP) API

We expose the following API which we refer to as our Machine learning Operational Layer (MOP layer). This layer abstracts the backend, so the some operations can be performed on a CPU, GPU, or future hardware backends.

The API consists of the two interface classes:

Interface Description
neon.backends.Tensor \(n\)-dimensional array data structure
neon.backends.Backend Backend interface used to manipulate Tensor data

Basic Data Structure

The Tensor class is used to represent an arbitrary dimensional array in which each element is stored using a consistent underlying type. For the CPU and GPU backends, Tensors are stored as the CPUTensor and GPUTensor subclasses, respectively.

We have the ability to instantiate and copy instances of this data structure, as well as initialize its elements, reshape its dimensions, and access metadata.

Tensor Creation

Method Description
neon.backends.Backend.empty Instantiate an empty Tensor
neon.backends.Backend.array Instantiate a new Tensor, populating elements based on a provided array.
neon.backends.Backend.zeros Instantiate a new Tensor, population each element with the value of 0.
neon.backends.Backend.ones Instantiate a new Tensor, population each element with the value of 1.
neon.backends.Tensor.copy Construct and return a deep copy of the Tensor passed.

Tensor Manipulation

Method Description
neon.backends.Tensor.get Convert the tensor to an in host memory numpy.ndarray.
neon.backends.Tensor.take Select a subset of elements from an array across an axis
neon.backends.Tensor.__getitem__ Extract a subset view of the items via slice style indexing along each dimension.
neon.backends.Tensor.__setitem__ Assign the specified value to a subset of elements found via slice style indexing along each dimension.
neon.backends.Tensor.fill Assign specified value to each element of this Tensor.
neon.backends.Tensor.transpose Return a transposed view of the data.
neon.backends.Tensor.reshape Adjusts the dimensions of the data to the specified shape.
neon.backends.Backend.take Select a subset of elements (based on provided indices) from a supplied dimension

Arithmetic Operations

Unary and binary arithmetic operations can be performed on Tensor objects. In all cases, the user must pre-allocate correctly sized output to store the result. All the below operations are performed element-wise.

Element-wise Binary Operations

These element-wise binary operations perform the following operations. An optional out= argument can be passed to store the result as a Tensor. If out=None (default), an op-tree is returned.

Method Description
neon.backends.Backend.add(a, b) \(a+b\)
neon.backends.Backend.subtract(a,b) \(a-b\)
neon.backends.Backend.multiply(a, b) \(a\times b\)
neon.backends.Backend.divide(a, b) \(\frac{a}{b}\)
neon.backends.Backend.dot(a, b) \(a \cdot b\)
neon.backends.Backend.power(a, b) \(a^b\)
neon.backends.Backend.maximum(a, b) \(\max(a, b)\)
neon.backends.Backend.minimum(a, b) \(\min(a,b)\)
neon.backends.Backend.clip(a, a_min, a_max) Clips each element of \(a\) between the corresponding elements in \(a_\text{min}\) and \(a_\text{max}\)

Element-wise Unary Functions

These element-wise operations operate on one input Tensor or Op-tree. Similar to the binary operations, an optional out= argument can be passed to store the result as a Tensor. If out=None (default), an op-tree is returned.

Method Description
neon.backends.Backend.log(a) \(\log(a)\)
neon.backends.Backend.log2(a) \(\log_2(a)\)
neon.backends.Backend.exp(a) \(e^a\)
neon.backends.Backend.exp2(a) \(2^a\)
neon.backends.Backend.abs(a) \(abs(a)\)
neon.backends.Backend.sgn(a) if \(x<0\), \(-1\); if \(x=0\), \(0\); if \(x>0\), \(1\)
neon.backends.Backend.sqrt(a) \(\sqrt{a}\)
neon.backends.Backend.square(a) \(a^2\)
neon.backends.Backend.reciprocal(a) \(1/a\)
neon.backends.Backend.negative(a) \(-a\)
neon.backends.Backend.sig(a) \(1/(1+\exp(-a))\)
neon.backends.Backend.tanh(a) \(\tanh(a)\)
neon.backends.Backend.finite(a) Element-wise test for finiteness (e.g. \(a \neq \infty\))

Element-wise Logical Operations

These methods perform element-wise logical testing on each corresponding element of the input \(a\) and \(b\). As before, an optional out= argument can be passed to store the output.

Method Description
neon.backends.Backend.equal(a, b) \(a=b\)
neon.backends.Backend.not_equal(a, b) \(a\neq b\)
neon.backends.Backend.greater(a, b) \(a>b\)
neon.backends.Backend.greater_equal(a, b) \(a \geq b\)
neon.backends.Backend.less(a, b) \(a<b\)
neon.backends.Backend.less_equal(a, b) \(a\leq b\)

Summary Operations

These operations perform a summary calculation over a single provided tensor a along a specified axis dimension. If axis=None (default), the calculation is performed over all the dimensions.

Method Description
neon.backends.Backend.sum Sums elements over the axis dimension
neon.backends.Backend.mean Computes the arithmetic mean over the axis dimension
neon.backends.Backend.var Computes the variance of the elements over the axis dimension
neon.backends.Backend.std Computes the standard deviation of the elements along the axis dimension
neon.backends.Backend.min Calculates the minimal element value over the axis dimension
neon.backends.Backend.max Calculates the maximal element value over the axis dimension
neon.backends.Backend.argmin Calculates the indices of the minimal element value along the axis dimension
neon.backends.Backend.argmax Calculates the indices of the maximal element value along the axis dimension

Random Number Generator

Both the NervanaGPU and NervanaCPU backends use the numpy random number generator (np.random.RandomState).

Method Description
neon.backends.Backend.gen_rng Setup the numpy rng and store the initial state. Called by the backend constructor
neon.backends.Backend.rng_get_state Return the state of the rng
neon.backends.Backend.rng_set_state Set the state of the rng
neon.backends.Backend.rng_reset Reset the state to the initial state
neon.backends.NervanaGPU.rand Generates random numbers uniformly distributed between 0 and 1.

To generate a Tensor with shape (100,100), where each element is uniformly distributed between 0 and 1, we can call

myTensor = be.empty((100,100))
myTensor[:] = be.rand()

Loop indicators

These two methods signal the start (and end) of a block of repeated computations, such as a loop. This operation can be used to help the compiler optimize instruction performance, but has no direct effect on the underlying calculations. Each Backend.start() call must be book-ended by a corresponding Backend.end() call. Note that multiple begin calls can appear adjacent in nested loops.

Method Description
neon.backends.Backend.begin Signal the start of a block
neon.backends.Backend.end Signal the end of a block

Higher-level Support

We have taken common operations used by neural network layers and performed many optimizations. Many of these operations include custom objects (such as ConvLayer or PoolLayer) which are used to track the layer parameters and cache some calculations.

Operations

Method Description
neon.backends.Backend.onehot Generates a one-hot representation from a set of indices (see Loading data)
neon.backends.Backend.fill_normal Fills a tensor with gaussian random variables
neon.backends.Backend.compound_dot Depending on the size of the input \(A\), \(B\), \(C\), performs \(\alpha A B + \beta C\)
neon.backends.NervanaGPU.make_binary_mask Creates a randomized binary mask for dropout layers
neon.backends.NervanaCPU.make_binary_mask Creates a randomized binary mask for dropout layers

Convolutional Layers

Method Description
neon.backends.Backend.conv_layer Creates a ConvLayer object that holds the filter parameters. This is passed to the below functions.
neon.backends.Backend.fprop_conv Forward propagate the inputs of a convolutional network layer
neon.backends.Backend.bprop_conv Backward propagate the error through a convolutional network layer.
neon.backends.Backend.update_conv Compute the updated gradient for a convolutional network layer.
neon.backends.Backend.deconv_layer Creates a DeconvLayer object that holds the filter parameters. This is passed to the above functions.

Pooling Layers

Method Description
neon.backends.Backend.pool_layer Create a new PoolLayer parameter object.
neon.backends.Backend.fprop_pool Forward propagate pooling layer.
neon.backends.Backend.bprop_pool Backward propagate pooling layer.

The below methods implement ROI-pooling, where the pooling window sizes are themselves hyper-parameters.

Method Description
neon.backends.Backend.roipooling_fprop Forward propagation through ROI-pooling layer
neon.backends.Backend.roipooling_bprop Backward propagation through ROI-pooling layer

Local Response Normalization

Method Description
neon.backends.Backend.lrn_layer Create the LRNLayer parameter object
neon.backends.Backend.fprop_lrn Forward propagation through LRN layer
neon.backends.Backend.bprop_lrn Backward propagation through LRN layer

Batch Normalization

Method Description
neon.backends.Backend.compound_fprop_bn Forward propagation through BatchNorm layer
neon.backends.Backend.compound_bprop_bn Backward propagation through BatchNorm layer

Linear layer with bias

Method Description
neon.backends.Backend.update_fc_bias Compute the updated bias gradient for a fully connected layer
neon.backends.Backend.add_fc_bias Add bias to a fully connected layer