ML Operational Layer (MOP) API

We expose the following API which we refer to as our Machine learning Operational Layer (MOP layer). This layer abstracts the backend, so the same operations can be performed on a CPU, GPU, or future hardware backends.

The API consists of two interface classes:

Interface Description
neon.backends.Tensor \(n\)-dimensional array data structure
neon.backends.Backend Backend interface used to manipulate Tensor data

Basic Data Structure

The Tensor class is used to represent an arbitrary dimensional array in which each element is stored using a consistent underlying type. For the CPU and GPU backends, Tensors are stored as CPUTensor and GPUTensor subclasses, respectively.

We have the ability to instantiate and copy instances of this data structure, as well as initialize its elements, reshape its dimensions, and access metadata.

Tensor Creation

Method Description
neon.backends.Backend.empty Instantiate an empty Tensor
neon.backends.Backend.array Instantiate a new Tensor, populating elements based on a provided array.
neon.backends.Backend.zeros Instantiate a new Tensor, populating each element with the value of 0.
neon.backends.Backend.ones Instantiate a new Tensor, populating each element with the value of 1.
neon.backends.Tensor.copy Construct and return a deep copy of the Tensor passed.

Tensor Manipulation

Method Description
neon.backends.Tensor.get Convert the tensor to a host memory numpy.ndarray.
neon.backends.Tensor.take Select a subset of elements from an array across an axis
neon.backends.Tensor.__getitem__ Extract a subset view of the items via slice style indexing along each dimension.
neon.backends.Tensor.__setitem__ Assign the specified value to a subset of elements found via slice style indexing along each dimension.
neon.backends.Tensor.fill Assign specified value to each element of this Tensor.
neon.backends.Tensor.transpose Return a transposed view of the data.
neon.backends.Tensor.reshape Adjust the dimensions of the data to the specified shape.
neon.backends.Backend.take Select a subset of elements (based on provided indices) from a supplied dimension

Arithmetic Operations

Unary and binary arithmetic operations can be performed on Tensor objects. In all cases, the user must pre-allocate correctly sized output to store the result. All the below operations are performed element-wise.

Element-wise Binary Operations

These element-wise binary operations perform the following operations. An optional out= argument can be passed to store the result as a Tensor. If out=None (default), an op-tree is returned.

Method Description
neon.backends.Backend.add(a, b) \(a+b\)
neon.backends.Backend.subtract(a,b) \(a-b\)
neon.backends.Backend.multiply(a, b) \(a\times b\)
neon.backends.Backend.divide(a, b) \(\frac{a}{b}\)
neon.backends.Backend.dot(a, b) \(a \cdot b\)
neon.backends.Backend.power(a, b) \(a^b\)
neon.backends.Backend.maximum(a, b) \(\max(a, b)\)
neon.backends.Backend.minimum(a, b) \(\min(a,b)\)
neon.backends.Backend.clip(a, a_min, a_max) Clip each element of \(a\) between the corresponding elements in \(a_\text{min}\) and \(a_\text{max}\)

Element-wise Unary Functions

These element-wise operations operate on one input Tensor or Op-tree. Similar to the binary operations, an optional out= argument can be passed to store the result as a Tensor. If out=None (default), an op-tree is returned.

Method Description
neon.backends.Backend.log(a) \(\log(a)\)
neon.backends.Backend.log2(a) \(\log_2(a)\)
neon.backends.Backend.exp(a) \(e^a\)
neon.backends.Backend.exp2(a) \(2^a\)
neon.backends.Backend.abs(a) \(abs(a)\)
neon.backends.Backend.sgn(a) if \(x<0\), \(-1\); if \(x=0\), \(0\); if \(x>0\), \(1\)
neon.backends.Backend.sqrt(a) \(\sqrt{a}\)
neon.backends.Backend.square(a) \(a^2\)
neon.backends.Backend.reciprocal(a) \(1/a\)
neon.backends.Backend.negative(a) \(-a\)
neon.backends.Backend.sig(a) \(1/(1+\exp(-a))\)
neon.backends.Backend.tanh(a) \(\tanh(a)\)
neon.backends.Backend.finite(a) Element-wise test for finiteness (e.g. \(a \neq \infty\))

Element-wise Logical Operations

These methods perform element-wise logical testing on each corresponding element of the input \(a\) and \(b\). As before, an optional out= argument can be passed to store the output.

Method Description
neon.backends.Backend.equal(a, b) \(a=b\)
neon.backends.Backend.not_equal(a, b) \(a\neq b\)
neon.backends.Backend.greater(a, b) \(a>b\)
neon.backends.Backend.greater_equal(a, b) \(a \geq b\)
neon.backends.Backend.less(a, b) \(a<b\)
neon.backends.Backend.less_equal(a, b) \(a\leq b\)

Summary Operations

These operations perform a summary calculation over a single provided tensor a along a specified axis dimension. If axis=None (default), the calculation is performed over all the dimensions.

Method Description
neon.backends.Backend.sum Sum elements over the axis dimension
neon.backends.Backend.mean Compute the arithmetic mean over the axis dimension
neon.backends.Backend.var Compute the variance of the elements over the axis dimension
neon.backends.Backend.std Compute the standard deviation of the elements along the axis dimension
neon.backends.Backend.min Calculate the minimal element value over the axis dimension
neon.backends.Backend.max Calculate the maximal element value over the axis dimension
neon.backends.Backend.argmin Calculate the indices of the minimal element value along the axis dimension
neon.backends.Backend.argmax Calculate the indices of the maximal element value along the axis dimension

Random Number Generator

Both the NervanaGPU and NervanaCPU backends use the numpy random number generator (np.random.RandomState).

Method Description
neon.backends.Backend.gen_rng Setup the numpy rng and store the initial state. Called by the backend constructor
neon.backends.Backend.rng_get_state Return the state of the rng
neon.backends.Backend.rng_set_state Set the state of the rng
neon.backends.Backend.rng_reset Reset the state to the initial state
neon.backends.NervanaGPU.rand Generate random numbers uniformly distributed between 0 and 1.

To generate a Tensor with shape (100,100), where each element is uniformly distributed between 0 and 1, we can call

myTensor = be.empty((100,100))
myTensor[:] = be.rand()

Loop indicators

These two methods signal the start (and end) of a block of repeated computations, such as a loop. This operation can be used to help the compiler optimize instruction performance, but has no direct effect on the underlying calculations. Each Backend.start() call must be book-ended by a corresponding Backend.end() call. Note that multiple begin calls can appear adjacent in nested loops.

Method Description
neon.backends.Backend.begin Signal the start of a block
neon.backends.Backend.end Signal the end of a block

Higher-level Support

We have taken common operations used by neural network layers and performed many optimizations. Many of these operations include custom objects (such as ConvLayer or PoolLayer) which are used to track the layer parameters and cache some calculations.

Operations

Method Description
neon.backends.Backend.onehot Generate a one-hot representation from a set of indices (see Loading data)
neon.backends.Backend.fill_normal Fill a tensor with gaussian random variables
neon.backends.Backend.compound_dot Depending on the size of the input \(A\), \(B\), \(C\), perform \(\alpha A B + \beta C\)
neon.backends.NervanaGPU.make_binary_mask Create a randomized binary mask for dropout layers
neon.backends.NervanaCPU.make_binary_mask Create a randomized binary mask for dropout layers

Convolutional Layers

Method Description
neon.backends.Backend.conv_layer Create a ConvLayer object that holds the filter parameters. This is passed to the below functions.
neon.backends.Backend.fprop_conv Forward propagate the inputs of a convolutional network layer
neon.backends.Backend.bprop_conv Backward propagate the error through a convolutional network layer.
neon.backends.Backend.update_conv Compute the updated gradient for a convolutional network layer.
neon.backends.Backend.deconv_layer Create a DeconvLayer object that holds the filter parameters. This is passed to the above functions.

Pooling Layers

Method Description
neon.backends.Backend.pool_layer Create a new PoolLayer parameter object.
neon.backends.Backend.fprop_pool Forward propagate pooling layer.
neon.backends.Backend.bprop_pool Backward propagate pooling layer.

The below methods implement ROI-pooling, where the pooling window sizes are themselves hyper-parameters.

Method Description
neon.backends.Backend.roipooling_fprop Forward propagate through ROI-pooling layer
neon.backends.Backend.roipooling_bprop Backward propagate through ROI-pooling layer

Local Response Normalization

Method Description
neon.backends.Backend.lrn_layer Create the LRNLayer parameter object
neon.backends.Backend.fprop_lrn Forward propagate through LRN layer
neon.backends.Backend.bprop_lrn Backward propagate through LRN layer

Batch Normalization

Method Description
neon.backends.Backend.compound_fprop_bn Forward propagate through BatchNorm layer
neon.backends.Backend.compound_bprop_bn Backward propagate through BatchNorm layer

Linear layer with bias

Method Description
neon.backends.Backend.update_fc_bias Compute the updated bias gradient for a fully connected layer
neon.backends.Backend.add_fc_bias Add bias to a fully connected layer