ML Operational Layer (MOP) API¶
We expose the following API which we refer to as our Machine learning Operational Layer (MOP layer). This layer abstracts the backend, so the same operations can be performed on a CPU, GPU, or future hardware backends.
The API consists of two interface classes:
Interface | Description |
---|---|
neon.backends.Tensor |
\(n\)-dimensional array data structure |
neon.backends.Backend |
Backend interface used to manipulate Tensor data |
Basic Data Structure¶
The Tensor
class is used to represent an arbitrary dimensional array
in which each element is stored using a consistent underlying type. For
the CPU and GPU backends, Tensors are stored as CPUTensor
and
GPUTensor
subclasses, respectively.
We have the ability to instantiate and copy instances of this data structure, as well as initialize its elements, reshape its dimensions, and access metadata.
Tensor Creation¶
Method | Description |
---|---|
neon.backends.Backend.empty |
Instantiate an empty Tensor |
neon.backends.Backend.array |
Instantiate a new Tensor, populating elements based on a provided array. |
neon.backends.Backend.zeros |
Instantiate a new Tensor, populating each element with the value of 0. |
neon.backends.Backend.ones |
Instantiate a new Tensor, populating each element with the value of 1. |
neon.backends.Tensor.copy |
Construct and return a deep copy of the Tensor passed. |
Tensor Manipulation¶
Method | Description |
---|---|
neon.backends.Tensor.get |
Convert the tensor to a host memory numpy.ndarray. |
neon.backends.Tensor.take |
Select a subset of elements from an array across an axis |
neon.backends.Tensor.__getitem__ |
Extract a subset view of the items via slice style indexing along each dimension. |
neon.backends.Tensor.__setitem__ |
Assign the specified value to a subset of elements found via slice style indexing along each dimension. |
neon.backends.Tensor.fill |
Assign specified value to each element of this Tensor. |
neon.backends.Tensor.transpose |
Return a transposed view of the data. |
neon.backends.Tensor.reshape |
Adjust the dimensions of the data to the specified shape. |
neon.backends.Backend.take |
Select a subset of elements (based on provided indices) from a supplied dimension |
Arithmetic Operations¶
Unary and binary arithmetic operations can be performed on Tensor
objects. In all cases, the user must pre-allocate correctly sized output
to store the result. All the below operations are performed
element-wise.
Element-wise Binary Operations¶
These element-wise binary operations perform the following operations.
An optional out=
argument can be passed to store the result as a
Tensor. If out=None
(default), an op-tree is returned.
Method | Description |
---|---|
neon.backends.Backend.add(a, b) |
\(a+b\) |
neon.backends.Backend.subtract(a,b) |
\(a-b\) |
neon.backends.Backend.multiply(a, b) |
\(a\times b\) |
neon.backends.Backend.divide(a, b) |
\(\frac{a}{b}\) |
neon.backends.Backend.dot(a, b) |
\(a \cdot b\) |
neon.backends.Backend.power(a, b) |
\(a^b\) |
neon.backends.Backend.maximum(a, b) |
\(\max(a, b)\) |
neon.backends.Backend.minimum(a, b) |
\(\min(a,b)\) |
neon.backends.Backend.clip(a, a_min, a_max) |
Clip each element of \(a\) between the corresponding elements in \(a_\text{min}\) and \(a_\text{max}\) |
Element-wise Unary Functions¶
These element-wise operations operate on one input Tensor or Op-tree.
Similar to the binary operations, an optional out=
argument can be
passed to store the result as a Tensor. If out=None
(default), an
op-tree is returned.
Method | Description |
---|---|
neon.backends.Backend.log(a) |
\(\log(a)\) |
neon.backends.Backend.log2(a) |
\(\log_2(a)\) |
neon.backends.Backend.exp(a) |
\(e^a\) |
neon.backends.Backend.exp2(a) |
\(2^a\) |
neon.backends.Backend.abs(a) |
\(abs(a)\) |
neon.backends.Backend.sgn(a) |
if \(x<0\), \(-1\); if \(x=0\), \(0\); if \(x>0\), \(1\) |
neon.backends.Backend.sqrt(a) |
\(\sqrt{a}\) |
neon.backends.Backend.square(a) |
\(a^2\) |
neon.backends.Backend.reciprocal(a) |
\(1/a\) |
neon.backends.Backend.negative(a) |
\(-a\) |
neon.backends.Backend.sig(a) |
\(1/(1+\exp(-a))\) |
neon.backends.Backend.tanh(a) |
\(\tanh(a)\) |
neon.backends.Backend.finite(a) |
Element-wise test for finiteness (e.g. \(a \neq \infty\)) |
Element-wise Logical Operations¶
These methods perform element-wise logical testing on each corresponding
element of the input \(a\) and \(b\). As before, an optional out=
argument can be passed to store the
output.
Method | Description |
---|---|
neon.backends.Backend.equal(a, b) |
\(a=b\) |
neon.backends.Backend.not_equal(a, b) |
\(a\neq b\) |
neon.backends.Backend.greater(a, b) |
\(a>b\) |
neon.backends.Backend.greater_equal(a, b) |
\(a \geq b\) |
neon.backends.Backend.less(a, b) |
\(a<b\) |
neon.backends.Backend.less_equal(a, b) |
\(a\leq b\) |
Summary Operations¶
These operations perform a summary calculation over a single provided
tensor a
along a specified axis
dimension. If axis=None
(default), the calculation is performed over all the dimensions.
Method | Description |
---|---|
neon.backends.Backend.sum |
Sum elements over the axis dimension |
neon.backends.Backend.mean |
Compute the arithmetic mean over the axis dimension |
neon.backends.Backend.var |
Compute the variance of the elements over the axis dimension |
neon.backends.Backend.std |
Compute the standard deviation of the elements along the axis dimension |
neon.backends.Backend.min |
Calculate the minimal element value over the axis dimension |
neon.backends.Backend.max |
Calculate the maximal element value over the axis dimension |
neon.backends.Backend.argmin |
Calculate the indices of the minimal element value along the axis dimension |
neon.backends.Backend.argmax |
Calculate the indices of the maximal element value along the axis dimension |
Random Number Generator¶
Both the NervanaGPU
and NervanaCPU
backends use the numpy random
number generator (np.random.RandomState
).
Method | Description |
---|---|
neon.backends.Backend.gen_rng |
Setup the numpy rng and store the initial state. Called by the backend constructor |
neon.backends.Backend.rng_get_state |
Return the state of the rng |
neon.backends.Backend.rng_set_state |
Set the state of the rng |
neon.backends.Backend.rng_reset |
Reset the state to the initial state |
neon.backends.NervanaGPU.rand |
Generate random numbers uniformly distributed between 0 and 1. |
To generate a Tensor with shape (100,100)
, where each element is
uniformly distributed between 0 and 1, we can call
myTensor = be.empty((100,100))
myTensor[:] = be.rand()
Loop indicators¶
These two methods signal the start (and end) of a block of repeated
computations, such as a loop. This operation can be used to help the
compiler optimize instruction performance, but has no direct effect on
the underlying calculations. Each Backend.start()
call must be
book-ended by a corresponding Backend.end()
call. Note that multiple
begin
calls can appear adjacent in nested loops.
Method | Description |
---|---|
neon.backends.Backend.begin |
Signal the start of a block |
neon.backends.Backend.end |
Signal the end of a block |
Higher-level Support¶
We have taken common operations used by neural network layers and
performed many optimizations. Many of these operations include custom
objects (such as ConvLayer
or PoolLayer
) which are used to track
the layer parameters and cache some calculations.
Operations¶
Method | Description |
---|---|
neon.backends.Backend.onehot |
Generate a one-hot representation from a set of indices (see Loading data) |
neon.backends.Backend.fill_normal |
Fill a tensor with gaussian random variables |
neon.backends.Backend.compound_dot |
Depending on the size of the input \(A\), \(B\), \(C\), perform \(\alpha A B + \beta C\) |
neon.backends.NervanaGPU.make_binary_mask |
Create a randomized binary mask for dropout layers |
neon.backends.NervanaCPU.make_binary_mask |
Create a randomized binary mask for dropout layers |
Convolutional Layers¶
Method | Description |
---|---|
neon.backends.Backend.conv_layer |
Create a ConvLayer object that holds the filter parameters. This is passed to the below functions. |
neon.backends.Backend.fprop_conv |
Forward propagate the inputs of a convolutional network layer |
neon.backends.Backend.bprop_conv |
Backward propagate the error through a convolutional network layer. |
neon.backends.Backend.update_conv |
Compute the updated gradient for a convolutional network layer. |
neon.backends.Backend.deconv_layer |
Create a DeconvLayer object that holds the filter parameters. This is passed to the above functions. |
Pooling Layers¶
Method | Description |
---|---|
neon.backends.Backend.pool_layer |
Create a new PoolLayer parameter object. |
neon.backends.Backend.fprop_pool |
Forward propagate pooling layer. |
neon.backends.Backend.bprop_pool |
Backward propagate pooling layer. |
The below methods implement ROI-pooling, where the pooling window sizes are themselves hyper-parameters.
Method | Description |
---|---|
neon.backends.Backend.roipooling_fprop |
Forward propagate through ROI-pooling layer |
neon.backends.Backend.roipooling_bprop |
Backward propagate through ROI-pooling layer |
Local Response Normalization¶
Method | Description |
---|---|
neon.backends.Backend.lrn_layer |
Create the LRNLayer parameter object |
neon.backends.Backend.fprop_lrn |
Forward propagate through LRN layer |
neon.backends.Backend.bprop_lrn |
Backward propagate through LRN layer |
Batch Normalization¶
Method | Description |
---|---|
neon.backends.Backend.compound_fprop_bn |
Forward propagate through BatchNorm layer |
neon.backends.Backend.compound_bprop_bn |
Backward propagate through BatchNorm layer |
Linear layer with bias¶
Method | Description |
---|---|
neon.backends.Backend.update_fc_bias |
Compute the updated bias gradient for a fully connected layer |
neon.backends.Backend.add_fc_bias |
Add bias to a fully connected layer |