# Activation functions¶

Activation functions such as the rectified linear unit (ReLu) or sigmoid
are treated as layers within neon. For convenience, these functions are
wrapped inside the `Activation`

layer, which takes
care of a lot of the layer-specific verbiage. Neon has the following
activation functions:

Name | Description |
---|---|

`neon.transforms.Identity` |
\(f(x) = x\) |

`neon.transforms.Rectlin` |
\(f(x) = \max(x, 0)\) |

`neon.transforms.Explin` |
\(f(x) = \max(x, 0) + \alpha (e^{\min(x, 0)}-1)\) |

`neon.transforms.Normalizer` |
\(f(x) = x / \alpha\) |

`neon.transforms.Softmax` |
\(f(x_j) = \frac{\exp{x_j}}{\sum_i \exp {x_i}}\) |

`neon.transforms.Tanh` |
\(f(x) = \tanh(x)\) |

`neon.transforms.Logistic` |
\(f(x) = \frac{1}{1+e^{-x}}\) |

## Creating custom activations¶

To create a new activation function, subclass from `Transform`

and
implement the `bprop()`

and `__call__()`

methods.

As an example, we implement the ReLu function:

```
class MyReLu(Transform):
" ReLu activation function. Implements f(x) = max(0,x) "
def __init__(self, name=None):
super(MyReLu, self).__init__(name)
# f(x) = max(0,x)
def __call__(self, x):
return self.be.maximum(x, 0)
# If x > 0, gradient is 1; otherwise 0.
def bprop(self, x):
return self.be.greater(x, 0)
```

Both methods receive as input a `Tensor`

`x`

.

In most models, activation functions are appended to a filtering (e.g.
convolution) or linear (all-to-all) layer. For this reason, Neon
provides several convenient `CompoundLayer`

classes (`Affine`

, `Conv`

, and `Deconv`

. For example, a linear layer followed by your
ReLu function can be instantiated via

```
layers = [Affine(nout = 1000, activation=MyReLu())]
```