Binary step / heaviside step

To use with vectors (R), use vmap'.

\[ \begin{cases} 0 & \text{for } x < 0 \\ 1 & \text{for } x \ge 0 \end{cases} \]

Logistic function

\[ \sigma(x) = \frac{1}{1 + e^{-x}} \]

Softsign activation function

\[ \frac{x}{1 + \lvert x \rvert} \]

Rectified linear unit.

To use with vectors (R), use vmap'.

\[ \max(0,x) \]

reLU = preLU 0

SoftPlus

\[ \ln(1 + e^x) \]

Bent identity

\[ \frac{\sqrt{x^2 + 1} - 1}{2} + x \]

Sigmoid-weighted linear unit. Multiply logistic by its input.

\[ x \sigma(x) \]

SoftExponential

To use with vectors (R), use vmap'.

\[ \begin{cases} - \frac{\ln(1 - \alpha (x + \alpha))}{\alpha} & \text{for } \alpha < 0 \\ x & \text{for } \alpha = 0 \\ \frac{e^{\alpha x} - 1}{\alpha} + \alpha & \text{for } \alpha > 0 \end{cases} \]

Sinc

\[ \begin{cases} 1 & \text{for } x = 0 \\ \frac{\sin(x)}{x} & \text{for } x \ne 0 \end{cases} \]

Gaussian

\[ e^{-x^2} \]

Note: if possible, use the potentially much more performant vmap'.

vmap, but potentially more performant. Only usable if the mapped function does not depend on any external BVars.

Usable with functions like *, isru, etc. to turn them into a form usable with PFP:

liftUniform (*)  :: BVar s Double -> BVar s (R n) -> BVar s (R n)
liftUniform isru :: BVar s Double -> BVar s (R n) -> BVar s (R n)

Basically turns a parmaeterized function on individual elements of into one that shares the same parameter across all elements of the vector.

Inverse square root unit

\[ \frac{x}{\sqrt{1 + \alpha x^2}} \]

See liftUniform to make this compatible with PFP.

You can also just use this after partially applying it, to fix the parameter (and not have it trained).

Parametric rectified linear unit

To use with vectors (R), use vmap'.

If scaling parameter is a fixed (and not learned) parameter, this is typically called a leaky recitified linear unit (typically with α = 0.01).

To use as a learned parameter:

vmap . preLU :: BVar s Double -> BVar s (R n) -> BVar s (R n)

This can be give directly to PFP.

To fix the paramater ("leaky"), just partially apply a parameter:

preLU 0.01           :: BVar s (R n) -> BVar s (R n)
preLU (realToFrac α) :: BVar s (R n) -> BVar s (R n)

See also rreLU.

\[ \begin{cases} \alpha x & \text{for } x < 0 \\ x & \text{for } x \ge 0 \end{cases} \]

S-shaped rectified linear activiation unit

See sreLUPFP for an uncurried and uniformly lifted version usable with PFP.

\[ \begin{cases} t_l + a_l (x - t_l) & \text{for } x \le t_l \\ x & \text{for } t_l < x < t_r \\ t_r + a_r (x - t_r) & \text{for } x \ge t_r \end{cases} \]

An uncurried and uniformly lifted version of sreLU directly usable with PFP.

Exponential linear unit

To use with vectors (R), use vmap'.

To use as a learned parameter:

vmap . eLU :: BVar s Double -> BVar s (R n) -> BVar s (R n)

This can be give directly to PFP.

To fix the paramater, just partially apply a parameter:

vmap' (eLU 0.01) :: BVar s (R n) -> BVar s (R n)

\[ \begin{cases} \alpha (e^x - 1) & \text{for } x < 0 \\ x & \text{for } x \ge 0 \end{cases} \]

Inverse square root linear unit

To use with vectors (R), use vmap'.

\[ \begin{cases} \frac{x}{1 + \alpha x^2} & \text{for } x < 0 \\ x & \text{for } x \ge 0 \end{cases} \]

Adaptive piecewise linear activation unit

See aplPFP for an uncurried version usable with PFP.

\[ \max(0, x_i) + \sum_j^M a_i^j \max(0, -x_i + b_i^j) \]

apl uncurried, to be directly usable with PFP.

Softmax normalizer

Maximum of vector.

Compare to norm_InfV, which gives the maximum absolute value.

Keep only the top k values, and zero out all of the rest.

Useful for postcomposing in between layers (with a logistic function before) to encourage the number of "activated" neurons is kept to be around k. Used in k-Sprase autoencoders (see KAutoencoder).

http://www.ericlwilkinson.com/blog/2014/11/19/deep-learning-sparse-autoencoders