snn.activations

Base

class snn.activations.Activation

Bases: object

Base class for all activation functions.

forward(x)

backward(grad)

Classic activations

class snn.activations.Linear

Bases: Activation

Identity / no-op activation. f(x) = x.

forward(x)

backward(grad)

class snn.activations.ReLU

Bases: Activation

Rectified Linear Unit. f(x) = max(0, x).

forward(x)

backward(grad)

class snn.activations.LeakyReLU(alpha=0.01)

Bases: Activation

Leaky ReLU. f(x) = x if x > 0 else alpha*x.

Parameters:

alpha (float) – Slope for negative inputs (default 0.01).

forward(x)

backward(grad)

class snn.activations.ELU(alpha=1.0)

Bases: Activation

Exponential Linear Unit. f(x) = x if x > 0 else alpha*(exp(x)-1).

Parameters:

alpha (float) – Scale for negative saturation region (default 1.0).

forward(x)

backward(grad)

class snn.activations.SELU

Bases: Activation

Scaled ELU — self-normalising activation for deep networks.

forward(x)

backward(grad)

class snn.activations.Sigmoid

Bases: Activation

Logistic sigmoid. f(x) = 1 / (1 + exp(-x)).

forward(x)

backward(grad)

class snn.activations.Tanh

Bases: Activation

Hyperbolic tangent. f(x) = tanh(x).

forward(x)

backward(grad)

class snn.activations.Softmax

Bases: Activation

Softmax over the last axis. Typically used on the output layer for multi-class classification.

forward(x)

backward(grad)

class snn.activations.Softplus

Bases: Activation

Smooth approximation of ReLU. f(x) = log(1 + exp(x)).

forward(x)

backward(grad)

class snn.activations.Swish

Bases: Activation

Swish / SiLU. f(x) = x * sigmoid(x). Smooth, non-monotonic.

forward(x)

backward(grad)

class snn.activations.Mish

Bases: Activation

Mish. f(x) = x * tanh(softplus(x)). Self-regularising, smooth.

forward(x)

backward(grad)

Non-linear / smooth activations

These activations are particularly effective when the target function is smooth or polynomial-like (e.g. y = x², signal reconstruction, physics- informed networks).

class snn.activations.GELU

Bases: Activation

Gaussian Error Linear Unit.

Uses the tanh approximation: f(x) ≈ 0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x³)))

The standard choice in modern Transformers (BERT, GPT). Smooth and non-monotonic; outperforms ReLU/ELU on many regression and NLP tasks.

forward(x)

backward(grad)

class snn.activations.PReLU(alpha=0.25)

Bases: Activation

Parametric ReLU with a learnable negative slope.

f(x) = x if x ≥ 0 f(x) = alpha * x if x < 0

alpha is updated by the optimiser just like any weight — pass the layer that contains this activation through a Sequential model and alpha will be trained automatically.

Parameters:

alpha (float) – Initial slope for the negative region (default 0.25).

forward(x)

backward(grad)

property params

property grads

class snn.activations.Sine

Bases: Activation

Sine activation for SIREN (Sinusoidal Representation Networks).

f(x) = sin(x)

Designed for learning smooth, periodic, and polynomial-like functions. A network with sine activations can exactly represent derivatives of itself and naturally approximates functions like y = x², y = sin(x), or any smooth manifold.

Initialisation note: use kernel_initializer="siren_uniform" (or scale weights by 1/fan_in for hidden layers) for best convergence. As a quick start, glorot_uniform works for shallow networks.

forward(x)

backward(grad)

class snn.activations.Hardswish

Bases: Activation

Hard Swish activation (MobileNetV3).

f(x) = x * ReLU6(x + 3) / 6

Piecewise-smooth approximation of Swish. Efficient for deployment while retaining the non-monotonic character of Swish.

forward(x)

backward(grad)

class snn.activations.BentIdentity

Bases: Activation

Bent Identity activation.

f(x) = (√(x² + 1) − 1) / 2 + x

Smooth everywhere, non-saturating, and non-linear at every point. The derivative is always positive (≥ 1) so it avoids the dying-neuron problem while still introducing the curvature needed to fit polynomial or quadratic data. Good default for regression tasks.

forward(x)

backward(grad)

class snn.activations.Squareplus(b=4.0)

Bases: Activation

Squareplus — smooth, monotonic ReLU alternative.

f(x) = (x + √(x² + b)) / 2

Always positive and differentiable, with quadratic behaviour near the origin controlled by b. Approaches ReLU as b → 0.

Parameters:

b (float) – Curvature at origin (default 4.0). Larger values = smoother transition.

forward(x)

backward(grad)

Recent additions (efficiency + NLP)

class snn.activations.ReLU6

Bases: Activation

ReLU capped at 6. f(x) = min(max(0, x), 6).

Used in MobileNet and other efficient architectures. The cap at 6 improves fixed-point quantisation precision.

forward(x)

backward(grad)

class snn.activations.Hardsigmoid

Bases: Activation

Piecewise-linear sigmoid approximation.

f(x) = clip((x + 3) / 6, 0, 1)

Computationally cheaper than the exact sigmoid; commonly used in quantisation-friendly networks (MobileNetV3).

forward(x)

backward(grad)

class snn.activations.LogSoftmax

Bases: Activation

Log-softmax. f(x) = x − log(Σ exp(x)).

Numerically stable; used together with negative log-likelihood loss (equivalent to CrossEntropy = NLLLoss(LogSoftmax(x))).

forward(x)

backward(grad)

class snn.activations.Sparsemax

Bases: Activation

Sparsemax — differentiable sparse softmax (Martins & Astudillo 2016).

Like softmax but returns a sparse probability distribution: many outputs are exactly 0. Useful for attention mechanisms where hard, interpretable focus is desired.

f(x) = max(0, x − τ(x)) where τ is the threshold that makes outputs sum to 1.

forward(x)

backward(grad)

class snn.activations.CELU(alpha=1.0)

Bases: Activation

Continuously Differentiable ELU (Barron 2017).

f(x) = x if x >= 0 else alpha*(exp(x/alpha) - 1)

Unlike ELU, CELU is continuously differentiable at x=0 for any alpha.

Parameters:

alpha (float) – Scale for the negative branch (default 1.0).

forward(x)

backward(grad)

class snn.activations.Softsign

Bases: Activation

Softsign activation. f(x) = x / (1 + |x|).

A smooth, bounded alternative to Tanh with heavier tails. Saturates more slowly, which can help with gradient flow.

forward(x)

backward(grad)

class snn.activations.Tanhshrink

Bases: Activation

Tanhshrink activation. f(x) = x - tanh(x).

A soft-thresholding function that shrinks values toward zero. Used in sparse autoencoders and shrinkage estimators.

forward(x)

backward(grad)

Factory

snn.activations.get(identifier)

Return an Activation instance from a string or object.

Parameters:

identifier (str, Activation, or None) – None or "linear" → Linear.

Raises:

ValueError – If the string key is not recognised.