snn.optimizers

Base

class snn.optimizers.base.Optimizer(learning_rate=0.01)

Bases: object

apply_gradients(params, grads)

get_config()

First-Order

class snn.optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False, weight_decay=0.0)

Bases: Optimizer

Stochastic Gradient Descent with optional momentum and Nesterov acceleration.

apply_gradients(params, grads)

get_config()

class snn.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, weight_decay=0.0, amsgrad=False)

Bases: Optimizer

Adam optimizer (Adaptive Moment Estimation).

apply_gradients(params, grads)

get_config()

class snn.optimizers.AdamW(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, weight_decay=0.01)

Bases: Adam

AdamW — Adam with decoupled weight decay regularization.

apply_gradients(params, grads)

class snn.optimizers.RMSprop(learning_rate=0.001, rho=0.9, epsilon=1e-08, momentum=0.0, weight_decay=0.0, centered=False)

Bases: Optimizer

RMSprop optimizer.

apply_gradients(params, grads)

get_config()

Adaptive

class snn.optimizers.Adagrad(learning_rate=0.01, epsilon=1e-08, initial_accumulator_value=0.1, weight_decay=0.0)

Bases: Optimizer

Adagrad optimizer — adapts learning rates using accumulated squared gradients.

apply_gradients(params, grads)

get_config()

class snn.optimizers.Adadelta(learning_rate=1.0, rho=0.95, epsilon=1e-07, weight_decay=0.0)

Bases: Optimizer

Adadelta optimizer — adapts learning rates without a global learning rate.

apply_gradients(params, grads)

get_config()

Modern

class snn.optimizers.Nadam(learning_rate=0.002, beta_1=0.9, beta_2=0.999, epsilon=1e-08, weight_decay=0.0)

Bases: Optimizer

Nadam — Nesterov-accelerated Adaptive Moment Estimation.

Combines Adam’s adaptive learning rates with Nesterov momentum, giving faster convergence than standard Adam in many settings.

f(θ) update rule:

m_t  = β₁ · m_{t-1} + (1 − β₁) · g_t
v_t  = β₂ · v_{t-1} + (1 − β₂) · g_t²
m̂    = β₁ · m_t / (1−β₁^{t+1}) + (1−β₁) · g_t / (1−β₁^t)
θ_t  = θ_{t-1} − lr · m̂ / (√v̂ + ε)

Parameters:

• learning_rate (float) – Step size (default 2e-3).

• beta_1 (float) – Exponential decay for first moment (default 0.9).

• beta_2 (float) – Exponential decay for second moment (default 0.999).

• epsilon (float) – Numerical stability constant (default 1e-8).

• weight_decay (float) – L2 regularisation coefficient (default 0.0).

apply_gradients(params, grads)

get_config()

class snn.optimizers.RAdam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, weight_decay=0.0)

Bases: Optimizer

RAdam — Rectified Adam.

Fixes the variance issue in Adam’s early training phase by computing the effective sample size of the second moment estimate. When the approximation is not reliable (small t), it falls back to an SGD-with- momentum step; otherwise it applies the full variance-corrected Adam update. This removes the need for a warmup schedule.

Reference: Liu et al. (2019) “On the Variance of the Adaptive Learning Rate and Beyond”. https://arxiv.org/abs/1908.03265

Parameters:

• learning_rate (float) – Step size (default 1e-3).

• beta_1 (float) – First-moment decay (default 0.9).

• beta_2 (float) – Second-moment decay (default 0.999).

• epsilon (float) – Numerical stability constant (default 1e-8).

• weight_decay (float) – L2 regularisation coefficient (default 0.0).

apply_gradients(params, grads)

get_config()

class snn.optimizers.Lion(learning_rate=0.0001, beta_1=0.9, beta_2=0.99, weight_decay=0.0)

Bases: Optimizer

Lion — EvoLved Sign Momentum.

Uses only the sign of the update vector, making every parameter receive an update of exactly ±learning_rate per step. This gives Lion 3–10× lower memory usage than Adam (one momentum buffer instead of two) and competitive or better performance on image and language tasks.

Update rule:

c_t  = β₁ · m_{t-1} + (1 − β₁) · g_t
θ_t  = θ_{t-1} − lr · (sign(c_t) + λ · θ_{t-1})
m_t  = β₂ · m_{t-1} + (1 − β₂) · g_t

Note: Lion typically needs a 3–10× smaller learning rate and 3–10× larger weight decay than Adam. A good starting point is lr=1e-4, weight_decay=1e-2.

Reference: Chen et al. (2023) “Symbolic Discovery of Optimization Algorithms”. https://arxiv.org/abs/2302.06675

Parameters:

• learning_rate (float) – Step size (default 1e-4).

• beta_1 (float) – Coefficient for computing the update direction (default 0.9).

• beta_2 (float) – Coefficient for maintaining the momentum buffer (default 0.99).

• weight_decay (float) – Decoupled weight decay (default 0.0).

apply_gradients(params, grads)

get_config()

class snn.optimizers.LAMB(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-06, weight_decay=0.0, clip_ratio=10.0)

Bases: Optimizer

LAMB — Layer-wise Adaptive Moments optimizer (Ginsburg et al. 2019).

Extends Adam with a layer-wise trust ratio that scales the update by ‖θ‖ / ‖adam_update‖. This allows training with very large batch sizes (e.g. 65 536) without learning-rate tuning, and is the standard optimizer for BERT-style pretraining.

Update rule:

m_t = β₁·m_{t-1} + (1−β₁)·g_t
v_t = β₂·v_{t-1} + (1−β₂)·g_t²
m̂   = m_t/(1−β₁^t)   v̂ = v_t/(1−β₂^t)
u   = m̂/(√v̂ + ε) + λ·θ
r   = clip(‖θ‖/‖u‖, 0, clip_ratio)
θ   = θ − lr·r·u

Note

LAMB is designed for large batch training. For small batches (< 512) Adam or AdamW will generally perform equally well.

Parameters:

• learning_rate (float) – Base step size (default 1e-3).

• beta_1 (float) – Moment decay rates (default 0.9, 0.999).

• beta_2 (float) – Moment decay rates (default 0.9, 0.999).

• epsilon (float) – Numerical stability (default 1e-6).

• weight_decay (float) – L2 regularisation coefficient λ (default 0.0).

• clip_ratio (float) – Upper bound for the trust ratio (default 10.0).

apply_gradients(params, grads)

get_config()

class snn.optimizers.Adan(learning_rate=0.001, beta_1=0.98, beta_2=0.92, beta_3=0.99, epsilon=1e-08, weight_decay=0.02)

Bases: Optimizer

Adan — Adaptive Nesterov Momentum Algorithm (Xie et al. 2022).

Uses three exponential moving averages — of the gradient, the gradient difference, and a combined Nesterov-like term — to get fast convergence on non-convex problems. Achieves competitive or better results than Adam on image classification and NLP benchmarks.

Update rule:

dk  = gk − g_{k-1}           (gradient difference)
m1  = β₁·m1 + (1−β₁)·gk
m2  = β₂·m2 + (1−β₂)·dk
m3  = β₃·m3 + (1−β₃)·(gk + (1−β₂)·dk)²
η   = lr / (√m3 + ε)
θ   = (1 + λ·lr)⁻¹ · (θ − η · (m1 + (1−β₂)·m2))

Parameters:

• learning_rate (float) – Step size (default 1e-3).

• beta_1 (float) – Decay for first moment (default 0.98).

• beta_2 (float) – Decay for gradient difference (default 0.92).

• beta_3 (float) – Decay for second-order moment (default 0.99).

• epsilon (float) – Numerical stability (default 1e-8).

• weight_decay (float) – Decoupled weight decay λ (default 0.02).

apply_gradients(params, grads)

get_config()

class snn.optimizers.Lookahead(optimizer, k=5, alpha=0.5)

Bases: Optimizer

Lookahead optimizer wrapper (Zhang et al. 2019).

Wraps any base optimizer and adds a slow-weights outer loop:

• Inner loop — the base optimizer updates “fast weights” for k steps as usual.

• Outer update — after every k inner steps, the slow weights interpolate toward the fast weights:

  θ_slow ← θ_slow + α · (θ_fast − θ_slow)
  θ_fast ← θ_slow
  

This stabilises training across a wide range of learning rates and often improves generalisation with minimal overhead.

Parameters:

• optimizer (Optimizer) – Any snn optimizer instance (Adam, SGD, Nadam, …).

• k (int) – Number of inner (fast) steps before each slow update (default 5).

• alpha (float) – Slow-weights interpolation coefficient (default 0.5).

Examples

>>> from snn.optimizers import Adam, Lookahead
>>> opt = Lookahead(Adam(learning_rate=1e-3), k=5, alpha=0.5)
>>> model.compile(opt, "categorical_crossentropy")

property learning_rate

property weight_decay

apply_gradients(params, grads)

get_config()

Factory

snn.optimizers.get(identifier)

Return an Optimizer instance from a string, instance, or dict.

Parameters:

identifier (str, Optimizer, or dict) –

• String key — "adam", "sgd", …

• Optimizer instance — returned unchanged.

• Config dict — {"name": "adam", "learning_rate": 1e-3}. Any key accepted by the optimizer constructor may appear; "name" (or "class") selects the class.

Raises:

ValueError – Unknown string or dict name.