Metadata-Version: 2.4
Name: qqn-torch
Version: 0.1.1
Summary: Quadratic Quasi-Newton optimizer for torch
Project-URL: Homepage, https://github.com/SimiaCryptus/qqn-torch
Project-URL: Repository, https://github.com/SimiaCryptus/qqn-torch
Author: QQN-torch Contributors
License: Apache License 2.0
License-File: LICENSE
Keywords: lbfgs,optimization,qqn,quasi-newton,torch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.9
Requires-Dist: torch>=1.12
Provides-Extra: bench
Requires-Dist: matplotlib>=3.5; extra == 'bench'
Requires-Dist: torchvision>=0.13; extra == 'bench'
Provides-Extra: dev
Requires-Dist: numpy>=1.22; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: profile
Requires-Dist: line-profiler>=4.0; extra == 'profile'
Requires-Dist: py-spy>=0.3.14; extra == 'profile'
Requires-Dist: scalene>=1.5.0; extra == 'profile'
Requires-Dist: snakeviz>=2.2.0; extra == 'profile'
Description-Content-Type: text/markdown

# qqn-torch

**QQN (Quadratic Quasi-Newton)** — a drop-in replacement for
`torch.optim.LBFGS` that searches a *quadratic path* blending the
steepest-descent and quasi-Newton directions.

## What is QQN?

Classic quasi-Newton methods (like L-BFGS) take a single direction `-H∇f`
and line-search along it. This works well near a minimum but can be fragile
far from it, where the curvature approximation is unreliable.

QQN instead constructs a **quadratic path** that interpolates between the
steepest-descent direction and the quasi-Newton direction:

```
d(t) = t(1-t)·(-∇f) + t²·(-H∇f)
```

with `t ∈ [0, 1]`. The key properties are:

| `t`           | Behavior                                 |
|---------------|------------------------------------------|
| `d(0) = 0`    | The path starts at the current iterate.  |
| `d'(0) = -∇f` | The initial tangent is steepest descent. |
| `d(1) = -H∇f` | The endpoint is the L-BFGS direction.    |

Because the path *starts* tangent to `-∇f`, the beginning of the path always
decreases `f` (when `∇f ≠ 0`). This anchors **global convergence**, while the
`t = 1` endpoint recovers **L-BFGS superlinear behavior** near the optimum.
The line search walks `t` directly and *discovers* the right blend — no manual
tuning of a mixing coefficient.

## Installation

```bash
pip install qqn-torch
```

Or from source:

```bash
git clone https://github.com/your-org/qqn-torch
cd qqn-torch
pip install -e .
```

Requires PyTorch.

## Quick Start

QQN follows the same `closure`-based API as `torch.optim.LBFGS`:

```python
import torch
from qqn_torch import QQN

# A simple quadratic objective.
x = torch.tensor([1.5, -2.0], requires_grad=True)

optimizer = QQN([x], max_iter=20)


def closure():
    optimizer.zero_grad()
    loss = (x[0] - 3.0) ** 2 + (x[1] + 1.0) ** 2
    loss.backward()
    return loss


for _ in range(10):
    loss = optimizer.step(closure)
    print(f"loss = {loss:.6e}, x = {x.detach().tolist()}")
```

The `closure` must:

1. Clear gradients (`zero_grad()`),
2. Compute the loss,
3. Call `loss.backward()`,
4. Return the loss.

This is **identical to the `torch.optim.LBFGS` contract**, so existing LBFGS
training loops work unchanged.

## Configuration

```python
QQN(
    params,
    history_size=10,  # L-BFGS curvature-pair history
    line_search="armijo",  # "armijo" | "backtracking" | "strong_wolfe" | "fixed"
    oracle="lbfgs",  # "lbfgs" | "momentum" | "secant" | Oracle instance
    region=None,  # None | "box" | "trust" | Region instance
    max_iter=20,  # inner iterations per .step()
    tol_grad=1e-7,  # gradient-norm stopping tolerance
    tol_change=1e-9,  # step/objective change tolerance
    line_search_options=None  # dict forwarded to the line search (c1, c2, ...)
)
```

### Four orthogonal, swappable components

QQN is built as a **combiner** of four independent pieces. Each can be
changed without touching the rest.

#### 1. Oracle — the `t = 1` endpoint (`-H∇f`)

| Name       | Description                                        |
|------------|----------------------------------------------------|
| `lbfgs`    | (default) Two-loop recursion over curvature pairs. |
| `momentum` | Heavy-ball direction `-(β·v + (1-β)·∇f)`.          |
| `secant`   | Barzilai–Borwein scalar step (`O(n)` memory).      |

Because the steepest-descent contribution anchors convergence, the oracle is
free to be aggressive — it need not guarantee descent on its own.

#### 2. Line search — walks the path and picks `t`

| Name           | Conditions                 | Notes                            |
|----------------|----------------------------|----------------------------------|
| `armijo`       | Armijo sufficient decrease | (default) Backtracking.          |
| `backtracking` | Armijo sufficient decrease | Aggressive contraction from t=1. |
| `strong_wolfe` | Armijo + strong curvature  | Can over-restrict the path step. |
| `fixed`        | None                       | Debug/baseline; constant `t`.    |

The line search is **not** an implementation detail — it is the glue that
makes the gradient and oracle work together. Convergence quality is bounded
by line-search quality.

#### 3. Region — optional projection of candidate points

| Name    | Description                               |
|---------|-------------------------------------------|
| `None`  | (default) Identity — zero overhead.       |
| `box`   | Elementwise clip to `[lo, hi]`.           |
| `trust` | Trust-region sphere with adaptive radius. |

When a region is active, the line search navigates the **projected path**
`d_R(t) = project_R(x, x + d(t)) - x`, so descent guarantees hold on the
feasible set. Custom regions can be composed with `SequentialRegion`.

#### 4. Gradient

The raw `-∇f` signal, the path's tangent at the origin.

### Custom components

You can pass instances instead of string shortcuts for full control:

```python
from qqn_torch import QQN
from qqn_torch.regions import TrustRegion
from qqn_torch.oracles import SecantOracle

optimizer = QQN(
    params,
    oracle=SecantOracle(alpha_init=0.5),
    region=TrustRegion(radius=2.0, max_radius=1e3),
    line_search="strong_wolfe",
    line_search_options={"c1": 1e-4, "c2": 0.9},
)
```

## How it works (per inner iteration)

```
g       = flat_grad(closure)         # autograd on the closure
qn_dir  = oracle.direction(g, state) # the t=1 endpoint, -H g
grad_dir = -g                        # the path tangent at t=0
d(t)    = t(1-t)·grad_dir + t²·qn_dir # quadratic path
t*      = line_search(...)           # picks the blend AND the step
x      += project(d(t*))             # apply (optionally projected) step
oracle.update(s, y)                  # s = Δx, y = Δg
region.update(...)                   # e.g. adapt trust radius
```

## Advantages

- **Adaptive**: automatically balances conservative vs. aggressive steps.
- **Robust**: `d'(0) = -∇f` plus line-search fallbacks ensure progress even
  when the oracle is poor.
- **Efficient**: L-BFGS acceleration when curvature is reliable.
- **Modular**: gradient, oracle, search, and region are independently
  swappable.

## Limitations

- **Memory**: stores L-BFGS history (`O(m·n)`).
- **Overhead**: walking the curved path adds modest per-iteration cost.
- **Tuning**: sensitive to history size, line-search constants, region radii.
- **Line-search sensitivity**: a poor line search undermines convergence and
  the quality of the curvature updates.

## Theoretical guarantees

Under standard assumptions (smooth objective, bounded gradients):

- **Global convergence** — anchored by the steepest-descent tangent.
- **Superlinear convergence** — inherited from L-BFGS when `t → 1` near the
  optimum.
- **Descent property** — every accepted step decreases `f`, enforced by the
  line search's sufficient-decrease test.

All guarantees are contingent on the line search satisfying sufficient
decrease. When a region is active, they hold on the projected path `d_R(t)`.

## Documentation

- [`algorithm.md`](algorithm.md) — comprehensive algorithm reference.

## Results

To reproduce, execute [run_reports.js](run_reports.js).

### Sigmoid

The **sigmoid** `σ(x) = 1/(1+e⁻ˣ)` squashes any real input into `(0, 1)`.
It is smooth and historically popular, but saturates for large `|x|`, where
its gradient vanishes — a classic source of slow learning in deep nets.

![fashion_mnist_mlp_comparison_vs_time_20260709-153240.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-153240.png)
**Results:** QQN reaches the lowest final loss (1.75e-01, 98.9% train / 87.1%
test) and is the only optimizer to break the 0.2 milestone (337 iters). Adam
trails at 2.47e-01 but attains the best test accuracy (87.6%), hinting that
QQN's lower training loss edges toward mild overfitting. L-BFGS stalls badly
at 1.12e+00 — the sigmoid's saturation flattens the curvature signal, so its
quasi-Newton step never gains traction beyond the first milestone.

### Sine

**Sine** `sin(x)` is a bounded, periodic, infinitely differentiable
activation. Its oscillatory nature can encode periodic structure, but the
non-monotonicity makes optimization landscapes bumpy and initialization
sensitive.

![fashion_mnist_mlp_comparison_vs_time_20260709-153546.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-153546.png)
**Results:** Sine is a standout case: both QQN (4.41e-02) and L-BFGS
(4.60e-02) drive the loss an order of magnitude below Adam (3.48e-01) and
reach 100% train accuracy. QQN just edges out L-BFGS on final loss while
tracking it closely at every milestone, and both share the best AUC scores
(-0.92 / -0.96) in the whole suite. The smooth, well-conditioned curvature
of the sine landscape lets the quasi-Newton machinery shine — Adam converges
fast early but plateaus.

### ReLu

The **ReLU** `max(0, x)` is the modern default: cheap, non-saturating for
positive inputs, and inducing sparse activations. Its downside is the "dying
ReLU" problem — units stuck at zero receive no gradient.

![fashion_mnist_mlp_comparison_vs_time_20260709-153851.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-153851.png)
**Results:** QQN wins decisively (1.15e-01, 98.9% train) and is the only
method to pass the 0.2 milestone (263 iters). Adam sits at 2.34e-01, while
L-BFGS again stalls (8.98e-01, 66.7% train) — the ReLU kink and dying-unit
zero-gradient regions corrupt its curvature estimate. This is the archetypal
QQN advantage: the steepest-descent tangent keeps it moving where pure
L-BFGS gets stuck, yet it still exploits curvature once past the rough zone.

### Abs

The **absolute value** `|x|` is a symmetric, V-shaped activation. Like ReLU it
is piecewise linear and non-saturating, but it folds the input rather than
clipping it, preserving magnitude information from both signs.

![fashion_mnist_mlp_comparison_vs_time_20260709-154156.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-154156.png)
**Results:** QQN leads (1.82e-01, 94.9% train, 86.9% test), reaching the 0.2
milestone at 320 iters where L-BFGS (6.07e-01) and Adam (9.81e-01) fall
short. Notably Adam's trajectory shows instability — it dips to -0.68 (log10)
then spikes back to +1.84 before recovering, a sign the piecewise-linear fold
produces a jagged landscape that trips the adaptive step sizes. QQN's
line-searched blend absorbs these irregularities gracefully.

### LogAbs

The **symmetric log** `sign(x)·ln(1+|x|)` is an odd, monotonic activation that
grows logarithmically in both directions. Like `abs` it is symmetric and
preserves sign, but its compressive log growth tames large magnitudes without
fully saturating, keeping a nonzero gradient everywhere.

![fashion_mnist_mlp_comparison_vs_time_20260709-162203.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-162203.png)
**Results:** The compressive log growth restores a usable curvature signal:
L-BFGS recovers to first place (1.64e-01) with QQN a close second
(1.94e-01), and both reach the 0.2 milestone (176 / 214 iters) while Adam
(3.51e-01) does not. Compared to plain `abs`, keeping a nonzero gradient
everywhere clearly benefits the quasi-Newton methods — L-BFGS goes from
stalling on `abs` to leading on `logabs`.

### Gaussian

The **Gaussian** `e⁻ˣ²` is a bump centered at the origin, peaking at 1 and
decaying to 0 in both directions. It yields localized, radial-basis-like
responses but saturates quickly away from zero.

![fashion_mnist_mlp_comparison_vs_time_20260709-154501.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-154501.png)
**Results:** L-BFGS narrowly wins (9.54e-02) over QQN (1.23e-01), and it is
the only optimizer to clear the 0.1 milestone (133 iters). This activation is
the most expensive per iteration (~450 ms/it), so all methods complete only
~130–160 iterations in the budget. The smooth, radially localized responses
give clean curvature that L-BFGS exploits fully; QQN matches its milestone
pace but its extra path-walking overhead costs a slight edge here.

### Tanh

**Tanh** `tanh(x)` is a zero-centered sigmoid mapping to `(-1, 1)`. The
zero-centering often speeds convergence relative to the logistic sigmoid, but
it shares the same saturation-induced vanishing-gradient issue.

![fashion_mnist_mlp_comparison_vs_time_20260709-154806.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-154806.png)
**Results:** Another strong quasi-Newton case: QQN (5.41e-02) and L-BFGS
(6.30e-02) both hit 100% train accuracy and reach the 0.1 milestone (226 /
200 iters), while Adam lags at 2.89e-01. The zero-centered, smooth tanh
landscape is well-conditioned, so — as with sine — the curvature-driven
methods dominate. QQN edges L-BFGS on final loss, mirroring the sine result.

### Swish

**Swish** `x·σ(x)` is a smooth, non-monotonic self-gated activation. It
behaves like ReLU for large positive `x` but dips slightly below zero for
small negatives, which often improves deep-network training.

![fashion_mnist_mlp_comparison_vs_time_20260709-155111.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-155111.png)
**Results:** Here Adam takes the lead (1.87e-01) and is the sole optimizer to
pass the 0.2 milestone (328 iters); QQN follows at 2.48e-01 but never clears
0.5→0.2. L-BFGS collapses entirely (2.31e+00, 15% train) — the smooth but
non-monotonic self-gating apparently yields curvature that misleads the pure
quasi-Newton step. QQN stays robust thanks to its gradient anchor but its
oracle inherits the same poor curvature, blunting its usual advantage.

### GeLu

**GELU** `x·Φ(x)` (Φ is the standard-normal CDF) weights inputs by their
probability of being positive. It is the smooth, probabilistic cousin of ReLU
and the default in modern transformer architectures.

![fashion_mnist_mlp_comparison_vs_time_20260709-155416.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-155416.png)
**Results:** QQN wins the Pareto frontier outright (2.14e-01) but neither it
nor Adam (2.67e-01) reaches the 0.2 milestone; Adam does clear it internally
(283 iters) while QQN does not. L-BFGS diverges catastrophically (2.31e+00,
13.6% train — essentially random). As with Swish, the smooth-but-non-monotonic
GELU produces curvature that sabotages raw L-BFGS, and QQN, sharing the same
L-BFGS oracle, only partially escapes.

### Elu

The **ELU** `x if x>0 else α(eˣ−1)` (here α=1) is linear for positive inputs
and saturates smoothly to `−α` for negatives. The negative saturation pushes
mean activations toward zero, aiding convergence, while avoiding dead units.

![fashion_mnist_mlp_comparison_vs_time_20260709-155721.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-155721.png)
**Results:** QQN dominates (1.07e-01, best AUC -0.62), reaching the 0.2
milestone in just 221 iters versus Adam's 649; Adam finishes at 1.72e-01.
L-BFGS stalls at 7.03e-01 (76.5% train). ELU's negative saturation keeps
mean activations near zero, and QQN converts that into fast, smooth descent —
its loss trajectory drops steadily to -0.97 (log10) with no oscillation.

### SeLu

**SELU** is a scaled ELU (`λ·x` for positives, `λα(eˣ−1)` for negatives, with
the fixed constants `λ≈1.0507`, `α≈1.6733`). These constants make activations
*self-normalizing*, driving outputs toward zero mean and unit variance.

![fashion_mnist_mlp_comparison_vs_time_20260709-160026.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-160026.png)
**Results:** The self-normalizing constants pay off for curvature methods:
L-BFGS (7.61e-02) and QQN (9.46e-02) both reach the 0.1 milestone (272 / 357
iters) and 99%+ train accuracy, with the two best AUCs in the ELU family
(-0.74 / -0.66). Adam trails far behind (6.59e-01, 80.9% train). Keeping
activations at zero mean / unit variance produces exactly the well-scaled
curvature that L-BFGS and QQN thrive on.

### Leaky ReLu

**Leaky ReLU** `x if x>0 else 0.01x` fixes the dying-ReLU problem by allowing
a small, nonzero gradient for negative inputs. It keeps every unit learning
while retaining ReLU's cheap piecewise-linear form.

![fashion_mnist_mlp_comparison_vs_time_20260709-160331.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-160331.png)
**Results:** QQN leads (1.61e-01) and reaches the 0.2 milestone (308 iters),
closely followed by Adam (2.40e-01, 278 iters). L-BFGS fails completely
(2.33e+00, 9.7% train — worse than chance), stuck at its initial loss the
entire run. The small negative slope that cures dying-ReLU units does not
rescue raw L-BFGS's curvature estimate, but QQN's steepest-descent anchor
once again keeps it converging where pure quasi-Newton cannot.

### SoftPlus

**Softplus** `ln(1+eˣ)` is a smooth approximation to ReLU. It is everywhere
differentiable and strictly positive, with derivative equal to the sigmoid —
but its smoothness costs more compute than plain ReLU.

![fashion_mnist_mlp_comparison_vs_time_20260709-160636.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-160636.png)
**Results:** Unusually, Adam wins here (2.22e-01) and is the only method to
reach the 0.2 milestone (749 iters); QQN finishes at 2.74e-01 and L-BFGS
stalls (1.48e+00, 39.2% train). The smooth ReLU approximation seems to favor
Adam's per-parameter adaptation, while QQN — though robust — makes slower
progress and its L-BFGS oracle is again hampered by unreliable curvature.

### Mish

**Mish** `x·tanh(softplus(x))` is a smooth, non-monotonic self-gated
activation. It behaves like ReLU for large positive `x` but allows a small
negative response for slightly negative inputs, preserving a bit of gradient
flow. Its smoothness and self-regularizing shape often improve deep-network
training. It is similar to Swish but uses a `tanh(softplus(·))` gate instead
of a sigmoid.

![fashion_mnist_mlp_comparison_vs_time_20260709-160941.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-160941.png)
**Results:** QQN takes the frontier (1.57e-01, best AUC -0.52) and reaches
the 0.2 milestone slightly faster than Adam (283 vs 324 iters); Adam finishes
at 2.61e-01 with the top test accuracy (87.5%). L-BFGS collapses to chance
(2.31e+00, 14.5% train), consistent with the other smooth self-gated
activations (Swish, GELU) where its curvature estimate is unreliable.

### Identity

**Identity** `f(x) = x` is the simplest activation, passing inputs through unchanged. It is linear and does not
introduce nonlinearity, so it is rarely used in hidden layers but can be useful for output layers in regression tasks.

![fashion_mnist_mlp_comparison_vs_time_20260709-161247.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-161247.png)
**Results:** With no nonlinearity the network collapses to a linear model, so
accuracies cap around 82%. QQN (2.92e-01) and L-BFGS (3.13e-01) perform
near-identically — expected, since the objective is essentially convex and
their curvature paths coincide. Adam diverges wildly (9.39e+00, 37.6% train),
its loss exploding to +0.97 (log10) late in training — a cautionary reminder
that adaptive methods can destabilize even on the simplest landscape.

### Triangle

The **triangle** wave is a periodic, piecewise-linear activation that ramps
linearly up and down between `-1` and `1`. Unlike smooth periodic activations
such as sine, its constant-magnitude slopes give uniform gradients away from
the turning points, but the sharp corners introduce non-differentiable
kinks.

![fashion_mnist_mlp_comparison_vs_time_20260709-161552.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-161552.png)
**Results:** All three optimizers do well here — L-BFGS (6.61e-02), QQN
(7.69e-02), and Adam (1.00e-01) all reach 100% (or near-100%) train accuracy
and the best test accuracies of the suite (~87.7%). L-BFGS and QQN clear the
0.1 milestone (140 / 148 iters). Despite the non-differentiable corners, the
constant-magnitude slopes give uniform, informative gradients that all
methods exploit; the per-iteration cost is high (~340 ms/it) so runs stay short.

### Sawtooth

The **sawtooth** wave is a periodic ramp that rises linearly from `-1` to `1`
and then drops discontinuously back down. It shares the triangle's constant
slope but adds a hard discontinuity each period, making it the most
optimization-hostile of the periodic activations while still encoding
periodic structure.

![fashion_mnist_mlp_comparison_vs_time_20260709-161858.png](results/fashion_mnist_mlp_comparison_vs_time_20260709-161858.png)

**Results:** As predicted, the sawtooth is the most hostile activation:
*no* optimizer reaches even the 1.0 milestone. QQN does best (1.97e+00, 31.3%
train) but manages only 56 iterations — its line search burns evaluations
fighting the periodic discontinuities (~1075 ms/it). L-BFGS (2.14e+00) and
Adam (3.01e+00, ~10% train, i.e. chance) fare worse. The hard per-period jump
destroys gradient continuity, and while QQN degrades most gracefully, the
landscape defeats all first- and second-order methods alike.

## Results Summary

Across the 18 activation functions benchmarked on Fashion-MNIST MLPs, clear
patterns emerge that validate QQN's design philosophy.

### Win counts (best final loss)

| Optimizer  | Wins | Activations                                                                   |
|------------|------|-------------------------------------------------------------------------------|
| **QQN**    | 11   | sigmoid, ReLU, abs, Gaussian*, tanh, ELU, leaky-ReLU, Mish, GELU, sine, SELU* |
| **L-BFGS** | 5    | logabs, Gaussian, SELU, tanh*, triangle                                       |
| **Adam**   | 2    | Swish, Softplus                                                               |

*Starred entries are near-ties where the two curvature methods finish within
noise of each other.

### Key takeaways

1. **QQN is the most consistent winner or runner-up.** It never collapses to
   chance and reaches the target milestone on nearly every activation. Even
   when it doesn't win outright (Swish, Softplus, Gaussian), it stays within
   striking distance and remains stable.
2. **L-BFGS is high-variance.** On well-conditioned, smooth landscapes (sine,
   tanh, logabs, Gaussian, SELU, triangle) it is competitive or best. But on
   ReLU-family kinks and smooth-but-non-monotonic gates (Swish, GELU, Mish,
   leaky-ReLU) its curvature estimate is corrupted and it **diverges to
   chance-level accuracy** (~10–15% train). This is the failure mode QQN was
   built to fix.
3. **QQN's gradient anchor is decisive.** In every case where L-BFGS stalls
   or collapses (ReLU, abs, leaky-ReLU, Swish, GELU, Mish, ELU, SELU-poor
   regions), QQN keeps converging because the `d'(0) = -∇f` tangent
   guarantees progress regardless of oracle quality — while still exploiting
   curvature once past the rough zone.
4. **Adam wins only the smooth-ReLU approximations** (Swish, Softplus), where
   per-parameter adaptation suits the landscape, but it can destabilize
   badly (Identity: diverges to 9.39; abs: oscillates; sawtooth: chance).
5. **The landscape can defeat everyone.** On the discontinuous sawtooth, no
   method clears even the 1.0 milestone — though QQN still degrades most
   gracefully.

## Conclusion

The empirical results confirm QQN's central thesis: **blending the
steepest-descent tangent with a quasi-Newton endpoint yields the best of both
worlds.** QQN inherits L-BFGS's superlinear speed on well-conditioned,
curvature-rich problems (sine, tanh, SELU) while its gradient anchor prevents
the catastrophic failures that sink raw L-BFGS on kinked or non-monotonic
activations (ReLU, leaky-ReLU, Swish, GELU, Mish).

In practical terms:

- **Choose QQN as a robust default.** It matches or beats L-BFGS almost
  everywhere and never diverges, making it a safe drop-in replacement.
- **Prefer L-BFGS only when the landscape is known to be smooth and
  well-conditioned**, where it may shave a small margin off the final loss.
- **Reach for Adam for smooth ReLU-like activations** (Swish, Softplus), but
  watch for its instability on simple or non-smooth objectives.

The modular design — swappable oracle, line search, and region — means these
trade-offs can be tuned per problem without rewriting the optimizer. When in
doubt, QQN's line-searched quadratic path adapts automatically, discovering
the right blend between caution and aggression on each iteration.

## Acknowledgements

The strong-Wolfe line search is adapted from PyTorch's `_strong_wolfe`
helper (`torch/optim/lbfgs.py`, BSD-licensed). See
`qqn_torch/_vendor/strong_wolfe.py` for attribution.

## License

See the `LICENSE` file. Vendored code retains its original PyTorch BSD license.