================================================================================
full_rcr.py — Usage Explainer for LS_MODE_DL + perform_bulk_rejection
================================================================================
Audience: another Claude instance (or any LLM/developer) integrating full_rcr.py
into downstream code. Focused on the specific combination LS_MODE_DL + bulk +
parametric, which is what the integrator is using.

Generated 2026-05-21 from the rcrpy package at this revision.

================================================================================
1. WHAT "LS_MODE_DL" MEANS
================================================================================
Three-part designator:

  LS    Lower-Sigma: sigma is estimated from points BELOW the current mu
        estimate only. Designed for data with asymmetric contamination
        (one side has more outliers than the other).

  MODE  Uses the half-sample mode as the mu estimator. Robust against
        outliers without assuming the distribution is symmetric.

  DL    Double-Line sigma estimator (fitDL_w). Sigma is computed by a
        piecewise two-line fit to the cumulative weight-of-residuals
        curve. More accurate than the simpler 68th-percentile estimator
        at low contamination.

WHEN TO USE IT:
  - Mixed one-sided + two-sided contaminants.
  - Larger N (typically > ~30).
  - Data where the contamination distribution is asymmetric or unknown.
  - Use LS_MODE_68 instead if your data is symmetrically distributed with
    one-sided contaminants (it's faster and the simpler 68th-percentile
    sigma estimator is sufficient there).

================================================================================
2. perform_bulk_rejection VS perform_rejection
================================================================================
Both methods produce the same kind of output (mu, sigma, flags, parameters).
They differ only in the first pass:

  perform_rejection         (iterative): rejects ONE point at a time, refits
                            the model, then iterates. Slower for large N.

  perform_bulk_rejection    (4-pass):    Pass 1 rejects all qualifying points
                            at once in a single sweep, then runs three
                            iterative refinement passes (the user's chosen
                            technique, then median+68th, then mean+stdev).
                            Much faster for large N because it skips N-k
                            re-fits in Pass 1.

BOTH FULLY SUPPORT PARAMETRIC (FunctionalForm) MODELS.
Bulk rejection is NOT single-value-only; it just changes the Pass 1 behavior.

For most downstream use, perform_bulk_rejection is the right choice unless
you specifically need to inspect every rejection step.

================================================================================
3. CORRECT CALLING PATTERN
================================================================================
Single-value (no parametric model):

    import full_rcr
    import numpy as np

    y = np.array([...])
    r = full_rcr.RCR(full_rcr.RejectionTech.LS_MODE_DL)
    r.perform_bulk_rejection(y)

    mu_recovered    = r.result.mu
    sigma_recovered = r.result.sigma
    kept_flags      = r.result.flags      # boolean array
    kept_y          = y[r.result.flags]
    kept_indices    = r.result.indices

Functional-form (parametric model fit + rejection):

    import full_rcr
    import numpy as np

    # 1. Define the model and its partial derivatives w.r.t. each parameter.
    def my_model(x, params):
        return params[0] + params[1] * x        # example: linear

    def d_my_model_0(x, params):                # partial w.r.t. params[0]
        return 1.0

    def d_my_model_1(x, params):                # partial w.r.t. params[1]
        return x

    # 2. Build the FunctionalForm.
    model = full_rcr.FunctionalForm(
        my_model, x_data, y_data,
        [d_my_model_0, d_my_model_1],
        guess=[0.0, 1.0],
    )

    # 3. Create the RCR object with the technique.
    r = full_rcr.RCR(full_rcr.RejectionTech.LS_MODE_DL)

    # 4. Attach the model BEFORE calling rejection.
    r.set_parametric_model(model)

    # 5. Run bulk rejection. Pass the full y array (list or np.array).
    r.perform_bulk_rejection(y_data)

    # 6. Read results — NOTE: parameters live on the MODEL, not on r.result.
    fitted_params = model.result.parameters
    flags         = r.result.flags             # boolean: True = kept
    indices       = r.result.indices
    kept_y        = y_data[r.result.flags]
    sigma         = r.result.sigma             # residual sigma

================================================================================
4. CERTAINTY LEVELS BY USE-CASE
================================================================================
Aspect                                                  Certainty   Evidence
-----------------------------------------------------------------------------
Single-value LS_MODE_DL + bulk_rejection                Very high   Bit-identical
                                                                    to C++ at
                                                                    machine eps,
                                                                    30/30 trials.

Functional-form LS_MODE_DL + iterative                  High        Sampling now
                                                                    bit-identical
                                                                    (Option B);
                                                                    optimizer
                                                                    LM-vs-GN floor
                                                                    of ~1e-3 max
                                                                    measured. Port
                                                                    is at least
                                                                    as accurate
                                                                    as C++ on
                                                                    truth recovery.

Functional-form LS_MODE_DL + bulk_rejection             Medium-high Code path
                                                                    exists, uses
                                                                    same primitives.
                                                                    Less directly
                                                                    benchmarked
                                                                    in coverage
                                                                    suite, but
                                                                    expected to
                                                                    behave like
                                                                    iterative
                                                                    + faster.

Determinism (same input -> same output)                 Very high   C++ was non-
                                                                    deterministic;
                                                                    port has been
                                                                    deterministic
                                                                    since v0.1.0.
                                                                    Both now use
                                                                    MT19937(0xC0FFEE).

Memory leaks / state issues                             High        Pivot fix
                                                                    (2026-05-21)
                                                                    made the
                                                                    static class
                                                                    attribute
                                                                    reset per-
                                                                    instance.
                                                                    No known
                                                                    leaks.

================================================================================
5. COMMON PITFALLS
================================================================================
1. Read fitted parameters from model.result.parameters, NOT from r.result.
   The RCR result object holds flags, mu, sigma, indices — but the model
   itself carries the fitted parameter vector.

2. Don't reuse a FunctionalForm instance for sequential fits on different
   datasets. Construct a fresh one each time. Internal state is initialized
   at construction (linear-in-params detection, ND check, initial flags),
   and reuse can cause subtle bugs.

3. Partials must be scalar functions of (x, params). They take a scalar x
   (or a 1-D array of length D for ND inputs) and a params sequence, and
   return a scalar. Don't return arrays from a scalar-input partial.

4. Initial guess matters but isn't critical. The scipy LM optimizer is
   robust. Reasonable starting values (e.g. the result of a quick OLS fit)
   give faster convergence than [0.0, 0.0, ...].

5. Don't mix perform_rejection and perform_bulk_rejection on the same RCR
   instance. The internal state isn't designed for that. Create a fresh
   RCR instance for each fit.

6. Pass the FULL y array to the rejection method, not the already-rejected
   subset. The rejection routine tracks which points get cut via flags.

7. Weighted fits:
     - For sample weights (relative importance), pass w= to
       perform_bulk_rejection(y, w=weights).
     - For per-point error bars (sigma_y for chi-squared weighting), pass
       error_y= to the FunctionalForm constructor. These are different
       code paths.

8. **AVOID ES_MODE_DL WITH FUNCTIONAL-FORM MODELS.** There's a documented
   shared algorithm bug (runaway rejection cascade) affecting BOTH the
   port and the C++ oracle on this combination. The bug rejects ~75% of
   inliers even on perfectly clean data. The README has an explicit
   warning. Use LS_MODE_68, LS_MODE_DL, or SS_MEDIAN_DL instead.

================================================================================
6. PERFORMANCE EXPECTATIONS
================================================================================
For LS_MODE_DL + parametric on N ~= 400 with low (~5%) contamination:
  - C++ oracle:  30 - 60 seconds (varies by data)
  - full_rcr.py: 3 - 10 seconds
  - Port is typically 5-15x FASTER than C++ on functional-form workloads,
    because scipy's Levenberg-Marquardt is much faster than the C++'s
    hand-rolled Gauss-Newton.

For single-value LS_MODE_DL:
  - Port is ~10-12x slower than C++ on N=1000. Acceptable for typical
    use cases (small N or non-time-critical).
  - For large N where this matters, single-value LS_MODE_68 is ~1.3x
    slower than C++ and likely a better choice if the contamination
    pattern is amenable.

================================================================================
7. WHEN TO DEBUG
================================================================================
If you see unexpected results, the most useful diagnostics:

1. Parameters way off truth:
   - Check the initial guess and partial derivative functions.
   - Try model.regression() directly (no RCR) to confirm the fit alone
     recovers truth on the clean subset. If model.regression() fails too,
     the bug is in the model/partials definition, not in RCR.

2. All points being rejected:
   - The residual sigma may be collapsing to near zero each iteration.
   - Check whether the data actually has the structure the model assumes.
   - If using ES_MODE_DL with functional-form: that's the known bug;
     switch to LS_MODE_68/LS_MODE_DL/SS_MEDIAN_DL.

3. Inconsistent results across runs:
   - Shouldn't happen anymore (port is fully deterministic since v0.1.0).
   - If you observe it, file as a bug.

4. Slow performance on small N (~ N<50):
   - LS_MODE_DL has a heavier per-iteration cost than LS_MODE_68.
   - For very small N, the LS_MODE_68 cost is similar but more predictable.

================================================================================
8. RESULTS SCHEMA AFTER perform_bulk_rejection
================================================================================
After r.perform_bulk_rejection(y) returns, r.result contains:

  r.result.mu               scalar or array — mu estimate
                            (for parametric: per-point model values)
  r.result.sigma            scalar — residual sigma estimate
  r.result.flags            np.ndarray(bool, shape=(N,)) — True = kept
  r.result.indices          np.ndarray(int) — indices of kept points
  r.result.clean_y          np.ndarray — y values that survived
  r.result.rejected_y       np.ndarray — y values that were rejected
  r.result.original_y       np.ndarray — original input (echo of y)
  r.result.st_dev_total     scalar — pooled stdev (for compatibility)
  r.result.st_dev_above     scalar — sigma estimate from above-mu side
  r.result.st_dev_below     scalar — sigma estimate from below-mu side

For LS_MODE_DL specifically, r.result.sigma is the lower-sigma estimate
(derived from below-mu residuals). st_dev_above and st_dev_below are also
populated for diagnostic purposes.

For parametric fits, the FITTED PARAMETERS live on the model:
  model.result.parameters             np.ndarray — best-fit parameter vector
  model.result.parameter_uncertainties np.ndarray — diagonal sqrt(diag(J^T J)^-1)

================================================================================
9. ONE-LINE TL;DR FOR THE OTHER CLAUDE
================================================================================
"For LS_MODE_DL + perform_bulk_rejection on functional-form data:
the port is deterministic, ~5-15x faster than C++, at least as accurate
as C++ on truth recovery, and bit-identical to C++ on the sampling step.
Just don't use ES_MODE_DL with parametric models (shared algorithm bug
in both implementations)."

================================================================================
