spacr.ml
========

.. py:module:: spacr.ml






Module Contents
---------------

.. py:class:: QuasiBinomial(link=logit(), dispersion=1.0)

   Bases: :py:obj:`statsmodels.genmod.families.Binomial`


   Binomial GLM family scaled by a dispersion parameter (quasi-binomial).

   :param link: statsmodels link instance. Default ``logit()``.
   :param dispersion: Multiplicative variance scaling. Default ``1.0``.


   .. py:attribute:: dispersion
      :value: 1.0



   .. py:method:: variance(mu)

      Adjust the variance with the dispersion parameter.



.. py:function:: calculate_p_values(X, y, model)

   Return OLS-style p-values for a fitted model's coefficients.

   :param X: Design matrix (``n x p``).
   :param y: Observed responses.
   :param model: Fitted estimator exposing ``predict`` and ``coef_``.
   :returns: 1D array of length ``p``; entries are ``NaN`` when
       ``n <= p + 1``.


.. py:function:: perform_mixed_model(y, X, groups, alpha=1.0)

   Fit a mixed-effects linear model, falling back to Ridge-adjusted fixed effects on high VIF.

   :param y: Response vector.
   :param X: Fixed-effects design matrix (DataFrame).
   :param groups: Cluster identifiers for random effects.
   :param alpha: Ridge penalty applied when any VIF exceeds 10.
       Default ``1.0``.
   :returns: Fitted ``statsmodels`` ``MixedLMResults``.
   :raises ValueError: if ``groups`` is None.


.. py:function:: create_volcano_filename(csv_path, regression_type, alpha, dst)

   Create and return the volcano plot filename based on regression type and alpha.


.. py:function:: scale_variables(X, y)

   Scale independent (X) and dependent (y) variables using MinMaxScaler.


.. py:function:: select_glm_family(y)

   Select the appropriate GLM family based on the data.


.. py:function:: prepare_formula(dependent_variable, random_row_column_effects=False)

   Return the regression formula using random effects for plate, row, and column.


.. py:function:: fit_mixed_model(df, formula, dst)

   Fit a mixed-effects model with plate/row/column random structure and return coefficients.

   :param df: DataFrame containing the model variables plus
       ``plateID``, ``rowID`` and ``columnID``.
   :param formula: Formula string for fixed effects.
   :param dst: Destination for the residual histogram PDF.
   :returns: ``(mixed_model, coef_df)`` — the fitted results object
       and a DataFrame with columns ``feature``, ``coefficient``,
       ``p_value``.


.. py:function:: check_and_clean_data(df, dependent_variable)

   Check for collinearity, missing values, or invalid types in relevant columns. Clean data accordingly.


.. py:function:: minimum_cell_simulation(settings, num_repeats=10, sample_size=100, tolerance=0.02, smoothing=10, increment=10)

   Plot the mean absolute difference with standard deviation as shaded area vs. sample size.
   Detect and mark the elbow point (inflection) with smoothing and tolerance control.


.. py:function:: process_model_coefficients(model, regression_type, X, y, nc, pc, controls)

   Return DataFrame of model coefficients, standard errors, and p-values.


.. py:function:: check_distribution(y, epsilon=1e-06)

   Check the distribution of y and recommend an appropriate model.


.. py:function:: pick_glm_family_and_link(y)

   Select the appropriate GLM family and link function based on data.


.. py:function:: regression_model(X, y, regression_type='ols', groups=None, alpha=1.0, cov_type=None, weights=None)

   Dispatch to the requested regression backend and return the fitted model.

   Supports OLS, GLM (auto-family), beta, GLM-binomial with logit/probit
   link (weighted by ``weights``), Lasso, Ridge and mixed-effects.
   Alpha is cross-validated when ``'auto'`` or ``None`` is supplied.

   :param X: Design matrix.
   :param y: Response variable.
   :param regression_type: One of ``'ols'``, ``'glm'``, ``'beta'``,
       ``'logit'``, ``'probit'``, ``'lasso'``, ``'ridge'``, ``'mixed'``.
   :param groups: Cluster identifiers for the mixed model.
   :param alpha: Regularisation strength; ``'auto'`` / ``None`` triggers
       internal CV.
   :param cov_type: Optional covariance type for OLS.
   :param weights: Optional per-observation weights (used by
       ``logit``/``probit`` via ``var_weights``).
   :returns: Fitted statsmodels / sklearn estimator.
   :raises ValueError: on unsupported ``regression_type``.


.. py:function:: regression(df, csv_path, dependent_variable='predictions', regression_type=None, alpha=1.0, random_row_column_effects=False, nc='233460', pc='220950', controls=None, dst=None, cov_type=None, plot=False)

   Run the full regression pipeline: clean, fit, extract coefficients, optional volcano plot.

   :param df: Long-format DataFrame with gRNA/gene fractions and the
       dependent variable.
   :param csv_path: Path used to derive the volcano-plot filename.
   :param dependent_variable: Response column name. Default
       ``'predictions'``.
   :param regression_type: Model type; auto-selected via
       :func:`check_distribution` when ``None``.
   :param alpha: Regularisation strength for penalised models.
   :param random_row_column_effects: If True, fit a mixed model with
       random row/column effects.
   :param nc: Negative-control gene identifier. Default ``'233460'``.
   :param pc: Positive-control gene identifier. Default ``'220950'``.
   :param controls: Explicit list of control identifiers.
   :param dst: Output directory for plots and summaries.
   :param cov_type: Optional OLS covariance type.
   :param plot: If True, render the volcano plot after fitting.
   :returns: ``(model, coef_df, regression_type)``.


.. py:function:: save_summary_to_file(model, file_path='summary.csv')

   Save the model's summary output to a CSV or text file.


.. py:function:: perform_regression(settings)

   Top-level regression driver: read score+count data, normalise, regress and plot.

   Merges per-plate score and count tables, normalises plate/well
   identifiers, runs :func:`regression`, joins metadata, and produces
   diagnostic plots (volcano, plate heatmaps, gene phenotype plots,
   GO enrichment).

   :param settings: Regression settings dict. See
       ``settings.get_perform_regression_default_settings`` for keys
       (notably ``score_data``, ``count_data``, ``dependent_variable``,
       ``regression_type``).
   :returns: Whatever the internal regression + plotting pipeline
       yields (typically the merged results DataFrame). See individual
       step outputs for details.


.. py:function:: process_reads(csv_path, fraction_threshold, plate, filter_column=None, filter_value=None)

   Load a per-gRNA read-count CSV and return per-well normalised fractions.

   Splits derived ``plate_row`` or ``prcfo`` identifiers, computes each
   gRNA's fraction of the well total, applies an optional
   fraction-cutoff filter and returns a compact ``(prc, grna, fraction)``
   frame (with ``gene`` derived from the gRNA when possible).

   :param csv_path: Path to the counts CSV, or an already-loaded DataFrame.
   :param fraction_threshold: Drop rows below this fraction; must be in
       ``[0, 1]`` or ``None``.
   :param plate: Plate identifier used when no ``plateID`` column is
       present.
   :param filter_column: Column (or list of columns) to filter rows on.
   :param filter_value: Values (or list of values) to drop from
       ``filter_column``.
   :returns: DataFrame with columns ``prc``, ``grna``, ``fraction``.
   :raises ValueError: on missing required columns, invalid
       ``fraction_threshold``, or when the threshold removes all rows.


.. py:function:: apply_transformation(X, transform)

   Return an sklearn ``FunctionTransformer`` for the named transform.

   :param X: Ignored (kept for compatibility with sklearn pipeline flow).
   :param transform: One of ``'log'``, ``'sqrt'``, ``'square'``. Any
       other value returns ``None``.
   :returns: A ``FunctionTransformer`` or ``None``.


.. py:function:: check_normality(data, variable_name, verbose=False)

   Check if the data is normally distributed using the Shapiro-Wilk test.


.. py:function:: clean_controls(df, values, column)

   Drop rows whose ``column`` is in ``values``.

   :param df: Source DataFrame.
   :param values: Value or list of values to remove.
   :param column: Column to check.
   :returns: Filtered DataFrame (unchanged if ``column`` is missing).


.. py:function:: process_scores(df, dependent_variable, plate, min_cell_count=25, agg_type='mean', transform=None, regression_type='ols', invert_dependent_variable=False)

   Aggregate per-object model scores to per-well summaries, ready for regression.

   Ensures ``plateID/rowID/columnID/prc`` columns exist, applies an
   optional inversion of the raw response, aggregates by well according
   to ``agg_type`` (or by count for Poisson), enforces
   ``min_cell_count`` and optionally transforms the aggregated response.

   :param df: Per-object score DataFrame.
   :param dependent_variable: Column being aggregated.
   :param plate: Plate identifier to stamp when the frame is
       single-plate; ignored (with warning) when multiple plates exist.
   :param min_cell_count: Wells with fewer objects are dropped.
       Default ``25``.
   :param agg_type: ``'mean'``, ``'median'``, ``'quantile'`` or None.
   :param transform: Optional post-aggregation transform name
       (see :func:`apply_transformation`).
   :param regression_type: If ``'poisson'``, aggregation uses ``sum``.
   :param invert_dependent_variable: ``False``/``0`` = no inversion;
       ``True``/``1`` = ``1 - x``; ``-1`` = ``1 / x``.
   :returns: ``(dependent_df, dependent_variable)`` — the per-well
       DataFrame and the (possibly transformed) response column name.
   :raises ValueError: on missing identifiers, unsupported ``agg_type``
       or unrecognised ``invert_dependent_variable``.


.. py:function:: generate_ml_scores(settings)

   Train a classical ML classifier over per-object measurements and score every well.

   Reads measurement DBs across one or more sources, merges tables,
   trains the configured model (via :func:`ml_analysis`) and persists
   per-well and per-object prediction scores back into the source DB.

   :param settings: Settings dict. See
       ``settings.set_default_analyze_screen`` for the accepted keys
       (``src``, ``channel_of_interest``, ``model_type``,
       ``positive_control``, ``negative_control``, ...).
   :returns: Whatever the internal training pipeline returns
       (typically per-plate performance and feature-importance tables).


.. py:function:: ml_analysis(df, channel_of_interest=3, location_column='columnID', positive_control='c2', negative_control='c1', exclude=None, n_repeats=10, top_features=30, reg_alpha=0.1, reg_lambda=1.0, learning_rate=1e-05, n_estimators=1000, test_size=0.2, model_type='xgboost', n_jobs=-1, remove_low_variance_features=True, remove_highly_correlated_features=True, prune_features=False, cross_validation=False, verbose=False)

   Train a per-object classifier on positive/negative control wells and score every row.

   Filters features, splits (or CVs) train/test, fits the requested
   model, computes permutation and native feature importances, tunes an
   optimal decision threshold, and writes predictions and probabilities
   back onto the input DataFrame.

   :param df: Per-object feature DataFrame.
   :param channel_of_interest: Channel index used to select features.
   :param location_column: Column identifying wells / plate columns.
       Default ``'columnID'``.
   :param positive_control: Values in ``location_column`` treated as the
       positive class. Default ``'c2'``.
   :param negative_control: Values treated as the negative class.
       Default ``'c1'``.
   :param exclude: Columns to remove from feature space.
   :param n_repeats: Repeats for permutation importance. Default ``10``.
   :param top_features: Feature cap when ``prune_features=True``.
   :param reg_alpha: XGBoost L1 penalty.
   :param reg_lambda: XGBoost L2 penalty.
   :param learning_rate: XGBoost learning rate.
   :param n_estimators: Tree count for tree-based models.
   :param test_size: Test-split fraction. Default ``0.2``.
   :param model_type: ``'random_forest'``, ``'logistic_regression'``,
       ``'gradient_boosting'`` or ``'xgboost'``.
   :param n_jobs: Parallel job count where applicable. Default ``-1``.
   :param remove_low_variance_features: Drop low-variance features.
   :param remove_highly_correlated_features: Drop highly correlated features.
   :param prune_features: If True, apply ``SelectKBest`` before training.
   :param cross_validation: If True, run 5-fold stratified CV.
   :param verbose: Log progress details.
   :returns: Tuple of results tables and figures — see call sites for
       exact positional structure.
   :raises ValueError: on unsupported ``model_type``.


.. py:function:: shap_analysis(model, X_train, X_test)

   Return a SHAP summary-plot figure for ``model`` explaining ``X_test``.

   :param model: Fitted estimator compatible with ``shap.Explainer``.
   :param X_train: Training features used to seed the explainer.
   :param X_test: Test features to explain.
   :returns: Matplotlib ``Figure`` holding the summary plot.


.. py:function:: find_optimal_threshold(y_true, y_pred_proba)

   Return the probability threshold maximising F1 on the precision-recall curve.

   :param y_true: Ground-truth binary labels.
   :param y_pred_proba: Predicted probabilities for the positive class.
   :returns: Optimal probability threshold.


.. py:function:: interperate_vision_model(settings=None)

   Explain a vision model's predictions using RF, permutation and SHAP importances.

   Merges per-object measurements with predicted scores, then runs any
   combination of feature importance, permutation importance and SHAP
   analyses. Aggregates SHAP into compartment / channel radar plots.

   :param settings: Settings dict — see
       ``settings.set_interperate_vision_model_defaults`` for keys
       (``src``, ``scores``, ``score_column``, ``tables``,
       ``feature_importance``, ``permutation_importance``, ``shap``,
       ``top_features``, ``n_jobs``, ``save``).
   :returns: None (results are plotted and optionally saved to CSV).


