spacr.sim
=========

.. py:module:: spacr.sim




Module Contents
---------------

.. py:function:: generate_gene_list(number_of_genes, number_of_all_genes)

   Return ``number_of_genes`` randomly-drawn gene indices without replacement.

   :param number_of_genes: Number of gene indices to draw.
   :param number_of_all_genes: Size of the pool ``[0, number_of_all_genes)``.
   :returns: List of drawn gene indices.


.. py:function:: generate_plate_map(nr_plates)

   Return a 384-well plate map DataFrame spanning ``nr_plates`` plates.

   :param nr_plates: Number of plates to enumerate.
   :returns: DataFrame with ``plate_row_column``, ``plate_id``, ``row_id``,
       ``column_id`` columns (16 rows x 24 columns per plate).


.. py:function:: gini_coefficient(x)

   Return the Gini coefficient of ``x`` via the pairwise absolute difference formula.

   :param x: 1-D array-like of non-negative values.
   :returns: Gini coefficient in ``[0, 1]``.


.. py:function:: gini_gene_well(x)

   Return the Gini coefficient of ``x`` using a memory-cheap upper-triangle sum.

   :param x: 1-D array-like of non-negative values.
   :returns: Gini coefficient in ``[0, 1]``; 0 is perfect equality, 1 is perfect inequality.


.. py:function:: gini(x)

   Return the Gini coefficient of ``x`` via the ranked-sum formulation.

   Reference: StatsDirect non-parametric methods
   (http://www.statsdirect.com/help/default.htm#nonparametric_methods/gini.htm).

   :param x: 1-D array-like; all values treated equally.
   :returns: Gini coefficient in ``[0, 1]``.


.. py:function:: dist_gen(mean, sd, df)

   Draw a length-``len(df)`` Poisson sample with gamma-distributed rates.

   :param mean: Mean of the gamma prior on the Poisson rate.
   :param sd: Standard deviation of the gamma prior.
   :param df: DataFrame whose length sets the sample size.
   :returns: Tuple ``(samples, length)`` where ``samples`` is a NumPy array of
       Poisson draws and ``length`` is ``len(df)``.


.. py:function:: generate_gene_weights(positive_mean, positive_variance, df)

   Draw ``len(df)`` gene weights from a Beta distribution matched to the given moments.

   :param positive_mean: Target mean of the Beta distribution in ``(0, 1)``.
   :param positive_variance: Target variance (must be feasible for the mean).
   :param df: DataFrame whose length sets the sample size.
   :returns: NumPy array of Beta-distributed weights.


.. py:function:: normalize_array(arr)

   Return ``arr`` min-max scaled into ``[0, 1]``.

   :param arr: Input NumPy array.
   :returns: Normalized array of the same shape.


.. py:function:: generate_power_law_distribution(num_elements, coeff)

   Return a normalized power-law probability vector of length ``num_elements``.

   :param num_elements: Length of the returned distribution.
   :param coeff: Positive exponent applied as ``i^-coeff``.
   :returns: NumPy array that sums to 1.


.. py:function:: power_law_dist_gen(df, avg, well_ineq_coeff)

   Return ``len(df)`` per-well values sampled from an average-scaled power-law.

   :param df: DataFrame of wells whose length sets the sample size.
   :param avg: Scale factor applied to each drawn probability.
   :param well_ineq_coeff: Power-law exponent (larger = more unequal).
   :returns: NumPy array of per-well quantities.


.. py:function:: run_experiment(plate_map, number_of_genes, active_gene_list, avg_genes_per_well, sd_genes_per_well, avg_cells_per_well, sd_cells_per_well, well_ineq_coeff, gene_ineq_coeff)

   Simulate one cell-level screening experiment and return per-cell + summary tables.

   Draws per-well gene assignments from a power-law distribution and per-well
   cell counts from a gamma/Poisson mixture, then labels each cell active/inactive.

   :param plate_map: DataFrame of wells with plate/row/column identifiers.
   :param number_of_genes: Total number of genes in the pool.
   :param active_gene_list: Gene indices considered active (positive class).
   :param avg_genes_per_well: Mean genes-per-well before power-law scaling.
   :param sd_genes_per_well: Standard deviation of genes-per-well.
   :param avg_cells_per_well: Mean cells-per-well.
   :param sd_cells_per_well: Standard deviation of cells-per-well.
   :param well_ineq_coeff: Power-law exponent for well-level inequality.
   :param gene_ineq_coeff: Power-law exponent for gene-level inequality.
   :returns: Tuple ``(cell_df, genes_per_well_df, wells_per_gene_df, df_ls)``
       where ``df_ls`` contains per-well gene counts, per-gene well counts,
       per-well Gini values, per-gene Gini values, gene weights and well weights.


.. py:function:: classifier(positive_mean, positive_variance, negative_mean, negative_variance, classifier_accuracy, df)

   Assign a Beta-distributed score to each row of ``df`` with class-swap noise.

   :param positive_mean: Mean of the Beta distribution for active cells.
   :param positive_variance: Variance of the Beta for active cells.
   :param negative_mean: Mean of the Beta for inactive cells.
   :param negative_variance: Variance of the Beta for inactive cells.
   :param classifier_accuracy: Probability in ``[0, 1]`` that the correct
       Beta is used for the row's ``is_active`` label.
   :param df: DataFrame containing an ``is_active`` column.
   :returns: The input DataFrame with an added ``score`` column.


.. py:function:: compute_roc_auc(cell_scores)

   Return ROC-curve arrays and AUC for a DataFrame of ``is_active``/``score`` rows.

   :param cell_scores: DataFrame with columns ``is_active`` and ``score``.
   :returns: Dict with keys ``threshold``, ``tpr``, ``fpr``, ``roc_auc``.


.. py:function:: compute_precision_recall(cell_scores)

   Return precision/recall/F1/PR-AUC arrays for a DataFrame of ``is_active``/``score`` rows.

   :param cell_scores: DataFrame with columns ``is_active`` and ``score``.
   :returns: Dict with keys ``threshold``, ``precision``, ``recall``,
       ``f1_score``, ``pr_auc``.


.. py:function:: get_optimum_threshold(cell_pr_dict)

   Return the classification threshold that maximises F1 in a PR result dict.

   :param cell_pr_dict: Dict as returned by :func:`compute_precision_recall`.
   :returns: Threshold value that maximises the F1 score.


.. py:function:: update_scores_and_get_cm(cell_scores, optimum)

   Add a per-threshold predicted-label column and return the confusion matrix.

   :param cell_scores: DataFrame with columns ``is_active`` and ``score``.
   :param optimum: Score threshold used to binarise predictions.
   :returns: Tuple ``(cell_scores, cell_cm)`` where ``cell_cm`` is a NumPy
       confusion matrix.


.. py:function:: cell_level_roc_auc(cell_scores)

   Compute cell-level ROC/PR metrics and confusion matrix at the F1-optimal threshold.

   :param cell_scores: DataFrame with columns ``is_active`` and ``score``.
   :returns: Tuple ``(cell_roc_dict_df, cell_pr_dict_df, cell_scores, cell_cm)``.


.. py:function:: generate_well_score(cell_scores)

   Aggregate cell-level scores into per-well summary rows.

   :param cell_scores: DataFrame indexed by cells with ``plate_row_column``,
       ``is_active`` and ``gene_id`` columns.
   :returns: DataFrame indexed by ``plate_row_column`` with
       ``average_active_score``, ``gene_list``, and ``score`` columns.


.. py:function:: sequence_plates(well_score, number_of_genes, avg_reads_per_gene, sd_reads_per_gene, sequencing_error=0.01)

   Simulate sequencing of every well and return per-well gene fraction and metadata.

   Each gene present in a well accrues a Poisson-distributed read count that
   may be reassigned to a random well with probability ``sequencing_error``.

   :param well_score: DataFrame with a ``gene_list`` column per well.
   :param number_of_genes: Number of distinct genes in the pool.
   :param avg_reads_per_gene: Mean of the per-gene read count distribution.
   :param sd_reads_per_gene: Standard deviation of that distribution.
   :param sequencing_error: Probability of assigning a read to the wrong well.
       Default ``0.01``.
   :returns: Tuple ``(gene_fraction_map, metadata)`` DataFrames indexed by well.


.. py:function:: regression_roc_auc(results_df, active_gene_list, control_gene_list, alpha=0.05, optimal=False)

   Score regression hits against ground truth and compute ROC/PR metrics.

   Marks each gene as active/inactive/control, derives a hit cutoff from the
   control coefficients, and returns ROC/PR curves, a confusion matrix and
   per-run summary statistics.

   :param results_df: Regression output with ``gene``, ``coef`` and ``P>|t|``.
   :param active_gene_list: Gene indices considered truly active.
   :param control_gene_list: Gene indices used to derive the coefficient cutoff.
   :param alpha: Significance threshold applied to p-values. Default ``0.05``.
   :param optimal: When True, use the F1-optimal probability threshold instead
       of ``0.5`` for the final confusion matrix.
   :returns: Tuple ``(results_df, reg_roc_dict_df, reg_pr_dict_df, reg_cm,
       sim_stats)`` where ``sim_stats`` is a single-row DataFrame.


.. py:function:: plot_histogram(data, x_label, ax, color, title, binwidth=0.01, log=False)

   Draw a Seaborn density histogram on ``ax`` for the given column.

   :param data: Data source passed to ``sns.histplot``.
   :param x_label: Column name plotted on the x-axis.
   :param ax: Matplotlib axes to draw into.
   :param color: Bar/fill color.
   :param title: Axes title.
   :param binwidth: Histogram bin width; falsy uses Seaborn's default.
   :param log: When True, apply a log scale to the y-axis.
   :returns: None.


.. py:function:: plot_roc_pr(data, ax, title, x_label, y_label)

   Plot a ROC or PR curve with a diagonal random-classifier reference line.

   :param data: DataFrame containing the ``x_label`` and ``y_label`` columns.
   :param ax: Matplotlib axes to draw into.
   :param title: Axes title.
   :param x_label: Column name plotted on the x-axis.
   :param y_label: Column name plotted on the y-axis.
   :returns: None.


.. py:function:: plot_confusion_matrix(data, ax, title)

   Render a 2x2 confusion matrix as an annotated Seaborn heatmap.

   :param data: 2x2 NumPy confusion matrix ordered ``[[TN, FP], [FN, TP]]``.
   :param ax: Matplotlib axes to draw into.
   :param title: Axes title.
   :returns: None.


.. py:function:: run_simulation(settings)

   Run one end-to-end pooled-screen simulation and return every intermediate table.

   Composes :func:`generate_gene_list`, :func:`generate_plate_map`,
   :func:`run_experiment`, :func:`classifier`, cell/well aggregation,
   :func:`sequence_plates` and :func:`regression_roc_auc` into a single call.

   :param settings: Dict of simulation parameters (gene counts, distribution
       moments, sequencing error, classifier accuracy, ...).
   :returns: Tuple ``(cell_scores, cell_roc_dict_df, cell_pr_dict_df,
       cell_cm, well_score, gene_fraction_map, metadata, results_df,
       reg_roc_dict_df, reg_pr_dict_df, reg_cm, sim_stats,
       genes_per_well_df, wells_per_gene_df, dists)``.


.. py:function:: vis_dists(dists, src, v, i)

   Save side-by-side histograms of the six per-run distributions in ``dists``.

   :param dists: Six arrays in order ``[genes/well, wells/gene, gini_well,
       gini_gene, gene_weights, well_weights]``.
   :param src: Output directory used by :func:`save_plot`.
   :param v: Variable label passed through to :func:`save_plot`.
   :param i: Simulation index passed through to :func:`save_plot`.
   :returns: None.


.. py:function:: visualize_all(output)

   Render the full 13-panel diagnostic figure for one simulation output.

   :param output: The 14-element list returned by :func:`run_simulation` (all
       elements before ``dists``).
   :returns: The generated Matplotlib figure.


.. py:function:: create_database(db_path)

   Ensure a SQLite database file exists at ``db_path``.

   :param db_path: Filesystem path for the SQLite database.
   :returns: None.


.. py:function:: append_database(src, table, table_name)

   Append a DataFrame to ``<src>/simulations.db`` under ``table_name``.

   :param src: Directory containing (or that should contain) ``simulations.db``.
   :param table: DataFrame written with ``if_exists='append'``.
   :param table_name: Target table name in the SQLite database.
   :returns: None.


.. py:function:: save_data(src, output, settings, save_all=False, i=0, variable='all')

   Persist one simulation's output tables to a SQLite database under ``src``.

   In the default mode only a concatenated summary row (settings + sim_stats
   + Gini metrics) is appended to a ``simulations`` table. When ``save_all``
   is True, every intermediate table is written under its canonical name.

   :param src: Output directory containing ``simulations.db``.
   :param output: 14-element list from :func:`run_simulation`.
   :param settings: Simulation settings dict recorded as the first row.
   :param save_all: When True, write every intermediate table separately.
   :param i: Simulation index used to tag the summary row.
   :param variable: Name of the swept variable used for tagging.
   :returns: None.


.. py:function:: save_plot(fig, src, variable, i)

   Save a Matplotlib figure to ``<src>/<variable>/<i>_figure.pdf``.

   :param fig: Figure to save.
   :param src: Root directory for outputs.
   :param variable: Sub-folder name (the swept variable label).
   :param i: Zero-padded simulation index used in the file name.
   :returns: None.


.. py:function:: run_and_save(i, settings, time_ls, total_sims)

   Worker that runs one simulation, saves outputs, and appends its runtime.

   :param i: Simulation index (used for filenames and tagging).
   :param settings: Simulation settings dict.
   :param time_ls: Shared list receiving the elapsed time in seconds.
   :param total_sims: Total simulation count (used for progress display only).
   :returns: Tuple ``(i, sim_time, None)``.


.. py:function:: validate_and_adjust_beta_params(sim_params)

   Clamp per-run Beta variances so the requested mean/variance is feasible.

   :param sim_params: List of per-run parameter dicts with ``positive_mean``,
       ``negative_mean``, ``positive_variance``, ``negative_variance``.
   :returns: The same list with any infeasible variances capped to 99% of the
       theoretical maximum for the requested mean.


.. py:function:: generate_paramiters(settings)

   Expand a sweep-settings dict into one settings dict per (Cartesian) simulation.

   :param settings: Config dict where each swept key holds an iterable of values.
   :returns: Shuffled list of per-run settings dicts, already run through
       :func:`validate_and_adjust_beta_params`.


.. py:function:: run_multiple_simulations(settings)

   Fan out the sweep from :func:`generate_paramiters` across a process pool.

   Uses a ``multiprocessing.Pool`` with ``max_workers`` (or ``cpu_count()-4``)
   workers, prints a progress line, and drives each worker through
   :func:`run_and_save`.

   :param settings: Sweep-settings dict. Must include ``max_workers``.
   :returns: None.


.. py:function:: generate_integers(start, stop, step)

   Return ``list(range(start, stop + 1, step))`` (inclusive upper bound).


.. py:function:: generate_floats(start, stop, step)

   Return an inclusive list of floats from ``start`` to ``stop`` with ``step`` spacing.


.. py:function:: remove_columns_with_single_value(df)

   Return ``df`` without columns whose values are constant across rows.

   :param df: Source DataFrame.
   :returns: Copy of ``df`` with zero-variance columns dropped.


.. py:function:: read_simulations_table(db_path)

   Return the ``simulations`` table from ``db_path`` as a DataFrame.

   :param db_path: Path to a SQLite database written by :func:`save_data`.
   :returns: DataFrame of the ``simulations`` table, or ``None`` on failure.


.. py:function:: plot_simulations(df, variable, x_rotation=None, legend=False, grid=False, clean=True, verbose=False)

   Grid-plot PR-AUC vs ``variable`` for every unique combination of the other sweep dimensions.

   :param df: DataFrame containing ``prauc``, ``variable`` and the standard
       grouping columns (``number_of_active_genes``, ``avg_reads_per_gene``, ...).
   :param variable: Column plotted on the x-axis of each subplot.
   :param x_rotation: Degrees to rotate x-tick labels. ``None`` uses 45.
   :param legend: When True, show the per-subplot legend.
   :param grid: When True, draw grid lines.
   :param clean: When True, drop grouping columns whose values never vary.
   :param verbose: When True, annotate each subplot with its filter conditions.
   :returns: The generated Matplotlib figure.


.. py:function:: plot_correlation_matrix(df, annot=False, cmap='inferno', clean=True)

   Render a lower-triangular correlation heatmap of the standard sweep + metric columns.

   :param df: DataFrame containing sweep variables plus ``prauc``, ``roc_auc``
       and related outputs.
   :param annot: When True, write numeric correlations in each cell.
   :param cmap: Colormap name or object (overridden internally to a diverging
       palette).
   :param clean: When True, drop constant columns before computing correlations.
   :returns: The generated Matplotlib figure.


.. py:function:: plot_feature_importance(df, target='prauc', exclude=None, clean=True)

   Train a RandomForestRegressor on sweep variables and plot the resulting importances.

   :param df: DataFrame with sweep columns and ``target``.
   :param target: Column predicted by the regressor. Default ``'prauc'``.
   :param exclude: Column name or list of columns to remove from the feature set.
   :param clean: When True, drop constant columns before fitting.
   :returns: The generated Matplotlib figure.


.. py:function:: calculate_permutation_importance(df, target='prauc', exclude=None, n_repeats=10, clean=True)

   Fit a RandomForest and plot permutation-based feature importances for the sweep columns.

   :param df: DataFrame with sweep columns and ``target``.
   :param target: Column predicted by the regressor. Default ``'prauc'``.
   :param exclude: Column name or list of columns to remove from the feature set.
   :param n_repeats: Number of permutations per feature. Default ``10``.
   :param clean: When True, drop constant columns before fitting.
   :returns: The generated Matplotlib figure.


.. py:function:: plot_partial_dependences(df, target='prauc', clean=True)

   Fit a GradientBoostingRegressor and plot partial dependences for every sweep feature.

   :param df: DataFrame with sweep columns and ``target``.
   :param target: Column predicted by the regressor. Default ``'prauc'``.
   :param clean: When True, drop constant columns before fitting.
   :returns: The generated Matplotlib figure.


.. py:function:: save_shap_plot(fig, src, variable, i)

   Save a SHAP figure to ``<src>/<variable>/<i>_figure.pdf``.


.. py:function:: generate_shap_summary_plot(df, target='prauc', clean=True)

   Fit a RandomForest and render a SHAP summary plot over the standard sweep features.

   :param df: DataFrame with sweep columns and ``target``.
   :param target: Column predicted by the regressor. Default ``'prauc'``.
   :param clean: When True, drop constant columns before fitting.
   :returns: The current Matplotlib figure (SHAP creates it as a side effect).


.. py:function:: remove_constant_columns(df)

   Return ``df`` limited to columns that contain more than one unique value.

   :param df: Source DataFrame.
   :returns: Copy of ``df`` with constant columns dropped.


