ise.evaluation package

Submodules

ise.evaluation.metrics module

ise.evaluation.metrics.calculate_ece(predictions, uncertainties, true_values, bins=10)

Calculate the Expected Calibration Error (ECE) for regression model predictions.

Args: predictions (numpy.ndarray): Array of predicted means by the model. uncertainties (numpy.ndarray): Array of predicted standard deviations (uncertainty estimates). true_values (numpy.ndarray): Array of actual values. bins (int): Number of bins to use for grouping predictions by their uncertainty.

Returns: float: The Expected Calibration Error.

ise.evaluation.metrics.crps(y_true, y_pred, y_std)

ise.evaluation.metrics.js_divergence(p: ndarray, q: ndarray)

Calculates the Jensen-Shannon Divergence between two distributions.

ise.evaluation.metrics.kl_divergence(p: ndarray, q: ndarray)

Calculates the Kullback-Leibler Divergence between two distributions.

ise.evaluation.metrics.kolmogorov_smirnov(x1, x2)

ise.evaluation.metrics.mape(y_true, y_pred)

Calculate Mean Absolute Percentage Error (MAPE).

Args: - y_true: numpy array or a list of actual numbers - y_pred: numpy array or a list of predicted numbers

Returns: - mape: Mean Absolute Percentage Error

ise.evaluation.metrics.mean_absolute_error(y_true, y_pred)

Calculate Mean Absolute Error (MAE).

Args: - y_true: numpy array or a list of actual numbers - y_pred: numpy array or a list of predicted numbers

Returns: - mae: Mean Absolute Error

ise.evaluation.metrics.mean_squared_error(y_true, y_pred)

Calculate Mean Squared Error (MSE).

Args: - y_true: numpy array or a list of actual numbers - y_pred: numpy array or a list of predicted numbers

Returns: - mse: Mean Squared Error

ise.evaluation.metrics.mean_squared_error_sector(sum_sectors_true, sum_sectors_pred)

ise.evaluation.metrics.r2_score(y_true, y_pred)

ise.evaluation.metrics.relative_squared_error(y_true, y_pred)

Calculate Relative Squared Error (RSE).

Args: - y_true: numpy array or a list of actual numbers - y_pred: numpy array or a list of predicted numbers

Returns: - rse: Relative Squared Error

ise.evaluation.metrics.sum_by_sector(array, grid_file)

ise.evaluation.metrics.t_test(x1, x2)

ise.evaluation.plots module

class ise.evaluation.plots.EvaluationPlotter(save_dir='.')

Bases: object

sector_side_by_side(y_true, y_pred, grid_file, outline_array_true=None, outline_array_pred=None, timestep=None, save_path=None, cmap=<matplotlib.colors.LinearSegmentedColormap object>)

spatial_side_by_side(y_true, y_pred, timestep=None, save_path=None, cmap=<matplotlib.colors.LinearSegmentedColormap object>, video=False)

class ise.evaluation.plots.SectorPlotter(results_dataset, column=None, condition=None, save_directory=None)

Bases: object

plot_callibration(color_by=None, alpha=0.2, column=None, condition=None, save=None)

plot_distributions(year, column=None, condition=None, save=None)

plot_ensemble(uncertainty='quantiles', column=None, condition=None, save=None)

plot_ensemble_mean(uncertainty=False, column=None, condition=None, save=None)

plot_histograms(year, column=None, condition=None, save=None)

plot_test_series(model, data_directory, time_series=True, approx_dist=True, mc_iterations=100, confidence='95', draws='random', k=10, save_directory=None)

class ise.evaluation.plots.UncertaintyBounds(data, confidence='95', quantiles=None)

Bases: object

ise.evaluation.plots.plot_callibration(dataset, column=None, condition=None, color_by=None, alpha=0.2, save=None)

ise.evaluation.plots.plot_distributions(dataset: DataFrame, year: int = 2100, column: str | None = None, condition: str | None = None, save: str | None = None, cache: dict | None = None)

Generates a plot of comparison of distributions at a given time slice (year) from the true simulations and the predicted emulation.

Parameters:

• dataset (pd.DataFrame) - testing results dataframe, result from [ise.utils.data.combine_testing_results](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#combine_testing_results).

• year (int, optional) - Distribution year (time slice). Defaults to 2100.

• column (str, optional) - Column to subset on, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Defaults to None.

• condition (str, optional) - Condition to subset with, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Can be int, str, float, etc. Defaults to None.

• save (str, optional) - Path to save plot. Defaults to None.

• cache (dict, optional) - Cached results from previous calculation, used internally in [ise.visualization.Plotter](https://brown-sciml.github.io/ise/ise/sectors/visualization/Plotter.html#Plotter). Defaults to None.

ise.evaluation.plots.plot_ensemble(dataset: DataFrame, uncertainty: str = 'quantiles', column: str | None = None, condition: str | None = None, save: str | None = None, cache: dict | None = None)

Generates a plot of the comparison of ensemble results from the true simulations and the predicted emulation. Adds uncertainty bounds and plots them side-by-side.

Parameters:

• dataset (pd.DataFrame) - testing results dataframe, result from [ise.utils.data.combine_testing_results](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#combine_testing_results).

• uncertainty (str, optional) - Type of uncertainty for creating bounds, must be in [quantiles, confidence]. Defaults to ‘quantiles’.

• column (str, optional) - Column to subset on, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Defaults to None.

• condition (str, optional) - Condition to subset with, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Can be int, str, float, etc. Defaults to None.

• save (str, optional) - Path to save plot. Defaults to None.

• cache (dict, optional) - Cached results from previous calculation, used internally in [ise.visualization.Plotter](https://brown-sciml.github.io/ise/ise/sectors/visualization/Plotter.html#Plotter). Defaults to None.

ise.evaluation.plots.plot_ensemble_mean(dataset: DataFrame, uncertainty: str = False, column=None, condition=None, save=None, cache=None)

Generates a plot of the mean sea level contribution from the true simulations and the predicted emulation.

Parameters:

• dataset (pd.DataFrame) - testing results dataframe, result from [ise.utils.data.combine_testing_results](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#combine_testing_results).

• uncertainty (str, optional) - Type of uncertainty for creating bounds. If not None/False, must be in [quantiles, confidence]. Defaults to ‘quantiles’.

• column (str, optional) - Column to subset on, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Defaults to None.

• condition (str, optional) - Condition to subset with, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Can be int, str, float, etc. Defaults to None.

• save (str, optional) - Path to save plot. Defaults to None.

• cache (dict, optional) - Cached results from previous calculation, used internally in [ise.visualization.Plotter](https://brown-sciml.github.io/ise/ise/sectors/visualization/Plotter.html#Plotter). Defaults to None.

ise.evaluation.plots.plot_histograms(dataset: DataFrame, year: int = 2100, column: str | None = None, condition: str | None = None, save: str | None = None, cache: dict | None = None)

Generates a plot of comparison of histograms at a given time slice (year) from the true simulations and the predicted emulation.

Parameters:

• dataset (pd.DataFrame) - testing results dataframe, result from [ise.utils.data.combine_testing_results](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#combine_testing_results).

• year (int, optional) - Histogram year (time slice). Defaults to 2100.

• column (str, optional) - Column to subset on, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Defaults to None.

• condition (str, optional) - Condition to subset with, used in [ise.utils.data.group_by_run](https://brown-sciml.github.io/ise/ise/sectors/utils/data.html#group_by_run). Can be int, str, float, etc. Defaults to None.

• save (str, optional) - Path to save plot. Defaults to None.

• cache (dict, optional) - Cached results from previous calculation, used internally in [ise.visualization.Plotter](https://brown-sciml.github.io/ise/ise/sectors/visualization/Plotter.html#Plotter). Defaults to None.

ise.evaluation.plots.plot_test_series(model, data_directory, time_series, approx_dist=True, mc_iterations=100, confidence='95', draws='random', k=10, save_directory=None)

Module contents