Metadata-Version: 2.4
Name: idkmetrics
Version: 0.1.0
Summary: Plain-English explanations for ML metrics.
Author: idkmetrics contributors
License: MIT
Keywords: machine-learning,metrics,explainability,education
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: License :: OSI Approved :: MIT License
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# idkmetrics

For when the model works... but you still want a human sentence.

`idkmetrics` is a tiny Python library that translates ML metrics into plain English for people who do not want to decode a wall of acronyms before coffee.

It is made for:

- new ML folks
- PMs and founders trying to understand model results
- engineers who know the metric name but want the "so what?"
- anyone who has ever stared at `RMSE = 12.7` and thought "cool, I guess?"

## What it does

You pass in one metric or a bundle of metrics.

It gives you:

- what the metric means
- whether higher or lower is better
- a plain-English read on the number
- warnings when the metric is easy to misuse
- optional next-step suggestions

## Install

```bash
pip install -e .
```

## Quick start

```python
from idkmetrics import explain

print(
    explain(
        {"mae": 4.3, "rmse": 6.1, "r2": 0.81},
        task="regression",
        include_next_steps=True,
    )
)
```

Example output:

```text
Metric vibes, one by one:
- Quick read: MAE = 4.3. This is in plain English; This one is mostly useful for relative comparison.
  MAE tells you average absolute error between predictions and the real value. For this run, a lower value of 4.3 looks context-dependent. Its meaning depends a lot on the scale of your target, so compare it with typical real-world error.
  Watch out: MAE uses the same units as the target.
  Next: Compare against a baseline like predicting the mean or last known value.
  Next: Plot prediction error so you can see whether the misses are concentrated on certain ranges.
...
```

## Main API

The main wrapper is `explain(...)`.

```python
explain(
    metrics=None,
    *,
    task=None,
    metric=None,
    value=None,
    audience="beginner",
    tone="casual",
    detail="standard",
    include_scale_tips=True,
    include_warnings=True,
    include_next_steps=False,
    thresholds=None,
    aliases=None,
    strict=False,
    sort_metrics=True,
    return_format="text",
)
```

### Useful parameters

- `task`: hint the model family, like `"regression"`, `"classification"`, `"forecasting"`, `"clustering"`, `"ranking"`, or `"generative"`
- `metric` + `value`: convenient single-metric mode
- `audience`: tune the wording for `"beginner"`, `"product"`, `"data_science"`, or `"exec"`
- `tone`: choose `"casual"`, `"neutral"`, `"hype"`, or `"blunt"`
- `detail`: choose `"short"`, `"standard"`, or `"detailed"`
- `include_scale_tips`: adds reminders when the score depends heavily on target scale
- `include_warnings`: includes caveats like class imbalance traps and baseline gotchas
- `include_next_steps`: adds practical suggestions for what to inspect next
- `thresholds`: override built-in interpretation bands for your use case
- `aliases`: teach the wrapper extra metric spellings
- `strict=True`: raise an error instead of quietly handling unknown metrics
- `return_format`: `"text"`, `"dict"`, or `"list"`

## Supported model types and metrics

This library is intentionally broad enough for the common "what does this even mean?" situations.

### Regression

Good for:

- house price prediction
- demand estimation
- churn value prediction
- any "predict a number" task

Built-in metrics:

- `mae`
- `mse`
- `rmse`
- `r2`
- `medae`

Example:

```python
from idkmetrics import explain

print(explain({"mae": 18.2, "rmse": 31.7, "r2": 0.72}, task="regression"))
```

### Classification

Good for:

- spam detection
- fraud detection
- disease prediction
- sentiment classification

Built-in metrics:

- `accuracy`
- `balanced_accuracy`
- `precision`
- `recall`
- `specificity`
- `f1`
- `roc_auc`
- `pr_auc`
- `log_loss`

Example:

```python
print(
    explain(
        {
            "accuracy": 0.94,
            "precision": 0.71,
            "recall": 0.43,
            "f1": 0.54,
            "roc_auc": 0.91,
        },
        task="classification",
        include_warnings=True,
    )
)
```

That kind of output is useful when a model looks "accurate" overall but still misses a lot of the positive cases.

### Forecasting / Time Series

Good for:

- sales forecasting
- traffic forecasting
- inventory planning
- energy usage prediction

Built-in metrics:

- `mape`
- `smape`
- `wape`
- `mase`

Example:

```python
print(explain({"mape": 0.18, "smape": 0.14, "mase": 0.82}, task="forecasting"))
```

### Clustering

Good for:

- customer segmentation
- behavior grouping
- topic clustering
- anomaly exploration

Built-in metrics:

- `silhouette`
- `davies_bouldin`
- `calinski_harabasz`
- `inertia`

Example:

```python
print(explain({"silhouette": 0.48, "davies_bouldin": 0.91}, task="clustering"))
```

### Ranking / Retrieval

Good for:

- search ranking
- recommendation systems
- retrieval pipelines
- "what should show up first?" systems

Built-in metrics:

- `ndcg`
- `map`
- `mrr`

Example:

```python
print(explain({"ndcg": 0.79, "mrr": 0.68}, task="ranking"))
```

### Generative / NLP-ish overlap metrics

Good for:

- summarization evaluation
- translation experiments
- text generation comparisons
- rough reference-based NLP scoring

Built-in metrics:

- `bleu`
- `rouge`
- `rouge_l`
- `bertscore`
- `perplexity`

Example:

```python
print(explain({"rouge_l": 0.44, "bertscore": 0.92}, task="generative"))
```

## Single metric mode

```python
print(explain(metric="rmse", value=12.6, task="regression"))
print(explain(metric="f1", value=0.87, task="classification", tone="hype"))
print(explain(metric="silhouette", value=0.21, task="clustering", tone="blunt"))
```

## Return structured data instead of text

```python
result = explain(
    {"accuracy": 0.91, "f1": 0.77},
    task="classification",
    return_format="dict",
)

print(result["items"][0]["summary"])
```

This is handy if you want to:

- display explanations inside a web app
- build your own UI on top
- send metric summaries to Slack, Discord, or logs

## Add custom thresholds

Different industries have different definitions of "good".

```python
print(
    explain(
        {"recall": 0.84},
        task="classification",
        thresholds={
            "recall": [
                (0.70, "too many positives are still slipping through"),
                (0.85, "pretty solid catch rate"),
                (0.95, "excellent catch rate"),
            ]
        },
    )
)
```

## Aliases and robustness

The wrapper already normalizes a lot of common naming variants:

- `r_squared`
- `r2_score`
- `mean_absolute_error`
- `roc auc`
- `average_precision`
- `rouge-l`
- and more

You can add your own too:

```python
print(
    explain(
        {"my_team_auc_name": 0.88},
        aliases={"my_team_auc_name": "roc_auc"},
    )
)
```

If you want it to fail loudly on unknown metrics:

```python
explain({"mystery_score": 42}, strict=True)
```

## Why this exists

A lot of metric libraries are technically correct and emotionally unhelpful.

`idkmetrics` is for the moment after training, when someone asks:

- "is 0.71 good?"
- "why is accuracy high but recall bad?"
- "does RMSE being 20 mean the model is broken?"
- "what does silhouette score even look like in human words?"

## Tiny roadmap ideas

- confidence interval and variance-aware explanations
- confusion-matrix-aware narrative summaries
- baseline comparison helpers
- prettier formatting for notebooks and dashboards

## License

MIT
