Metadata-Version: 2.4
Name: anomaly_pipeline
Version: 0.1.107
Summary: Ensemble framework for detecting outliers in grouped time-series data
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pandas
Requires-Dist: numpy<2
Requires-Dist: joblib
Requires-Dist: prophet
Requires-Dist: scikit-learn
Requires-Dist: google-cloud-bigquery
Requires-Dist: google-cloud-storage
Requires-Dist: statsmodels>=0.13.0
Requires-Dist: plotly
Requires-Dist: pandas-gbq
Requires-Dist: gcsfs
Requires-Dist: ipython
Requires-Dist: tqdm
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# anomaly-pipeline

anomaly-pipeline is a production-grade ensemble framework for detecting outliers in grouped time-series data. It automates the entire workflow from data cleaning and calendar interpolation to running 8 different detection algorithms and generating visual diagnostic reports.

## Key Capabilities

- Ensemble Scoring: Combines 8 models (Statistical + ML) to provide a robust Anomaly_Score and a final is_Anomaly consensus.
- Hierarchical Processing: Natively handles grouped data (e.g., detecting anomalies per Region, Product, or Channel).
- Automated Preprocessing: Handles missing dates via linear interpolation and filters out "low-quality" unique_ids      automatically.
- Parallel Execution: Leverages joblib for multi-core processing of large datasets.
- Visual Analytics: Generates pie charts, stacked bar plots, and detailed group-level time-series breakdowns.

## Included Models

The pipeline utilizes an ensemble of the following methodologies:

- Statistical: Percentile, Standard Deviation (SD), Median Absolute Deviation (MAD), and Interquartile Range (IQR).

- Time-Series Specific: EWMA (Exponentially Weighted Moving Average) and FB Prophet (Walk-forward validation).

- Machine Learning: Isolation Forest (General & Time-series optimized) and DBSCAN.

## Detailed Functionality

- Decomposition Engine: Uses STL decomposition (Trend + Seasonality) to isolate residuals before modeling.

- Multiplicative Residual Modeling: Statistical models now operate on residual proportions (relative deviation) rather than   absolute units, ensuring scale-invariant detection across high and low-volume series.

- Magnitude-Aware Gating: Integrated thresholding to filter out deviations that meet statistical criteria but lack sufficient business volume.

- Robust Input Validation: Clear error messaging for missing parameters or incorrect data types.

- Quality Control: Automatically generates a Success Report 

- Visual Suite: Automated rendering of Pie Charts (Summary), Stacked Bars (Distribution), and Top-5 Anomaly Heatmaps.

## 🚀 Quick Start

```python
!pip install anomaly-pipeline
import pandas as pd
from anomaly_pipeline import timeseries_anomaly_detection

 # Load your data
df = pd.read_csv("your_data.csv")

 # Run the pipeline
anomaly_df, success_report = timeseries_anomaly_detection( master_data=df,
                                                                             unique_ids=['category', 'region'],
                                                                             variable='sales',
                                                                             date_column='timestamp',
                                                                             freq='W-MON',
                                                                             eval_period=1  # Evaluate the most recent recor
                                                                             )

```
## 📊 Visualizing Results & Deep Dives
Inspecting a Specific Group, if a specific group shows a high anomaly rate, use the evaluation_info tool to render detailed diagnostic plots.

```python
from anomaly_pipeline import evaluation_info

# Filter the specific group you want to inspect. Define the group values (must match the order in unique_ids)
group_values = ['appliances', 'TX'] 

# Filter the results for this group
mask = anomaly_df[unique_ids].eq(group_values).all(axis=1)
group_df = anomaly_df[mask]

# Generate detailed diagnostic plots
evaluation_info(group_df,
                unique_ids,
                variable,
                date_column,
                eval_period=1
                )
```

The Evaluation Dashboard provides:

- Model Breakdown: Individual charts for FB Prophet, EWMA, and Isolation Forest with confidence intervals.

- Ensemble View: A summary highlighting where multiple models overlap.

- Statistical Thresholds: Visual markers for IQR, MAD, percentile and SD limits.

## Input_data: 

### Mandatory

* **master_data** (pd.DataFrame) : Name of your dataframe containing inputs to be evaluated for anomalies
                                   include variables, dates, and unique_ids.
* **unique_ids** (list[str])  : List of columns used to segment data ['SKU', 'channel', "store_id"].
* **variable** (str)             : The numerical target column name to analyze for presence of anomalies.
* **date_column** (str)          : The datetime column representing the time dimension ["date","week","month"].
* **freq** (str)                : Frequency of the date column. Default: 'W-MON'. Accepts 'D', or 'MS'.

### Default
* **eval_period** (int)         : Number of trailing records or periods to evaluate for anomalies. Default: 1.
* **max_records** (int)         : Max history to consider starting from the most recent date. Default: all history
* **imputation_method**         : Technique to fill missing time units. Default: 'linear'. Acceptable values are : 'mean',                                     'mode', 'zero', 'linear'
* **percentile_threshold** (int): Percentile parameter, Identifies top 2 and bottom 2 percentile as anomaly. Default: 0.02.
* **SD_threshold** (int)        : SD parameter, controls the number of deviation from mean. Default: 3.
* **mad_threshold** (int)       : MAD parameter, controls Median Absolute Deviation sensitivity. Default: 5.
* **mad_scale_factor** (int)    : MAD parameter, The scaling constant used to normalize the MAD. Default: 0.6745.
* **IQR_threshold** (int)       : IQR parameter, controls IQR multiplicative term. Default: 3.
* **magnitude_threshold** (int) : statistical model parameter, controls the % deviation from base_signal as a gate for                                         statistical anomaly check. Accepts 0 to 1. Default: 0.
* **alpha** (float)             : EWMA parameter, controls the smoothing factor for EWMA trend. Default: 0.3.
* **sigma** (float)             : EWMA parameter, determines the standard deviation multiplier for upper and lower                                             bounds. Default: 2.
* **prophet_CI** (float)        : Prophet parameter, determines the confidence interval. Range 0 to 1,                                                         Default: 0.95.
* **contamination** (float)     : Isolation Forest parameter, expected % of outliers (0 to 0.5). Default: 0.05. 
* **random_state** (int)        : Seed for model reproducibility. Default: 42.
* **enable_models** (list)      : selected model list. Default : ['PERCENTILE', 'SD', 'MAD', 'IQR', 'EWMA', 'FB', 'ISF','DBSCAN']

## 📤 RETURNS

`tuple` [pd.DataFrame, pd.DataFrame]:

* **final_results**         :   The main output, a dataframe that identifies anomalies with `Anomaly_Votes` and                                               `is_Anomaly`.
* **evaluation_report**     :   Summary of interpolation %, record counts, and anomaly rates.

---

## Output columns of final_results : All the output values are at "unique_ids" level. 

**is_missing_record**
`True` if the original data point was missing and imputed; `False` otherwise.
________________________________________

**Percentile_low / Percentile_high**
Dynamic thresholds used to detect unusually low or high values based on the distribution of the residual proportion. These are calculated in the percentage domain and scaled back to raw units for reporting.
________________________________________
**Percentile_anomaly**
Categorical flag indicating the direction of a percentile-based outlier:
• Low → value < Percentile_low
• High → value > Percentile_high
• None → within the range
________________________________________

**Mean**
The rolling average of the residual proportion (typically over 52 weeks) used to establish the expected baseline of noise. These are calculated in the percentage domain and scaled back to raw units for reporting.
________________________________________

**SD (Standard Deviation)**
The rolling standard deviation of the residual proportion, measuring the volatility of the signal relative to the trend.
________________________________________

**SD_low / SD_high**
Volatility-based thresholds (Mean ± SD_threshold × SD). These boundaries expand and contract based on the historical variability of the residual proportion. These are calculated in the percentage domain and scaled back to raw units for reporting.
__________________________________
**SD_anomaly**
Categorical flag indicating an outlier detected via standard deviation limits:
• Low → value < SD_low
• High → value > SD_high
• None → within the range
________________________________________
**Median / MAD (Median Absolute Deviation)**
Median is the Rolling Median of the residual proportion. Mad is the median of the absolute deviation of residual proprtion at each point from median Deviation.  Median is scaled back to raw values for reporting.
________________________________________
**MAD_low / MAD_high**
Robust statistical thresholds (Median ± mad threhold × MAD/mad_scale_factor) calculated on the residual proportion and projected back into raw units. 
________________________________________
**MAD_anomaly**
Categorical flag indicating a robust statistical outlier::
• Low → value < MAD_low
• High → value > MAD_high
• None → within the range
________________________________________

**Q1 / Q3 / IQR (Interquartile Range)***
• Q1/Q3: 25th and 75th percentiles of the residual proportion.
• IQR = The middle 50% range (Q3 - Q1).
Used to establish boundaries that ignore the extreme tails of the distribution. Implemented on residual proportion over a rolling window of 52 weeks and scaled back to raw values.
________________________________________

**IQR_low / IQR_high**
IQR-based limits:
• IQR_low = Q1 − 3 × IQR (floored at 0)
• IQR_high = Q3 + 3 × IQR 
Implemented on residual proportion and scaled back to raw values.
______________________________________
**IQR_anomaly**
Categorical flag based on IQR fences.
• Low → value < IQR_low
• High → value > IQR_high
• None → within the range
________________________________________
**is_Percentile_anomaly / is_SD_anomaly / is_MAD_anomaly / is_IQR_anomaly**
Boolean indicators (True/False) stating whether the specific statistical method classified the point as an anomaly. Note: These may be suppressed if the Consensus Filter determines volume is below the floor magnitude threshold gates.
________________________________________
**EWMA_forecast**
Boolean indicators (True/False) stating whether the specific statistical method classified the point as an anomaly. Note: These may be suppressed if the Consensus Filter determines only statistical models voted and volume is below the floor.
________________________________________
**EWMA_STD**
The rolling standard deviation of residuals relative to the EWMA forecast, used to calculate dynamic error bands.
________________________________________
**EWMA_high / EWMA_low**
The upper and lower detection bands for the EWMA model (EWMA_Forecast ± Sigma × EWMA_STD).
_____________________________________ 
**Is_EWMA_anomaly**
Boolean flag indicating the observed raw value fell outside the EWMA dynamic bands.
________________________________________
**FB_forecast**
The expected value generated by the FB Prophet model, incorporating trend, seasonality, and holiday effects.
________________________________________
**FB_low / FB_high**
The lower and upper bounds of the Prophet uncertainty interval (Credible Interval)
_____________________________________ 
**FB_residual**
The raw difference between the actual observed value and the FB_forecast.
_____________________________________ 
**FB_anomaly**
Categorical flag (low/high) based on whether the observation exceeded Prophet’s forecast intervals.
_____________________________________ 
**Is_FB_anomaly**
Boolean flag indicating a Prophet-detected anomaly.
______________________________________   
**isolation_forest_score**
Score from the Isolation Forest model indicating anomaly severity. Typical range: –0.5 to +0.5
- Higher/Positive → More normal.
- Lower/Negative → More anomalous (isolated early in the tree).
________________________________________
**is_IsoForest_anomaly**
Boolean flag based on Isolation Forest model output:
- True → model predicts anomaly (prediction = –1)
- False → model predicts normal (prediction = 1)
______________________________________   
**dbscan_score**
The cluster assignment or density score from the DBSCAN algorithm (-1 indicates the point is "noise" or an outlier).
________________________________________
**is_DBSCAN_anomaly**
Boolean flag indicating DBSCAN-detected anomaly.
________________________________________
**Anomaly_Votes**
The count of individual models (out of 8) that flagged the point as an anomaly.
Ranges from 0 to 8.
________________________________________
**Vote_Cnt**
The total number of models enabled and executed for that specific group.
Ranges from 0 to 8.
________________________________________
**is_Anomaly**
Set to True if the majority of models (typically >= 50%) agree. Automatically flipped to False if a Directional Conflict exists or if the Consensus Filter determines the detection is purely statistical noise at a low volume.
________________________________________
**Anomaly_Votes_Display**
A string representation of the ensemble consensus. It displays the number of models that flagged an anomaly against the total number of models enabled (e.g., "3 out of 8").
________________________________________
**is_conflict**
A boolean flag indicating a directional disagreement within the ensemble. It is marked True if at least one model flagged a "high" anomaly while another model simultaneously flagged a "low" anomaly for the same data point.
_______________________________________
**consensus_type**
Indicates the prevailing direction of the anomaly based on the net sum of model votes (High = +1, Low = -1).
- high: More models flagged a positive deviation than a negative one.
- low: More models flagged a negative deviation than a positive one.
- none: No clear directional majority or no anomalies detected.

________________________________________
**max_threshold**
The upper boundary used for data cleansing. If a point is a confirmed "high" anomaly, this value represents the most lenient (highest) upper bound among all voting models; otherwise, it defaults to the raw variable value.

________________________________________
**min_threshold**
The lower boundary used for data cleansing. If a point is a confirmed "low" anomaly, this value represents the most lenient (lowest) lower bound among all voting models; otherwise, it defaults to the raw variable value.

________________________________________
**raw_total_views_cleansed**
The final "corrected" version of the input variable. Anomalous values are clipped to the min_threshold or max_threshold to neutralize the outlier while preserving the underlying trend and seasonality.

________________________________________
**[Model]_score_scaled**
(Applies to Percentile, SD, MAD, IQR, EWMA, FB, IsolationForest, DBSCAN)
The individual anomaly score from each specific model, normalized to a standard 0–100 scale.
- 0: Perfectly normal according to that specific model.
- 100: Maximum anomalousness according to that specific model.
________________________________________
**Anomaly_Score**
The final aggregated ensemble score (0–100). This is typically a weighted average of all enabled [Model]_score_scaled values, representing the overall confidence that a point is anomalous.
________________________________________
**is_Anomaly**
The final ensemble decision based on a majority rule.
- True: Flagged as anomalous by 50% or more of the enabled models (and passing the directional conflict check).
- False: Fewer than 50% of models agree, or the models are in directional conflict.
________________________________________
**anom_score_cat**
A categorical grouping of the Anomaly_Score used for simplified reporting (e.g., "Normal", "Suspect", "High Anomaly").

