Metadata-Version: 2.4
Name: traffic_anomaly
Version: 1.0.5
Summary: Robust decomposition and anomaly detection on multiple time series for any SQL backend. Designed for traffic data.
Author-email: Shawn Strasser <shawn.strasser@odot.oregon.gov>
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ibis-framework[duckdb]<11.0.0,>=9.0.0
Dynamic: license-file

# Traffic Anomaly

<!-- Package Info -->
[![PyPI](https://img.shields.io/pypi/v/traffic-anomaly)](https://pypi.org/project/traffic-anomaly/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/traffic-anomaly)](https://pypi.org/project/traffic-anomaly/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/traffic-anomaly)](https://pypi.org/project/traffic-anomaly/)

<!-- Repository Info -->
[![GitHub License](https://img.shields.io/github/license/ShawnStrasser/traffic-anomaly)](https://github.com/ShawnStrasser/traffic-anomaly/blob/main/LICENSE)
[![GitHub issues](https://img.shields.io/github/issues/ShawnStrasser/traffic-anomaly)](https://github.com/ShawnStrasser/traffic-anomaly/issues)
[![GitHub stars](https://img.shields.io/github/stars/ShawnStrasser/traffic-anomaly)](https://github.com/ShawnStrasser/traffic-anomaly/stargazers)

<!-- Status -->
[![Unit Tests](https://github.com/ShawnStrasser/traffic-anomaly/actions/workflows/pr-tests.yml/badge.svg)](https://github.com/ShawnStrasser/traffic-anomaly/actions/workflows/pr-tests.yml)
[![codecov](https://codecov.io/gh/ShawnStrasser/traffic-anomaly/branch/main/graph/badge.svg)](https://codecov.io/gh/ShawnStrasser/traffic-anomaly)

`traffic-anomaly` is a production ready Python package for robust decomposition and anomaly detection on multiple time series at once. It uses Ibis to integrate with any SQL backend in a production pipeline, or run locally with the included DuckDB backend.

Designed for real world messy traffic data (volumes, travel times), `traffic-anomaly` uses medians to decompose time series into trend, daily, weekly, and residual components. Anomalies are then classified, and Median Absolute Deviation may be used for further robustness. Missing data are handled, and time periods without sufficient data can be thrown out. Try it out, sample data included! [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1abv0GaEiapu6FFiKEsFI6NFxtX8kgTXb?usp=sharing)



# Installation & Usage

```bash
pip install traffic-anomaly
```
    
```python
import traffic_anomaly
from traffic_anomaly import sample_data

decomp = traffic_anomaly.median_decompose(
    data=travel_times, # Pandas DataFrame or Ibis Table (for compatibility with any SQL backend)
    datetime_column='timestamp',
    value_column='travel_time',
    entity_grouping_columns=['id', 'group'],
    freq_minutes=60, # Frequency of the time series in minutes
    rolling_window_days=7, # Rolling window size in days. Should be a multiple of 7 for traffic data
    drop_days=7, # Should be at least 7 for traffic data
    min_rolling_window_samples=56, # Minimum number of samples in the rolling window, set to 0 to disable.
    min_time_of_day_samples=7, # Minimum number of samples for each time of day (like 2:00pm), set to 0 to disable
    drop_extras=False, # lets keep seasonal/trend for visualization below
    to_sql=False # Return SQL queries instead of Pandas DataFrames for running on SQL backends
)
decomp.head(3)
```
| id         | timestamp           | travel_time | group           | median    | season_day | season_week | resid      | prediction |
|------------|---------------------|-------------|-----------------|-----------|------------|-------------|------------|------------|
| 448838574  | 2022-09-29 06:00:00 | 24.8850     | SE SUNNYSIDE RD | 24.963749 | -4.209375  | 0.57875     | 3.5518772  | 21.333122  |
| 448838574  | 2022-09-22 06:00:00 | 20.1600     | SE SUNNYSIDE RD | 24.842501 | -4.209375  | 0.57875     | -1.0518752 | 21.211876  |
| 448838574  | 2022-09-15 06:00:00 | 22.2925     | SE SUNNYSIDE RD | 24.871250 | -4.209375  | 0.57875     | 1.0518752  | 21.240623  |

```python
# Apply anomaly detection
anomaly = traffic_anomaly.find_anomaly(
    decomposed_data=decomp, # Decomposed time series as a Pandas DataFrame or Ibis Table
    datetime_column='timestamp',
    value_column='travel_time',
    entity_grouping_columns=['id'],
    entity_threshold=3.5 # Threshold for entity-level anomaly detection (z-score or GEH statistic)
)
anomaly.head(3)
```
| id         | timestamp           | travel_time | group          | prediction | anomaly |
|------------|----------------------|-------------|----------------|------------|---------|
| 448838575  | 2022-09-09 06:00:00  | 19.3575     | SE SUNNYSIDE RD| 16.926249  | False   |
| 448838575  | 2022-09-09 07:00:00  | 22.5200     | SE SUNNYSIDE RD| 20.826252  | False   |
| 448838575  | 2022-09-09 08:00:00  | 23.0350     | SE SUNNYSIDE RD| 22.712502  | False   |


<br>

The image below is showing an example application on actual traffic counts. Note that this package does not produce plots.

![ExampleAnomaly](anomaly1.png)

Here's a plot showing what it looks like to decompose a time series. The sum of compoenents is equal to the original data. After extracting the trend and seasonal components, what is left are residuals that are more stationary so they're easier to work with.
![Example](example_plot.png)
# Considerations

The seasonal components are not allowed to change over time, therefore, it is important to limit the number of weeks included in the model, especially if there is yearly seasonality (and there is). The recommended use for application over a long date range is to run the model incrementally over a rolling window of about 6 weeks.

Because traffic data anomalies usually skew higher, forecasts made by this model are systemically low because in a right tailed distribution the median will be lower than the mean. This is by design, as the model is meant primarily for anomaly detection and not forecasting.

# Notes On Anomaly Detection

`traffic_anomaly` can classify two separate types of anomalies:

1. Entity-Level Anomalies are detected for individual entities based on their own historical patterns, without considering the group context.
2. Group-Level Anomalies are detected for entities when compared to the behavior of other entities within the same group. Group-level anomalies are more rare because in order to be considered for classification as a group-level anomaly, a time period must also have been classified as an entity-level anomaly.

Why is that needed? Well, say you're data is vehicle travel times within a city and there is a snow storm. Travel times across the city drop, and if you're looking at roadway segments in isolation, everything is an anomaly. That's nice, but what if you're only interested in things that are broken? That's where group-level anomalies come in. They are more rare, but they are more likely to be actionable. Probably not much you can do about that snow storm...

# Future Plans/Support
It would be nice to add support for Holidays and a yearly component... please help?

### Change Point Detection
I have working code from the `ruptures` package but it's not integrated here yet, and it's slower than molasses. I'll get to it eventually.
