Metadata-Version: 2.4
Name: inferyx-monitoring
Version: 1.0.56
Summary: Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI
Author-email: Inferyx DevOps <devops@inferyx.com>
License: Copyright (c) 2026 Inferyx. All rights reserved.
        
        Proprietary software. Unauthorized copying, distribution, or use is prohibited
        unless agreed in writing with Inferyx.
        
        For open-source release, replace this file with your chosen license (e.g. MIT, Apache-2.0)
        before publishing to public PyPI.
        
Keywords: batch,monitoring,pipeline,email,inferyx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: python-dateutil>=2.8.2
Requires-Dist: fastapi>=0.110.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: uvicorn[standard]>=0.27.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: lxml>=5.0.0
Requires-Dist: signxml>=3.2.0
Provides-Extra: admin
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: check-manifest; extra == "dev"
Dynamic: license-file

# inferyx-monitoring

Batch pipeline monitor with email alerts and Admin UI.

**Install directory:** `/opt/pipeline-monitor`  
**Admin URL:** `https://<your-host>/monitoring/admin/` (nginx required)  
**PyPI:** [inferyx-monitoring](https://pypi.org/project/inferyx-monitoring/)

---

## Quick reference

| Task | Command |
|------|---------|
| **Fresh install** | See [Install from scratch](#install-from-scratch) |
| **Upgrade** | `pip install --upgrade` → `release.sh upgrade --skip-pip` |
| **Uninstall** | `release.sh uninstall --skip-pip` |
| **Remove everything** | `release.sh uninstall --purge --skip-pip` |
| **Service status** | `sudo systemctl status inferyx-monitoring` |
| **Logs** | `sudo journalctl -u inferyx-monitoring -n 50` |

Maintainer build docs: [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) (not for server operators).

---

## Prerequisites

| Requirement | Notes |
|-------------|-------|
| **OS** | Ubuntu 22.04+ (or Debian with `apt`) |
| **Python** | 3.10+ (`python3`, `python3-venv`) |
| **User** | `inferyx` system user (created by install script) |
| **nginx** | Reverse proxy for `/monitoring/admin/` and `/monitoring/api/` |
| **Network** | SMTP, Inferyx API, Google OAuth and/or AWS IDC SAML |

---

## Install from scratch

### Step 1 — Install Python package

Replace `VERSION` with the target release (e.g. `1.0.51`):

```bash
export VERSION=1.0.51
export IM_HOME=/opt/pipeline-monitor

sudo mkdir -p "$IM_HOME"
sudo useradd --system --home-dir "$IM_HOME" --shell /usr/sbin/nologin inferyx 2>/dev/null || true

sudo -u inferyx python3 -m venv "$IM_HOME/.venv"
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade pip
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install "inferyx-monitoring==${VERSION}"
```

**Or from a local wheel:**

```bash
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install /path/to/inferyx_monitoring-${VERSION}-py3-none-any.whl
```

### Step 2 — Run install script

```bash
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" install --skip-pip
```

This creates:

| Path | Purpose |
|------|---------|
| `/opt/pipeline-monitor/.env` | SMTP, API, alert settings |
| `/opt/pipeline-monitor/auth.policy` | Admin UI, OAuth, paths |
| `/opt/pipeline-monitor/batch_file.csv` | Batch schedules |
| `/opt/pipeline-monitor/secrets/` | session_secret, OAuth secrets |
| `/opt/pipeline-monitor/www/` | Admin UI static files |
| `/opt/pipeline-monitor/logs/` | Monitor log |
| `inferyx-monitoring.service` | systemd unit |

### Step 3 — Configure

```bash
sudo nano /opt/pipeline-monitor/.env
sudo nano /opt/pipeline-monitor/auth.policy
sudo nano /opt/pipeline-monitor/batch_file.csv
```

**Secrets** (never commit):

```bash
sudo mkdir -p /opt/pipeline-monitor/secrets
sudo openssl rand -hex 32 | sudo tee /opt/pipeline-monitor/secrets/session_secret
sudo chmod 600 /opt/pipeline-monitor/secrets/*
sudo chown -R inferyx:inferyx /opt/pipeline-monitor
```

**`.env` templates:** `inferyx_pipeline_monitor/data/.env.example` (in package: `.venv/.../data/`)

**`auth.policy`:** set `ui.public_base_url`, login providers, and paths under `/opt/pipeline-monitor`.

### AWS IAM Identity Center (SAML 2.0)

Choose **SAML 2.0** (not OAuth) when creating the AWS IDC application.

| AWS console field | `auth.policy` key | Example |
|-------------------|-------------------|---------|
| Application ACS URL | `acs_url` | `https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc` |
| Application SAML audience | `entity_id` | `https://devops.inferyx.com/monitoring` |
| IAM Identity Center SAML issuer URL | `idp_entity_id` | from IDC metadata |
| IAM Identity Center sign-in URL | `idp_sso_url` | from IDC metadata |
| IDC certificate | `idp_certificate_path` | `/opt/pipeline-monitor/secrets/aws_idc_idp.crt` |

```json
"aws_idc": {
  "enabled": true,
  "entity_id": "https://devops.inferyx.com/monitoring",
  "acs_url": "https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc",
  "idp_entity_id": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_sso_url": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_certificate_path": "/opt/pipeline-monitor/secrets/aws_idc_idp.crt",
  "allowed_domains": ["inferyx.com"]
}
```

SP metadata (optional download for AWS setup):

```text
https://devops.inferyx.com/monitoring/api/auth/saml/aws_idc/metadata
```

Google Workspace stays on **OAuth 2.0** (`auth.google`).

---

### Step 4 — nginx

```bash
sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/nginx.conf.example \
        /etc/nginx/sites-available/pipeline-monitor
sudo ln -sf /etc/nginx/sites-available/pipeline-monitor /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
```

Edit `server_name` and SSL paths in the nginx file for production.

### Step 5 — Verify

```bash
sudo systemctl status inferyx-monitoring
curl -s http://127.0.0.1:8090/api/health
curl -s http://127.0.0.1:8090/api/ui/status
```

Open `https://<your-host>/monitoring/admin/` in a browser.

---

## Upgrade

```bash
export VERSION=1.0.51   # new version
export IM_HOME=/opt/pipeline-monitor

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade "inferyx-monitoring==${VERSION}"
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip
```

Upgrade **without** service restart:

```bash
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip --no-restart
```

The upgrade script:

- Deploys new Admin UI to `/opt/pipeline-monitor/www/`
- Merges new `.env` keys (existing values kept)
- Updates systemd unit
- Restarts `inferyx-monitoring` service

**Your config is never overwritten:** `.env` secrets, `batch_file.csv` rows, and `auth.policy` values stay as-is unless you edit them.

---

## Uninstall / remove

### Stop service, keep config

Keeps `/opt/pipeline-monitor` (`.env`, CSV, secrets, logs):

```bash
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --skip-pip
```

### Remove everything

Deletes `/opt/pipeline-monitor` entirely:

```bash
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --purge --skip-pip
```

### Manual cleanup (optional)

```bash
# nginx
sudo rm -f /etc/nginx/sites-enabled/pipeline-monitor
sudo rm -f /etc/nginx/sites-available/pipeline-monitor
sudo nginx -t && sudo systemctl reload nginx

# service user (only if nothing else uses it)
sudo userdel inferyx 2>/dev/null || true
```

### Reinstall from scratch

After `--purge`, repeat [Install from scratch](#install-from-scratch).

---

## Configuration files

| File | Template | Description |
|------|----------|-------------|
| `.env` | `data/.env.example` | SMTP, API token, alert tuning |
| `batch_file.csv` | `data/batch_file.csv.example` | Batch names and schedules |
| `auth.policy` | `release/auth.policy.example` | Admin UI, OAuth, file paths |

All paths in `auth.policy` should point under `/opt/pipeline-monitor/` on the server.

---

## Batch CSV

**Required columns:** `Name`, `Frequency`  
**Optional:** `ExpectedStartTime`, `AvgExecutionTime`, `ExpectedDayOfMonth`, `Status` (`Active` / `Suspended`)

### Scheduled (Daily, Hourly, Weekly, …)

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
payroll_job,Daily,9:00:00,"30 mins",,Active
```

Alerts: **missed**, **failed**, **stuck**, **no_data**

### On-demand (runtime trigger)

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
manual_export,OnDemand,,"30 mins",,Active
```

Use `OnDemand` (or `Adhoc`, `Manual`, `Unscheduled`). Leave start time empty.  
`AvgExecutionTime` enables **stuck** alerts (API start + avg + grace).  
Polling uses `CHECK_WINDOW_MINUTES` windows, not 24/7.

### Continuous (always-on)

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
stream_job,Continuous,,,,Active
```

Alerts: **failed** and **no_data** only.

---

## Alert tuning (.env)

Key variables (full list in `.env.example`):

| Variable | Default | Purpose |
|----------|---------|---------|
| `PIPELINE_CHECK_MODE` | `schedule_windows` | Poll in short windows vs continuous |
| `PIPELINE_CHECK_WINDOW_MINUTES` | `10` | Poll window length |
| `PIPELINE_SCHEDULE_GRACE_MINUTES` | `5` | Delay before missed/stuck alerts |
| `PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS` | `true` | One new email per issue per day |
| `PIPELINE_ALERT_REPLY_TO_THREAD` | `false` | Follow-ups as reply-all in same thread |

Test email threading:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --test-mail-thread
```

---

## Troubleshooting

| Symptom | Fix |
|---------|-----|
| `No such file ... release.sh` | Run `pip install inferyx-monitoring` first |
| Service restart loop | Check `.env` — `journalctl -u inferyx-monitoring -n 50` |
| nginx 404 on `/monitoring/admin/` | Use `nginx.conf.example`; check `ui_base_path` in `auth.policy` |
| Admin UI "Unavailable" | `curl http://127.0.0.1:8090/api/ui/status` |
| Missed vs no_data | Empty API after schedule → **missed**; API error → **no_data** |
| OAuth / SAML login fails | Google: match `redirect_uri`. AWS IDC: use SAML 2.0; match `acs_url` / `entity_id`; check IdP cert file |

See [CHANGELOG.md](CHANGELOG.md).

---

## Service commands

```bash
sudo systemctl start inferyx-monitoring
sudo systemctl stop inferyx-monitoring
sudo systemctl restart inferyx-monitoring
sudo systemctl status inferyx-monitoring
sudo journalctl -u inferyx-monitoring -f
```

Or via package CLI:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl status
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl restart
```
