Metadata-Version: 2.4
Name: inferyx-monitoring
Version: 1.0.51
Summary: Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI
Author-email: Inferyx DevOps <devops@inferyx.com>
License: Copyright (c) 2026 Inferyx. All rights reserved.
        
        Proprietary software. Unauthorized copying, distribution, or use is prohibited
        unless agreed in writing with Inferyx.
        
        For open-source release, replace this file with your chosen license (e.g. MIT, Apache-2.0)
        before publishing to public PyPI.
        
Keywords: batch,monitoring,pipeline,email,inferyx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: python-dateutil>=2.8.2
Requires-Dist: fastapi>=0.110.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: uvicorn[standard]>=0.27.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: admin
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: check-manifest; extra == "dev"
Dynamic: license-file

# inferyx-monitoring

**Version:** 1.0.50 · **PyPI:** [inferyx-monitoring](https://pypi.org/project/inferyx-monitoring/)

Batch monitor + admin UI. One package, one service, one install directory (`/opt/pipeline-monitor`).

---

## Quick reference

| Task | Command |
|------|---------|
| **Build** (maintainer) | `./scripts/build.sh` |
| **Publish** (maintainer) | `./scripts/build.sh --release yes --publish` |
| **Install** (server) | `pip install` → `release.sh install --skip-pip` |
| **Upgrade** (server) | `pip install --upgrade` → `release.sh upgrade --skip-pip` |

**Browser URL:** `https://<host>/monitoring/admin/` (nginx required)

---

## Project layout

```
Monitor/
├── README.md
├── CHANGELOG.md
├── VERSION
├── pipeline_status_mailer.py         ← monitor + alerts
├── auth.policy                       ← gitignored (same file local + server)
├── .env                              ← gitignored
├── batch_file.csv                    ← gitignored
├── secrets/                          ← gitignored (session_secret, etc.)
├── scripts/build.sh                  ← build UI + wheel (not in PyPI)
├── release/
│   ├── release.sh                    ← server install | upgrade
│   └── *.example                     ← auth.policy, nginx, systemd templates
├── admin-ui/                         ← Angular source → admin_static in wheel
└── inferyx_pipeline_monitor/         ← Python package
    ├── settings.py                   ← loads .env
    ├── admin/                        ← Admin API
    └── data/
        ├── .env.example              ← .env template
        └── batch_file.csv.example    ← CSV template
```

### Config files (one set for local and server)

| What | Template (in git) | Runtime (gitignored) |
|------|-------------------|----------------------|
| SMTP, API, alerts | `inferyx_pipeline_monitor/data/.env.example` | `.env` |
| Batch schedules | `inferyx_pipeline_monitor/data/batch_file.csv.example` | `batch_file.csv` |
| Admin UI / OAuth | `release/auth.policy.example` | `auth.policy` |

On a **server**, paths in `auth.policy` point to `/opt/pipeline-monitor/...` (default in the example).  
For **local dev**, copy the same template and use relative paths (see below).

There is **no `config/` folder** and **no `development/` folder**. Settings load from `.env` via `inferyx_pipeline_monitor/settings.py`.

---

## Build (maintainer)

```bash
./scripts/build.sh
./scripts/build.sh --release yes --publish
make clean
```

---

## Install (server)

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install inferyx-monitoring==1.0.50
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh install --skip-pip
```

Edit `/opt/pipeline-monitor/.env` and `/opt/pipeline-monitor/auth.policy`.

---

## Upgrade (server)

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring==1.0.50
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh upgrade --skip-pip
```

---

## Local development

Same config files as production — only paths and URLs differ.

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

cp inferyx_pipeline_monitor/data/.env.example .env
cp inferyx_pipeline_monitor/data/batch_file.csv.example batch_file.csv
cp release/auth.policy.example auth.policy
mkdir -p secrets logs
echo "change-me" > secrets/session_secret

# Edit auth.policy for local:
#   paths.work_dir = "."
#   paths.env_file = ".env"
#   paths.csv_file = "batch_file.csv"
#   paths.session_secret_path = "secrets/session_secret"
#   ui.public_base_url = "http://localhost:4200"
#   auth.google.client_secret_path = "<inline secret or file path>"
#   auth.google.redirect_uri = "http://localhost:4200/api/auth/callback/google"

export PIPELINE_ADMIN_POLICY=auth.policy
./scripts/build.sh
inferyx-monitoring-service --work-dir . --with-admin --admin-port 8091
```

Admin UI hot reload: `cd admin-ui && npm ci && npm start` → http://localhost:4200/

---

## Batch CSV — scheduled vs runtime / on-demand

**Required columns:** `Name`, `Frequency`.  
**Optional:** `ExpectedStartTime`, `AvgExecutionTime`, `ExpectedDayOfMonth`, `Status` (`Active` / `Suspended`).

### Fixed schedule (Daily, Hourly, Weekly, …)

Use when the batch runs at a known time:

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
my_daily_job,Daily,9:00:00,"30 mins",,Active
my_monthly_job,Monthly,2:00:00,"1 hr",15,Active
```

Alerts: **missed** (did not start on time), **failed**, **stuck/running** (past expected end + grace), **no_data** (API error).

### Runtime / on-demand (no fixed time)

Use when the batch is triggered manually or by an external event — **not** on a fixed clock:

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
user_triggered_job,OnDemand,,"30 mins",,Active
```

**Frequency** must be one of: `OnDemand`, `Adhoc`, `Manual`, `Unscheduled`, `Anytime` (case-insensitive).

Leave **ExpectedStartTime** and **ExpectedDayOfMonth** **empty**.  
Set **AvgExecutionTime** if you want **stuck** alerts (e.g. `"30 mins"` — alert if still running 30 min + grace after actual start from API).

| Alert | Applies? | Why |
|-------|----------|-----|
| **Failed** | Yes | API status = failed |
| **no_data** | Yes | API/token/network error |
| **Stuck / running too long** | Only if **AvgExecutionTime** is set | Uses API actual start + avg time + grace (checked in END window) |
| **Running (immediate)** | No | Normal while batch is executing |
| **Missed** | No | No schedule to miss |

**Polling (default `PIPELINE_CHECK_MODE=schedule_windows`):** OnDemand is **not** polled every loop.

| Phase | When API is called |
|-------|-------------------|
| **Idle** (no run) | Discovery poll every `PIPELINE_CHECK_WINDOW_MINUTES` (default 10 min) |
| **Run detected** | START window: actual start + grace → + `CHECK_WINDOW_MINUTES` |
| **Run active** | END window: expected end + grace → + `CHECK_WINDOW_MINUTES` (stuck check) |
| **Between windows** | No API call |
| **After run finished** | Back to idle discovery |

Set `PIPELINE_CHECK_MODE=full_window` to poll OnDemand continuously (old behaviour).

**Tip:** Set `Status=Suspended` for batches you are not monitoring right now.

### Always-on / streaming jobs

For jobs that should run continuously (not slot-based):

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
stream_processor,Continuous,,,,Active
```

**Frequency:** `Continuous`, `Realtime`, `Always`, etc.  
Alerts: **failed** and **no_data** only (no missed / schedule-based stuck).

---

## Troubleshooting

| Symptom | Fix |
|---------|-----|
| `No such file ... release.sh` | Run `pip install inferyx-monitoring` first |
| nginx 404 on `/monitoring/admin/` | Use `release/nginx.conf.example` |
| Admin UI "Unavailable" | `curl http://127.0.0.1:8090/api/ui/status` |
| Service restart loop | Check `.env` — `journalctl -u inferyx-monitoring -n 50` |
| Missed vs no_data | Empty API after schedule → **missed**; API error → **no_data** |

See [CHANGELOG.md](CHANGELOG.md).

---

## Scheduling / tuning (.env)

### When to poll

| Variable | Default | Used when | Meaning |
|----------|---------|-----------|---------|
| `PIPELINE_CHECK_MODE` | `schedule_windows` | Always | **`schedule_windows`** — poll only in short windows after expected start + grace, and after expected end + grace. **`full_window`** — poll continuously from expected start until the slot ends. |
| `PIPELINE_CHECK_INTERVAL` | `60` | Always | Seconds between monitor loop runs. |
| `PIPELINE_CHECK_WINDOW_MINUTES` | `10` | `schedule_windows` only | Length of each poll window. Example: start 09:00 + 5 min grace → poll 09:05–09:15. |
| `PIPELINE_POST_RUN_GRACE_MINUTES` | `60` | `full_window` only | How long to keep polling after expected end. Not used in `schedule_windows`. |

### When to alert

| Variable | Default | Meaning |
|----------|---------|---------|
| `PIPELINE_SCHEDULE_GRACE_MINUTES` | `5` | Minutes after **expected start** before a **missed** alert, and after **expected end** before a **stuck/running** alert. Example: start 09:00 → missed earliest 09:05. |

### API retries

| Variable | Default | Meaning |
|----------|---------|---------|
| `PIPELINE_API_RETRY_COUNT` | `3` | Retries per batch per loop when the API call fails. |
| `PIPELINE_API_RETRY_BACKOFF_SEC` | `5` | Seconds between API retries. |

### Email alert frequency (hourly, daily, weekly — all schedules)

| Variable | Default | Used when | Meaning |
|----------|---------|-----------|---------|
| `PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS` | `true` | Always | At most one **new** email per issue type (missed, failed, running, no_data) per batch per day. |
| `PIPELINE_ALERT_REPLY_TO_THREAD` | `false` | Optional | When `true`, first alert opens a new thread; later alerts for the same ongoing issue are **reply-all** (same conversation). Applies to all frequencies. |
| `PIPELINE_ALERT_COOLDOWN_MINUTES` | `60` | `REPLY=true` **or** `ALL_SCENARIOS=false` | Minimum minutes between repeat alerts. **Ignored** with defaults (`schedule_windows` + once-per-day + no reply) — once-per-day already prevents repeats. Needed for reply-all follow-ups across slots (e.g. hourly) or when once-per-day is off. |
| `PIPELINE_FAILED_ALERT_ONCE_PER_DAY` | `true` | `ALL_SCENARIOS=false` only | Failed-only once-per-day. **Ignored** when `ALL_SCENARIOS=true` (default). |

**Typical flow with defaults (`schedule_windows`):** batch expected 09:00 → poll only 09:05–09:15 and after expected end → grace 5 min → if still not running, **missed** alert (one email per day) → if still running past expected end + 5 min grace, **stuck** alert.

**Minimal config for most users:** keep defaults; only set `PIPELINE_ALERT_REPLY_TO_THREAD=true` if you want follow-up reply-all emails instead of silence after the first alert.

**Test reply-all threading (Office365/Gmail):**

```bash
inferyx-monitoring --test-mail-thread
# or: python3 pipeline_status_mailer.py --test-mail-thread
```

Sends two emails: a new alert, then a reply-all follow-up. Both should appear in one conversation in inbox and Sent.

