Metadata-Version: 2.4
Name: inferyx-monitoring
Version: 1.0.1
Summary: Monitor batch pipelines via API and send email alerts for failures and missed runs
Author-email: Inferyx DevOps <devops@inferyx.com>
License: Copyright (c) 2026 Inferyx. All rights reserved.
        
        Proprietary software. Unauthorized copying, distribution, or use is prohibited
        unless agreed in writing with Inferyx.
        
        For open-source release, replace this file with your chosen license (e.g. MIT, Apache-2.0)
        before publishing to public PyPI.
        
Project-URL: Homepage, https://github.com/inferyx/inferyx-devops
Project-URL: Repository, https://github.com/inferyx/inferyx-devops
Project-URL: Documentation, https://github.com/inferyx/inferyx-devops/tree/monitor
Keywords: batch,monitoring,pipeline,email,inferyx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: python-dateutil>=2.8.2
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: check-manifest; extra == "dev"
Dynamic: license-file

# Pipeline Batch Monitor

Monitor batch jobs from CSV, check status via API, and email alerts for failures, missed runs, long-running jobs, and missing API data.

**Version:** see `VERSION` file.

## Quick start (development)

```bash
cp .env.example .env          # edit with real SMTP/API credentials
./pipeline-monitor setup      # create .venv + dependencies
./pipeline-monitor validate   # check .env, CSV, Python
./pipeline-monitor run-once   # single test cycle
./pipeline-monitor run        # continuous local run
```

Same commands via Make: `make setup`, `make validate`, `make run-once`, `make build`.

## Package workflow (build → deploy → use)

One CLI drives the full lifecycle:

```bash
./pipeline-monitor <command>
```

| Command | Purpose |
|---------|---------|
| `setup` | Create `.venv`, install `requirements.txt` |
| `validate` | Validate `.env`, `jfl_batch.csv`, syntax |
| `run` | Run locally (foreground, continuous) |
| `run-once` | One monitoring cycle (cron/debug) |
| `build` | Create `dist/pipeline-status-mailer-<version>.tar.gz` + `dist/latest.tar.gz` |
| `install` | Production install from source (+ systemd, needs sudo) |
| `deploy` | Install from tarball (needs sudo) |
| `status` | `systemctl status` |
| `logs` | Follow journal or app log |

### 1. Build release package

On your build machine (with `.env` configured for validation only — secrets are **not** bundled by default):

```bash
./pipeline-monitor validate
./pipeline-monitor build
```

Output:

- `dist/pipeline-status-mailer-1.0.0.tar.gz`
- `dist/latest.tar.gz` (copy of latest build)
- `MANIFEST.txt` inside the package (file list + SHA256)

Private pipeline (bundle `.env` — handle tarball as secret):

```bash
./pipeline-monitor build --with-env
```

### 2. Copy to target server

```bash
scp dist/latest.tar.gz user@prod-server:/tmp/
```

### 3. Deploy on target server

```bash
ssh user@prod-server
tar -xzf /tmp/latest.tar.gz -C /tmp
cd /tmp/pipeline-status-mailer-*
cp .env.example .env && vi .env
sudo ./pipeline-monitor install --user inferyx --group inferyx
```

**One-step deploy** (from dev machine with `.env` present, copies `.env` into package during deploy):

```bash
sudo ./pipeline-monitor deploy dist/latest.tar.gz --user inferyx --group inferyx
```

On server after install:

```bash
sudo ./pipeline-monitor status
sudo ./pipeline-monitor logs
/opt/pipeline-status-mailer/pipeline-monitor run-once
```

See `package/INSTALL.md` inside the release tarball for server-side notes.

### Default production paths

| Item | Path |
|------|------|
| Application | `/opt/pipeline-status-mailer` |
| Python venv | `/home/inferyx/monitoring` |
| Config | `/opt/pipeline-status-mailer/.env` |
| Batches | `/opt/pipeline-status-mailer/jfl_batch.csv` |
| systemd unit | `pipeline-status-mailer.service` |

## Batch CSV (`jfl_batch.csv`)

| Column | Required | Description |
|--------|----------|-------------|
| `Name` | Yes | Batch name for API |
| `Frequency` | Yes | `Daily`, `Hourly`, `Monthly`, `Continuous`, … |
| `ExpectedStartTime` | Scheduled | e.g. `9:30`, `7:20:00`, or `09:00,12:00` |
| `AvgExecutionTime` | Recommended | e.g. `10 mins`, `1 Hr` |
| `ExpectedDayOfMonth` | Monthly | Day 1–31 (default **1**) |

Monthly example:

```csv
batch_monthly_finance_close,Monthly,9:30,10 mins,1
```

## Configuration

Required in `.env` (see `.env.example`):

- SMTP: `PIPELINE_SMTP_*`, `PIPELINE_MAIL_TO`, `PIPELINE_FROM_NAME`
- API: `PIPELINE_API_BASE_URL`, `PIPELINE_API_TOKEN`, `PIPELINE_API_TOKEN_HEADER`
- `PIPELINE_DEVOPS_EMAIL`

Optional tuning:

| Variable | Default |
|----------|---------|
| `PIPELINE_CHECK_INTERVAL` | 60 |
| `PIPELINE_SCHEDULE_GRACE_MINUTES` | 5 |
| `PIPELINE_POST_RUN_GRACE_MINUTES` | 60 |

Email templates: `config.py`.

## Project layout

```
Monitor/
  pipeline-monitor      # Main CLI (build / deploy / run)
  pipeline_status_mailer.py
  config.py
  jfl_batch.csv
  .env.example
  VERSION
  Makefile
  scripts/
    lib/common.sh
    build_release.sh
    deploy_release.sh
    install_pipeline_status_mailer.sh
    validate_env.sh
    setup_python_env.sh
    run_with_venv.sh
  package/INSTALL.md
  dist/                 # Created by build
```

## Publish to PyPI (pip install)

Package name: **`inferyx-monitoring`**

Full step-by-step guide: **[docs/PYPI_PUBLISHING.md](docs/PYPI_PUBLISHING.md)**

Quick publish:

```bash
pip install build twine
make pip-build          # creates dist/*.whl and dist/*.tar.gz
make pip-check
make pip-test-upload    # optional: TestPyPI first
make pip-upload         # production PyPI (needs token)
```

Install from PyPI:

```bash
pip install inferyx-monitoring
mkdir ~/pipeline-monitor && cd ~/pipeline-monitor
inferyx-monitoring --init-config
vi .env jfl_batch.csv
inferyx-monitoring --once --work-dir .
```

## Makefile shortcuts

```bash
make setup validate build
make install    # sudo install from source
make deploy     # sudo deploy dist/latest.tar.gz
make status logs clean
make pip-build pip-upload   # PyPI release
```
