Metadata-Version: 2.4
Name: inferyx-monitoring
Version: 1.0.26
Summary: Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI
Author-email: Inferyx DevOps <devops@inferyx.com>
License: Copyright (c) 2026 Inferyx. All rights reserved.
        
        Proprietary software. Unauthorized copying, distribution, or use is prohibited
        unless agreed in writing with Inferyx.
        
        For open-source release, replace this file with your chosen license (e.g. MIT, Apache-2.0)
        before publishing to public PyPI.
        
Keywords: batch,monitoring,pipeline,email,inferyx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: python-dateutil>=2.8.2
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: check-manifest; extra == "dev"
Dynamic: license-file

# inferyx-monitoring — Server deployment

Monitor batch jobs from CSV, poll status via API, and send email alerts for failures, missed runs, long-running jobs, and missing API data.

**Package:** `inferyx-monitoring`  
**CLI:** `inferyx-monitoring`  
**Current version:** `1.0.26` ([PyPI](https://pypi.org/project/inferyx-monitoring/))

---

## What's new in 1.0.26

- **Hotfix** — fixes service crash on startup (`ImportError: CHAT_CONFIG` missing from `config.py` in 1.0.25). Upgrade immediately if you installed 1.0.25.

## What's new in 1.0.25

- **Microsoft Teams alerts** — optional webhook (`PIPELINE_TEAMS_ENABLED`, `PIPELINE_TEAMS_WEBHOOK_URL`).
- **Google Chat alerts** — optional webhook (`PIPELINE_GCHAT_ENABLED`, `PIPELINE_GCHAT_WEBHOOK_URL`).
- Chat sent with **failed**, **running**, and **missed** email alerts (same alert types as the monitor).
- **`no_data`** stays **email-only** to DevOps by default; set `PIPELINE_CHAT_ALERT_NO_DATA=true` to include in chat.
- **`--test-chat-alerts`** — test Teams/GChat webhooks without waiting for a batch alert.
- New optional `.env` keys: `PIPELINE_CHAT_GREETING`, `PIPELINE_CHAT_SIGNATURE`.

## What's new in 1.0.24

- **PyPI documentation updated** — README on PyPI now matches this deployment guide (install, config, upgrade, troubleshooting).
- **Monitoring-only package** — PyPI sdist excludes `dev/`, `docs/`, and `scripts/` (server docs only, no developer publishing guides in the wheel).

## What's new in 1.0.21 – 1.0.23

- **Fix alert email send crash** — resolves `NameError: name 'signature' is not defined` when sending running/failed/missed alerts.
- **Left-aligned email layout** — alert HTML aligns to the left in Gmail/Outlook (no centered block).
- **Safer `.env` migration** — fixes glued comment lines, removes duplicate keys, and keeps non-empty values (e.g. `PIPELINE_MAIL_SIGNATURE`).

## What's new in 1.0.20

- **Auto config migration on upgrade** — each start (and `--init-config`) merges **missing** `.env` keys only; keeps SMTP/API/subjects/batch rows.
- Retires legacy plain `PIPELINE_MAIL_BODY_*` lines so structured emails apply automatically.
- Copies `jfl_batch.csv` → `batch_file.csv` if needed; adds `Status` column when missing.

## What's new in 1.0.19

- **Structured alert emails** — greeting, intro, Batch Details table, Additional Information, Alert Summary, Action Required, Current Status, sign-off, and footer note.
- **Fully configurable from `.env`** — `PIPELINE_MAIL_GREETING`, `PIPELINE_MAIL_ENVIRONMENT`, section headings, per-alert intro/summary/action/status, `PIPELINE_MAIL_SIGNATURE`, `PIPELINE_MAIL_FOOTER_NOTE`.

## What's new in 1.0.18

- **Styled HTML alert emails** — tables, status badges, formatted sign-off.

## What's new in 1.0.17

- **Email bodies** — greeting, **Expected Start**, **Grace Time**, **Alert After**, and closing block.
- New `.env` keys: `PIPELINE_MAIL_GREETING`, placeholders `{grace_time}` and `{alert_after_time}`.

## What's new in 1.0.16

- **Updated deployment docs** — full upgrade guide on PyPI (this README).
- **Upgrade paths** — separate steps for **1.0.15 → 1.0.16** (pip only) vs **1.0.14 or older** (pip + manual config).
- No code changes — documentation release only.

## What's new in 1.0.15

- **`batch_file.csv`** — default batch list filename (replaces `jfl_batch.csv`).
- **`--init-config`** — creates `.env` and `batch_file.csv` only; removes leftover `.example` files.
- **Example templates** — dummy batch names, `localhost` API default, `{batch_name}` email subjects.
- **`PIPELINE_CHECK_MODE=schedule_windows`**, **`Status` column**, API URL fixes.

### Check modes

| `PIPELINE_CHECK_MODE` | Behavior |
|-----------------------|----------|
| `schedule_windows` | **Recommended.** START window at `ExpectedStartTime + grace`; END window at `ExpectedStartTime + AvgExecutionTime + grace`. No API calls outside those windows. |
| `full_window` | Legacy: poll from expected start until end of day. |

Example — `Daily, 7:00, "10 mins", Active`, grace `5`, window `10`:

| Window | Time | Alerts |
|--------|------|--------|
| START | 7:05 – 7:15 | missed, no_data, failed |
| END | 7:15 – 7:25 | long-running, failed |

---

## Paths

| Item | Path |
|------|------|
| Install directory | `/opt/pipeline-monitor` |
| Config | `/opt/pipeline-monitor/.env` |
| Batch list | `/opt/pipeline-monitor/batch_file.csv` |
| Log file | `/opt/pipeline-monitor/pipeline_script.log` |
| Python venv | `/opt/pipeline-monitor/.venv` |
| Service user | `inferyx` |
| systemd unit | `inferyx-monitoring.service` |

---

## 1. Install

```bash
sudo apt update
sudo apt install -y python3 python3-venv python3-pip
```

**Service user** — check first; create only if missing:

```bash
id inferyx
```

If the command returns `no such user`, create the user:

```bash
sudo useradd --system --home-dir /opt/pipeline-monitor --shell /usr/sbin/nologin inferyx
```

If the user already exists, skip the command above and continue.

```bash
sudo mkdir -p /opt/pipeline-monitor
sudo chown inferyx:inferyx /opt/pipeline-monitor

sudo -u inferyx python3 -m venv /opt/pipeline-monitor/.venv
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade pip inferyx-monitoring
```

Fresh install pulls the latest release from PyPI (currently **1.0.26**).

---

## 2. Create config files (first time)

```bash
cd /opt/pipeline-monitor
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor
```

Creates `.env` and `batch_file.csv` if missing.  
On every run, **auto-migrates** existing config: adds missing `.env` keys, retires old mail-body overrides, migrates CSV safely.  
**Never** overwrites SMTP/API secrets, custom subjects, or batch rows.

---

## 3. Configure `.env`

```bash
sudo -u inferyx vi /opt/pipeline-monitor/.env
sudo chmod 600 /opt/pipeline-monitor/.env
```

### Required

| Variable | Description |
|----------|-------------|
| `PIPELINE_SMTP_HOST` | SMTP server (e.g. `smtp.office365.com`) |
| `PIPELINE_SMTP_PORT` | SMTP port (usually `587`) |
| `PIPELINE_SMTP_USERNAME` | SMTP login / sender email |
| `PIPELINE_FROM_NAME` | Display name in From header |
| `PIPELINE_SMTP_PASSWORD` | SMTP password |
| `PIPELINE_MAIL_TO` | Primary alert recipients (comma-separated) |
| `PIPELINE_API_BASE_URL` | API path only — **no** `name=` in URL |
| `PIPELINE_API_TOKEN` | API authentication token |
| `PIPELINE_API_TOKEN_HEADER` | Header name for token (usually `token`) |
| `PIPELINE_DEVOPS_EMAIL` | Recipient for `no_data` and script-failure alerts |

### API example

```env
PIPELINE_API_BASE_URL=http://localhost:8080/framework/metadata/getBaseEntityStatusByCriteria
PIPELINE_API_TOKEN=your_inferyx_api_token
PIPELINE_API_TOKEN_HEADER=token
PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false
```

Replace `localhost` with your Inferyx host. Use the **base path only** — the monitor adds `name=<batch>` for each row in `batch_file.csv`.  
Keep `PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false` unless your API supports `startDate` / `endDate`.

### Teams / Google Chat (optional)

Sent with batch email alerts for **`failed`**, **`running`**, and **`missed`** only.

**`no_data`** (no API record for the batch in the schedule window) sends **email to DevOps only** — not Teams/GChat by default. Set `PIPELINE_CHAT_ALERT_NO_DATA=true` to include it in chat with a plain-language explanation.

| Variable | Default | Purpose |
|----------|---------|---------|
| `PIPELINE_TEAMS_ENABLED` | `false` | Enable Microsoft Teams (Power Automate / webhook) |
| `PIPELINE_TEAMS_WEBHOOK_URL` | empty | Teams incoming webhook URL |
| `PIPELINE_GCHAT_ENABLED` | `false` | Enable Google Chat webhook |
| `PIPELINE_GCHAT_WEBHOOK_URL` | empty | Google Chat space webhook URL |
| `PIPELINE_CHAT_GREETING` | built-in | Opening line in chat message |
| `PIPELINE_CHAT_SIGNATURE` | built-in | Sign-off line in chat message |
| `PIPELINE_CHAT_ALERT_NO_DATA` | `false` | Also post `no_data` alerts to Teams/GChat (default: email only) |

Example:

```env
PIPELINE_TEAMS_ENABLED=true
PIPELINE_TEAMS_WEBHOOK_URL=https://your-teams-webhook-url
PIPELINE_GCHAT_ENABLED=true
PIPELINE_GCHAT_WEBHOOK_URL=https://chat.googleapis.com/v1/spaces/.../messages?key=...&token=...
```

Test webhooks without waiting for a batch alert:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --test-chat-alerts --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env
```

| Chat alert | Meaning |
|------------|---------|
| `failed` | Batch execution failed |
| `running` | Batch still running past expected end + grace |
| `missed` | Batch did not start in schedule window |
| `no_data` | No API record returned (email only unless `PIPELINE_CHAT_ALERT_NO_DATA=true`) |

### Email templates (optional)

| Variable | Purpose |
|----------|---------|
| `PIPELINE_MAIL_CC` | CC recipients |
| `PIPELINE_MAIL_GREETING` | Opening line (default: `Hello Team,`) |
| `PIPELINE_MAIL_ENVIRONMENT` | Shown in Additional Information (default: `Production`) |
| `PIPELINE_MAIL_HEADING_*` | Section titles: `BATCH_DETAILS`, `ADDITIONAL_INFO`, `ALERT_SUMMARY`, `ACTION_REQUIRED`, `CURRENT_STATUS` |
| `PIPELINE_MAIL_INTRO_*` | Intro paragraph per alert: `RUNNING`, `FAILED`, `MISSED`, `NO_DATA`, `DEFAULT` |
| `PIPELINE_MAIL_SUMMARY_*` | Alert Summary section (supports `{batch_name}`, etc.) |
| `PIPELINE_MAIL_ACTION_*` | Action Required section (HTML `<ul>` supported) |
| `PIPELINE_MAIL_STATUS_*` | Current Status section |
| `PIPELINE_MAIL_SIGNATURE` | Sign-off block |
| `PIPELINE_MAIL_FOOTER_NOTE` | Automated notification / ITSM footer |
| `PIPELINE_MAIL_BODY_*` | Optional full HTML layout override per alert type |

Dynamic fields in tables: `{batch_name}`, `{status}`, `{expected_start_time}`, `{expected_end_time}`, `{grace_time}`, `{alert_after_time}`, `{frequency}`, `{avg_time}`, `{current_time}`, `{delay}`, `{environment}`.

Omit `PIPELINE_MAIL_BODY_*` to use the **built-in structured layout** (recommended). Customize sections via `PIPELINE_MAIL_INTRO_RUNNING`, etc. See `.env.example`.

Example subjects (dummy names — real batch name comes from `{batch_name}` at runtime):

```env
PIPELINE_MAIL_SUBJECT_DEFAULT=Inferyx | Batch Alert | {batch_name} | {issue_type_upper}
PIPELINE_MAIL_SUBJECT_FAILED=Inferyx | Batch Failed: {batch_name}
```

### Scheduling (optional)

| Variable | Default | Meaning |
|----------|---------|---------|
| `PIPELINE_CHECK_MODE` | `full_window` | `schedule_windows` = check only at start/end windows; `full_window` = poll all day |
| `PIPELINE_CHECK_WINDOW_MINUTES` | `10` | Minutes each start/end check window stays open |
| `PIPELINE_CHECK_INTERVAL` | `60` | Seconds between service poll cycles |
| `PIPELINE_SCHEDULE_GRACE_MINUTES` | `5` | Minutes after expected start/end before alerts |
| `PIPELINE_POST_RUN_GRACE_MINUTES` | `60` | Used only when `CHECK_MODE=full_window` |
| `PIPELINE_ALERT_COOLDOWN_MINUTES` | `60` | Cooldown between repeat alerts |
| `PIPELINE_FAILED_ALERT_ONCE_PER_DAY` | `true` | One failed alert per day per batch |
| `PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS` | `true` | One alert per issue type per day |

Recommended production settings:

```env
PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10
PIPELINE_SCHEDULE_GRACE_MINUTES=5
```

---

## 4. Configure `batch_file.csv`

```bash
sudo -u inferyx vi /opt/pipeline-monitor/batch_file.csv
```

**Rules:**

- **One batch per line** — each row must end with `Active` or `Suspended` on its own line.
- Do not glue rows (bad: `Activebatch_other` or `Statusbatch_other`).
- Quote values with commas or spaces: `"09:00,10:00"`, `"3 mins"`.
- Use 24-hour time: `0:30:00`, `16:30:00`.

| Column | Required | Description |
|--------|----------|-------------|
| `Name` | Yes | Batch name as in the Inferyx API |
| `Frequency` | Yes | `Daily`, `Hourly`, `Monthly`, weekday name, etc. |
| `ExpectedStartTime` | Scheduled | `9:30:00` or `"09:00,10:00,11:00"` |
| `AvgExecutionTime` | Recommended | `"3 mins"`, `"10 mins"`, `"1 Hr"` |
| `ExpectedDayOfMonth` | Monthly | Day 1–31 |
| `Status` | Optional | `Active` or `Suspended` (default: Active) |

### Example (replace dummy names with your Inferyx batch names)

```csv
Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
dummy_batch_name,Daily,9:00:00,"10 mins",,Active
dummy_batch_other,Daily,12:00:00,"5 mins",,Suspended
```

**Upgrading from `jfl_batch.csv`?** See [§7 Upgrade](#7-upgrade).

---

## 5. Test

Run as **one single line** (do not use `\` line breaks — they cause `unrecognized arguments` errors):

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
```

```bash
tail -100 /opt/pipeline-monitor/pipeline_script.log
```

---

## 6. Start service (systemd)

```bash
sudo tee /etc/systemd/system/inferyx-monitoring.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Batch Monitor
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
Restart=always
RestartSec=10
Environment=PYTHONUNBUFFERED=1
Environment=PIPELINE_ENV_FILE=/opt/pipeline-monitor/.env
Environment=PIPELINE_LOG_FILE=/opt/pipeline-monitor/pipeline_script.log
Environment=PIPELINE_CSV_FILE=/opt/pipeline-monitor/batch_file.csv

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring.service
sudo systemctl status inferyx-monitoring.service
```

Logs:

```bash
sudo journalctl -u inferyx-monitoring.service -f
```

---

## 7. Upgrade

Check installed version:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip show inferyx-monitoring | grep Version
```

| Installed version | What to do |
|-------------------|------------|
| **1.0.26** | Nothing — you are current. |
| **1.0.25** | **Upgrade to 1.0.26 immediately** — startup crash hotfix. |
| **1.0.24** | §7.1 only (`pip install` + restart). Add Teams/GChat keys to `.env` if needed (§3). |
| **1.0.21 – 1.0.23** | §7.1 only (`pip install` + restart). |
| **1.0.16 – 1.0.20** | §7.1 only (`pip install` + restart). Migration runs automatically. |
| **1.0.14 or older** | §7.1 + review §7.2 if paths still use `jfl_batch.csv`. |

### 7.1 Upgrade the package

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service
```

**Auto migration (1.0.20+)** runs on every service start — no manual `.env` edit required. **1.0.21+** also cleans duplicate/glued `.env` lines safely:

| Action | Safe? |
|--------|-------|
| Add **missing** keys to `.env` (e.g. new `PIPELINE_MAIL_INTRO_RUNNING`) | Yes — existing keys kept |
| Retire legacy `PIPELINE_MAIL_BODY_*` (plain HTML) | Yes — enables structured emails |
| Copy `jfl_batch.csv` → `batch_file.csv` if only legacy file exists | Yes — original kept |
| Add `Status` column to CSV if missing (default `Active`) | Yes — rows preserved |
| Overwrite SMTP/API token, subjects, batch names | **Never** |

Optional — see migration output:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor
```

### 7.2 Manual changes (1.0.14 or older only)

Skip §7.2–7.5 if you are already on **1.0.15** or **1.0.16**.

| # | Item | Action |
|---|------|--------|
| 1 | **Batch CSV filename** | Default is now `batch_file.csv` (was `jfl_batch.csv`). **Choose one:** rename your file, or keep the old name and point the service at it (step 2). |
| 2 | **systemd unit** | Update `ExecStart`, `--csv-file`, and `PIPELINE_CSV_FILE` if you renamed the CSV (see step 3). |
| 3 | **`.env` (recommended)** | Add or confirm these lines (your SMTP/API secrets stay as they are): |

```env
PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10
PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false
```

Optional — only if you want custom email subjects (use `{batch_name}` for the real batch name at runtime):

```env
PIPELINE_MAIL_SUBJECT_DEFAULT=Inferyx | Batch Alert | {batch_name} | {issue_type_upper}
PIPELINE_MAIL_SUBJECT_NO_DATA=Inferyx | Batch No Data: {batch_name}
PIPELINE_MAIL_SUBJECT_FAILED=Inferyx | Batch Failed: {batch_name}
PIPELINE_MAIL_SUBJECT_RUNNING=Inferyx | Batch Running: {batch_name}
PIPELINE_MAIL_SUBJECT_MISSED=Inferyx | Batch Missed: {batch_name}
```

| # | Item | Action |
|---|------|--------|
| 4 | **API URL** | Must be base path only — no `name=` in `PIPELINE_API_BASE_URL`. Example: `http://<your-inferyx-host>:8080/framework/metadata/getBaseEntityStatusByCriteria` |
| 5 | **CSV `Status` column** | Add `Active` or `Suspended` as the last column per row if missing. |
| 6 | **Restart service** | Required after pip upgrade and any systemd edits. |

### 7.3 Option A — Rename CSV to `batch_file.csv` (recommended)

```bash
sudo -u inferyx mv /opt/pipeline-monitor/jfl_batch.csv /opt/pipeline-monitor/batch_file.csv
```

Update systemd (if it still references `jfl_batch.csv`):

```bash
sudo sed -i 's|jfl_batch.csv|batch_file.csv|g' /etc/systemd/system/inferyx-monitoring.service
sudo systemctl daemon-reload
sudo systemctl restart inferyx-monitoring.service
```

### 7.4 Option B — Keep `jfl_batch.csv` (no rename)

Edit the service file and set the old path explicitly:

```bash
sudo vi /etc/systemd/system/inferyx-monitoring.service
```

Ensure these lines use `jfl_batch.csv`:

```ini
ExecStart=.../inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/jfl_batch.csv
Environment=PIPELINE_CSV_FILE=/opt/pipeline-monitor/jfl_batch.csv
```

Then:

```bash
sudo systemctl daemon-reload
sudo systemctl restart inferyx-monitoring.service
```

### 7.5 Test after upgrade

Run as **one single line**:

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
```

If you kept `jfl_batch.csv`, change `--csv-file` to that path.

```bash
tail -50 /opt/pipeline-monitor/pipeline_script.log
sudo systemctl status inferyx-monitoring.service
```

### 7.6 What is not changed by upgrade

| File / setting | Changed by `pip upgrade`? |
|----------------|---------------------------|
| `/opt/pipeline-monitor/.env` | No |
| Your batch CSV (`batch_file.csv` or `jfl_batch.csv`) | No |
| systemd unit file | No — edit manually if paths changed |
| Log file | No |

### 7.7 Future upgrades

```bash
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service
```

Check **What's new** at the top of this README and on [PyPI](https://pypi.org/project/inferyx-monitoring/) for version-specific notes.

---

## Troubleshooting

| Symptom | Fix |
|---------|-----|
| `unrecognized arguments:` | Run the test command as **one line** — no `\` at end of lines |
| API error / empty response | `PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false`; remove `name=` from `PIPELINE_API_BASE_URL` |
| Wrong batch in API URL | Base URL must not contain a fixed batch name |
| CSV errors / wrong batches | One batch per line; fix glued rows like `Activebatch_...` |
| No email | Check SMTP settings in `.env` |
| `NameError: signature` on alert | Upgrade to **1.0.21** or newer |
| Email body centered in Gmail | Upgrade to **1.0.21** or newer |
| Teams/GChat not firing | Check `PIPELINE_TEAMS_ENABLED` / `PIPELINE_GCHAT_ENABLED` and webhook URLs; run `--test-chat-alerts` |
| Service `activating (auto-restart)` after 1.0.25 | Upgrade to **1.0.26** (`ImportError: CHAT_CONFIG`) |
| Skip a batch | `Status=Suspended` in CSV |
| Still using `jfl_batch.csv` | Set `PIPELINE_CSV_FILE` or `--csv-file` to the old path, or rename to `batch_file.csv` |
