Metadata-Version: 2.4
Name: loggair
Version: 0.1.0
Summary: Modern, multiprocess-safe logging for High-Performance Computing and ML.
Author: Gert Behiels
License-Expression: MIT
Project-URL: Homepage, https://github.com/Gearlux/loggair
Project-URL: Repository, https://github.com/Gearlux/loggair.git
Project-URL: Issues, https://github.com/Gearlux/loggair/issues
Project-URL: Changelog, https://github.com/Gearlux/loggair/blob/main/CHANGELOG.md
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Logging
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: loguru>=0.7.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: alerts
Requires-Dist: apprise>=1.9.0; extra == "alerts"
Provides-Extra: dev
Requires-Dist: apprise>=1.9.0; extra == "dev"
Requires-Dist: black<25.0.0,>=24.0.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=5.0.0; extra == "dev"
Requires-Dist: isort<6.0.0,>=5.13.0; extra == "dev"
Requires-Dist: flake8<8.0.0,>=6.0.0; extra == "dev"
Requires-Dist: flake8-junit-report>=2.1.0; extra == "dev"
Requires-Dist: mypy<2.0.0,>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
Requires-Dist: pytest<9.0.0,>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov<7.0.0,>=4.0.0; extra == "dev"
Dynamic: license-file

# Loggair

Modern, multiprocess-safe logging specifically engineered for High-Performance Computing (HPC) and Machine Learning (ML).

## Why Loggair?
ML experiments and distributed training (like PyTorch DDP) present unique logging challenges:
- **Log Storms:** 128 identical lines when 128 GPUs log simultaneously.
- **Multiprocess Safety:** Corrupted log files when multiple processes write to the same file.
- **Startup Consistency:** Tracking which logs belong to which experiment run.

Loggair solves these by being **distributed-aware** and **framework-agnostic**.

## Design Goals & Requirements

### Core Functionality
- **High-Fidelity Logging:** Provide a thread-safe and multiprocess-safe logging engine.
- **Unified Observability:** Standardize logging levels (TRACE, DEBUG, INFO, SUCCESS, WARNING, ERROR, CRITICAL).
- **Auto-Infrastructure:** Automatically create log directories if they do not exist.
- **Global Configuration:** Support XDG-standard configuration (~/.config/loggair/config.yaml).

### Integration
- **Log-Symmetry:** Support automatic log file naming based on the active script/config name.
- **TTY-Aware Console Output:** Provide colorized, rank-filtered console output (powered by [Loguru](https://github.com/Delgan/loguru)), with ANSI coloring auto-disabled when stderr is piped or redirected.
- **Framework Interception:** Intercept standard library logging and redirect to Loggair.

### Performance
- **Zero-Overhead Inactive Levels:** Ensure that TRACE/DEBUG levels have minimal impact when disabled.
- **Asynchronous Sinks:** Support enqueued logging to prevent blocking the hot path.

## Key Features
- **Rank-Aware:** Automatically filters console output to Rank 0 (supports SLURM, DDP, MPI).
- **Multiprocess Safe:** Uses Loguru's `enqueue=True` for thread/process safety.
- **Startup Rotation:** Archives old logs on script start, giving every run a fresh log file.
- **Runtime Rotation:** Optionally rotate the active log by size or time (`rotation: "100 MB"`, `"daily"`, `"00:00"`) for long-running processes — training loops, FastAPI/MCP services (see [Runtime Log Rotation](#8-runtime-log-rotation-long-running-processes)).
- **Framework Interoperability:** Automatically intercepts and formats logs from **TensorFlow**, **PyTorch**, **JAX**, and standard Python `logging`.
- **Zero-Blocking:** Non-blocking logging via background sinking.
- **Runtime Kill-Switch:** Set `LOGGAIR_DISABLE_LOGGING=1` (or `LOGGAIR_DISABLE_MULTIPROCESS_LOGGING=1` for workers only) to get a zero-overhead no-op logger — no files, sinks, or queues (see [Disabling Logging](#5-disabling-logging)).
- **Compressed Archives:** Optionally gzip/zip rotated log archives via `compression` (see [Compressing Rotated Archives](#6-compressing-rotated-archives)).
- **Structured / JSON Logging:** Set `serialize: true` to emit one JSON object per line on both sinks — machine-readable for Elasticsearch/Kibana, Grafana Loki, or Datadog (see [Structured / JSON Logging](#7-structured--json-logging)).
- **Per-Worker Log Files:** `worker_files: true` gives every rank/worker its own file (`app.rank2.log`, `app.worker-1.log`) — no shared-file races, full rotation everywhere; merge at read time with lnav (see [Per-Worker Log Files](#16-per-worker-log-files)).
- **Interception Modes:** Embed Loggair next to a framework that owns logging (uvicorn, gunicorn) with `intercept: coexist`, protect specific loggers with `intercept_exclude`, or switch interception `off` (see [Interception Modes](#17-interception-modes)).
- **Webhook & Alerting Sinks:** Route `ERROR`/`CRITICAL` records to Slack, Teams, email, or any of ~100 platforms via [Apprise](https://github.com/caronc/apprise) URLs — batched, throttled, and off the logging hot path (see [Webhook & Alerting Sinks](#15-webhook--alerting-sinks)).
- **Experiment Context Injection:** Stamp `epoch`/`step`/`run_id` once with `set_context()` and every record in the process carries it — in the default line layout and as structured JSON fields (see [Experiment Context](#14-experiment-context-injection)).
- **Config Introspection:** `get_active_config()` returns the fully-resolved, JSON-serializable settings currently in effect (see [Config Introspection](#13-config-introspection)).
- **Dynamic Level Adjustment:** Change verbosity in a *running* job — `reconfigure(console_level="DEBUG")` programmatically, or via POSIX signals: a reload signal re-reads the config files, a debug signal toggles DEBUG on/off (see [Dynamic Level Adjustment](#12-dynamic-level-adjustment-at-runtime)).
- **Custom Log Formats:** Override the per-sink loguru format strings (`console_format` / `file_format`) to add thread/process IDs or custom telemetry fields (see [Custom Log Formats](#11-custom-log-formats)).
- **TTY-Aware Color:** Console colorization is auto-detected by default — colors on an interactive terminal, plain text when stderr is redirected or piped (e.g. `2> run.log`, `| tee`), so log files never get polluted with ANSI escape codes. Set the standard `NO_COLOR` env var to force plain text when auto-detection is wrong (see [Console Coloring](#4-console-coloring)).

## Installation

```bash
pip install loggair                     # from PyPI
pip install "loggair[alerts]"           # + webhook/alerting sinks (apprise)
```

Latest development version straight from GitHub:

```bash
pip install git+https://github.com/Gearlux/loggair.git@main
```

```python
import loggair
print(loggair.__version__)              # installed distribution version
```

## Quick Start
```python
from loggair import get_logger, configure_logging

# Optional: customize levels and directories
configure_logging(log_dir="./experiment_logs", console_level="INFO")

logger = get_logger(__name__)

logger.info("Starting training loop...")
logger.debug("Hyperparameters: batch_size=32, lr=0.001")
logger.success("Model checkpoint saved!")
```

`get_logger(name)` mirrors `logging.getLogger(name)` semantics: the given name
**overrides** the auto-detected module name as the `{name}` field in both sinks
and participates in [`module_levels`](#3-per-logger-per-sink-level-overrides)
prefix matching. The usual `get_logger(__name__)` idiom behaves identically to
the bare logger.

## Configuration
Loggair supports a hierarchical configuration system that allows you to manage settings across different projects and environments.

### 1. Configuration Priority
Settings are resolved in the following order (highest to lowest):
1.  **Function Arguments** (passed to `configure_logging()`)
2.  **Environment Variables** (prefixed with `LOGGAIR_`)
3.  **Local `loggair.yaml` / `loggair.yml`**
4.  **Local `pyproject.toml`** (under `[tool.loggair]`)
5.  **XDG User Config** (`~/.config/loggair/config.yaml`)
6.  **Defaults**

Resolution is **presence-based**: falsy-but-valid values win over defaults, so
`rotation_on_startup: false` or `retention: 0` in any config file are honored.
Config files merge **shallowly** — a `module_levels:` block in a local
`loggair.yaml` fully *replaces* one from the XDG global config (rules are not
merged per-prefix).

### 2. Usage Examples

#### via `pyproject.toml`
```toml
[tool.loggair]
log_dir = "./custom_logs"
console_level = "DEBUG"
retention = 10
```

#### via `loggair.yaml`
```yaml
log_dir: "./experiment_logs"
file_level: "TRACE"
enqueue: true
rotation_on_startup: true
compression: "gz"   # compress rotated archives (see §6)
```

#### via Environment Variables
```bash
export LOGGAIR_DIR="/var/log/myapp"
export LOGGAIR_CONSOLE_LEVEL="ERROR"
export NO_COLOR=1                 # force plain console output (see §4)
```

### 3. Per-logger, Per-sink Level Overrides

`file_level` / `console_level` set global thresholds for the file and console sinks. To override the threshold for specific loggers — and, optionally, only on one of the two sinks — use `module_levels`:

```yaml
file_level: "DEBUG"
console_level: "INFO"

module_levels:
  # Silence a chatty discovery logger on the file sink, in DataLoader workers only.
  # The main process still emits INFO so first-time discovery stays visible.
  "waivefront.rfuav.data.archive":
    file: WARNING
    workers_only: true

  # Quiet on screen, full detail on disk — promotes this logger below the global file_level.
  "noisy.lib.but.useful.in.debug":
    console: ERROR
    file: DEBUG
```

Rules:

- Keys match loguru's `record["name"]` (the calling module's dotted path) by **dotted-segment prefix**: `"pkg.sub"` matches `"pkg.sub"` and `"pkg.sub.mod"`, but **not** `"pkg.subway"`. `"foo"` does not match `"foobar.baz"`.
- On overlap, the **longest matching prefix wins**. YAML order is irrelevant.
- `console:` and `file:` are each optional; an omitted sink falls back to the global level for that logger.
- `workers_only: true` restricts the override to non-`MainProcess` processes (typically PyTorch DataLoader workers).
- Unknown sub-keys or invalid level strings raise `ValueError` at startup — no silent ignores.

**Third-party defaults:** interception gates a few chronically chatty stdlib
loggers (`asyncio`, `httpx`, `urllib3`, `datasets`, `filelock`) at `WARNING` by
default. A `module_levels` rule targeting one of them lifts that gate and lets
your per-sink thresholds decide instead:

```yaml
module_levels:
  "httpx":
    file: DEBUG      # see full httpx detail on disk despite the default silence
```

See [`loggair.example.yaml`](https://github.com/Gearlux/loggair/blob/main/loggair.example.yaml) for the full reference.

### 4. Console Coloring

The console sink colorizes its output with ANSI codes. By default this is
**auto-detected** from the terminal: colors on an interactive TTY, plain text
when stderr is redirected or piped (`2> run.log`, `| tee`, a plain Jenkins
console), so log files never accumulate escape codes.

TTY auto-detection is unreliable in some terminals and wrappers. The **only**
environment control is the standard [`NO_COLOR`](https://no-color.org) variable
(there is no bespoke Loggair color env var); for programmatic control, the
`colorize=` argument is a tri-state. Precedence (highest first):

| Source | Value | Effect |
| --- | --- | --- |
| `configure_logging(colorize=...)` arg | `True` / `False` / `None` | force on / off / auto |
| `NO_COLOR` env | any non-empty value | force off (the cross-tool [no-color.org](https://no-color.org) convention) |
| `NO_COLOR` env | empty (`NO_COLOR=`) | re-allow colors → auto-detect |
| *(unset)* | — | auto-detect |

```bash
# Force plain text (e.g. a terminal whose TTY detection misfires):
export NO_COLOR=1
```

To change coloring **after** logging is already configured — including the lazy
auto-configuration the first `get_logger()` triggers — use `reconfigure()`
(a thin `configure_logging(force=True, ...)` wrapper that reloads the sinks).
Setting the env var alone is too late once the console sink exists: its
`colorize` was baked at first configure, so only a reload re-resolves it.

```python
import loggair

loggair.reconfigure(colorize=False)  # reload with colors off
```

**Non-interactive consumers** (whose stderr is captured or relayed, so ANSI is
noise) default coloring **off** for themselves with one call at import:

```python
import loggair

loggair.force_no_color()  # setdefault NO_COLOR=1 + reload if already configured
```

`force_no_color()` both sets `NO_COLOR=1` (inherited by child processes) **and**
reloads loggair if it was already configured — because env inheritance alone
doesn't help once a coloring decision is baked into the live sink. The
[navigaitor](https://github.com/Gearlux/navigaitor) MCP server and the
[FluxStudio](https://github.com/Gearlux/fluxstudio) ComfyUI extension call it at
import; to deliberately re-allow colors there, pre-set `NO_COLOR=` (empty).

### 5. Disabling Logging

For perf-critical runs or test suites that want **no** logging overhead — no log
files, no sinks, no background queue — disable Loggair entirely with an
environment variable. `get_logger()` then returns a no-op `NullLogger` that
mirrors the loguru API (every call is a safe no-op, chains and context managers
still work), so call-sites need no changes:

```bash
# Disable in every process:
export LOGGAIR_DISABLE_LOGGING=1

# OR disable only in worker/child processes (keeps the main process logging,
# silences e.g. PyTorch DataLoader workers that would re-emit the same lines):
export LOGGAIR_DISABLE_MULTIPROCESS_LOGGING=1
```

Both accept the usual truthy values (`1`, `true`, `yes`, `t`, `y`). These gate
`get_logger()` only: the common lazy-config path never creates a file because
`get_logger()` short-circuits before configuring. An explicit
`configure_logging()` call still sets up sinks if you make one.

### 6. Compressing Rotated Archives

[Startup rotation](#key-features) archives the previous run's log with a
timestamp. Set `compression` to gzip/zip those archives:

```python
from loggair import configure_logging

configure_logging(compression="gz")   # or "zip"
```

```bash
export LOGGAIR_COMPRESSION=gz          # env equivalent
```

```yaml
# loggair.yaml
compression: "zip"
```

The archive becomes `{script}.{timestamp}.log.gz` (or `.zip`); the active
`{script}.log` stays plain. Valid values are `"gz"`, `"zip"`, or unset (no
compression) — any other value raises `ValueError` at configure time.
`retention` accounts for compressed archives just like plain ones.

> **Note:** Because Loggair performs its *own* startup rotation (it renames the
> file rather than using loguru's managed rotation), compression is applied by
> Loggair right after the rename — **not** by loguru's sink-level `compression=`
> argument, which would never fire here.

### 7. Structured / JSON Logging

For containerized / cloud runs (Kubernetes, AWS ECS) whose logs are ingested by
an aggregator (Elasticsearch/Kibana, Grafana Loki, Datadog), switch both sinks
to structured JSON — one object per line — with `serialize`:

```python
from loggair import configure_logging

configure_logging(serialize=True)
```

```bash
export LOGGAIR_SERIALIZE=true          # env equivalent
```

```yaml
# loggair.yaml
serialize: true
```

Each line is loguru's native serialization (`serialize=True` on the sinks — no
bespoke formatter):

```json
{"text": "2026-07-04 15:26:39 | INFO     | svc.worker:run:42 | structured hello\n",
 "record": {"message": "structured hello", "level": {"name": "INFO", "no": 20},
            "time": {"repr": "2026-07-04 15:26:39.809+02:00", "timestamp": 1783171599.809},
            "name": "svc.worker", "function": "run", "line": 42,
            "process": {"id": 53369, "name": "MainProcess"},
            "extra": {"rank_tag": ""}, "...": "..."}}
```

Everything else composes unchanged: rank-zero console filtering, `[rank N]`
tags (as `record.extra.rank_tag`), `module_levels` per-sink thresholds,
rotation/retention, and `enqueue`. When `serialize` is on, console coloring is
**forced off** regardless of `colorize`/`NO_COLOR` — ANSI escapes would
otherwise be embedded inside the JSON `text` field.

### 8. Runtime Log Rotation (Long-Running Processes)

Startup rotation alone lets the active file grow unbounded in a process that
never restarts (multi-week training, FastAPI servers, MCP hosts). Set
`rotation` to also rotate **at runtime**, by size or time — the value is passed
to [loguru's `rotation=`](https://loguru.readthedocs.io/en/stable/api/logger.html#file)
verbatim, so all its forms work:

```yaml
# loggair.yaml
rotation: "100 MB"     # or "500 KB", "daily", "1 week", "00:00", "monday at 12:00"
```

```bash
export LOGGAIR_ROTATION="daily"        # env equivalent
```

```python
configure_logging(rotation="100 MB")   # programmatic
```

Invalid values raise `ValueError` at configure time. When runtime rotation is
active, `retention` and `compression` are additionally enforced **at rotation
time** by loguru's sink (same per-stem, live-log-safe semantics as the startup
sweep — verified against Loggair's own archives), so a service that never
restarts still gets pruning and compressed archives. Runtime archives are named
`{name}.{timestamp}_{microseconds}.log`; startup retention recognises both
naming schemes.

> **Multi-writer caveat:** rotation renames the active file, so with a shared
> file it runs in the **main process only**. Fork-inherited children route
> records through the enqueue queue and are unaffected, but a *spawn*-started
> worker that reconfigures its own sink keeps its file descriptor after a
> main-process rotation, so its subsequent records land in the rotated archive.
> **Setting [`worker_files: true`](#16-per-worker-log-files) removes this
> caveat entirely** — every process owns its own file and rotates it safely.

### 9. Retention Semantics (Shared Log Dirs Are Safe)

`retention` is enforced **per script name (stem)**, counting only *timestamped
archives* (`{name}.{timestamp}.log`, compressed or plain). On startup, Loggair
prunes each stem's archives — including stems left behind by earlier runs under
other script names — down to `retention`.

A bare `{name}.log` of **another** script is **never** deleted: in a shared log
directory (e.g. a centralized `~/logs`) it may be the live sink of a
concurrently running process. Only that script's own next run rotates it.

### 10. Embedding & Teardown

`shutdown_logging()` flushes pending records (sinks stay installed) — call it at
normal process exit. Hosts that need to switch Loggair **off** cleanly (pytest
plugins, notebooks, long-lived servers) use `reset_logging()`, which flushes and
removes all sinks, restores `warnings.showwarning` and the stdlib root handler
interception, un-patches the lazy-enqueue hooks, and returns Loggair to the
unconfigured state — a later `get_logger()` starts fresh:

```python
import loggair

loggair.reset_logging()  # full teardown; safe to reconfigure afterwards
```

### 11. Custom Log Formats

The per-sink format strings are configurable through the standard hierarchy —
`console_format` / `file_format` as arguments, `LOGGAIR_CONSOLE_FORMAT` /
`LOGGAIR_FILE_FORMAT` env vars, or YAML/`pyproject.toml` keys. Any
[loguru format field](https://loguru.readthedocs.io/en/stable/api/logger.html#record)
works — `{process.id}`, `{thread.id}`, `{extra[...]}`, color markup on the
console sink:

```yaml
# loggair.yaml — add process/thread IDs for multi-worker debugging
file_format: "{time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | pid={process.id} tid={thread.id} | {name}:{function}:{line} | {message}"
```

The defaults are public, so a custom format can **extend** rather than restate
them:

```python
from loggair import DEFAULT_FILE_FORMAT, configure_logging

configure_logging(file_format=DEFAULT_FILE_FORMAT + " | job={extra[job]}")
```

Notes:

- `{extra[rank_tag]}` is always stamped by Loggair's sink filters and available
  to any custom format; omitting it simply drops the rank tag from the layout.
- Invalid **color markup** fails fast at configure time. A format referencing a
  **nonexistent field** does not — loguru reports a loud per-record error on
  stderr and drops the record, so test a custom format once before deploying.
- In [JSON mode](#7-structured--json-logging) the format only shapes the `text`
  field; the structured `record` is unaffected.

### 12. Dynamic Level Adjustment at Runtime

Levels are resolved once at configure time, but a long-running job (multi-week
training, a FastAPI/MCP service) doesn't need a restart to change verbosity:

**Programmatic** — `reconfigure()` reloads the sinks with overrides, keeping
every explicitly-passed setting from the original call:

```python
import loggair

loggair.reconfigure(console_level="DEBUG")   # elevate a live run
loggair.reconfigure()                        # re-read env + config files
```

**Signal-driven** (POSIX) — for processes you can't call into, opt in to one or
both signals:

```yaml
# loggair.yaml
reload_signal: "SIGUSR1"   # re-resolve env + config files on signal
debug_signal:  "SIGUSR2"   # toggle DEBUG on both sinks on signal
```

```bash
# Diagnose a live anomaly: flip the running job to DEBUG, then back
kill -USR2 <pid>          # debug ON
kill -USR2 <pid>          # debug OFF (previous levels restored)

# Or edit loggair.yaml (levels, module_levels, ...) and apply it live
kill -USR1 <pid>
```

Details:

- Handlers are installed in the **main process and main thread only**; invalid
  or platform-unavailable signal names raise `ValueError` at configure time.
  The signal handler defers all work to a short-lived worker thread — safe to
  deliver at any moment, even mid-log-call.
- A reload re-applies the original call's explicit arguments, so it only picks
  up changes for settings driven by env vars or config files. Drive levels via
  `loggair.yaml` if you want them signal-reloadable.
- Each toggle/reload announces itself in the log
  (`Loggair debug mode ON (signal)` / `Loggair Re-initialized`).

### 13. Config Introspection

`get_active_config()` returns the **resolved** settings the sinks are actually
running with — all hierarchy layers applied — as a JSON-serializable dict, for
third-party plugins, diagnostics scripts, or an MCP tool surface:

```python
import json, loggair

print(json.dumps(loggair.get_active_config(), indent=2))
# {
#   "configured": true,
#   "log_dir": "/home/me/logs",
#   "log_file": "/home/me/logs/train.log",
#   "file_level": "DEBUG", "console_level": "INFO",
#   "module_levels": {"pkg.noisy": {"file": "WARNING"}},
#   "rotation_on_startup": true, "retention": 5, "compression": "gz",
#   "rotation": "100 MB", "enqueue": true, "enqueue_active": false,
#   "colorize": null, "serialize": false,
#   "console_format": "...", "file_format": "...",
#   "reload_signal": "SIGUSR1", "debug_signal": null,
#   "debug_mode_active": false, "is_main_process": true, "rank": null
# }
```

It is purely read-only (never triggers the lazy configuration — before the
first configure it returns `{"configured": false}`) and hands out a fresh copy
each call. `enqueue` vs `enqueue_active` distinguishes the *requested* queue
from the lazily-materialized one; `debug_mode_active` reflects the
[debug-signal toggle](#12-dynamic-level-adjustment-at-runtime).

### 14. Experiment Context Injection

Stamp the training loop's current position once, and **every** record logged
anywhere in the process afterwards carries it — Loggair loggers and
intercepted third-party stdlib logging alike:

```python
import loggair

for epoch in range(epochs):
    loggair.set_context(epoch=epoch)
    for step, batch in enumerate(loader):
        if step % 100 == 0:
            loggair.set_context(step=step)      # merges — epoch stays
        ...
# 2026-07-04 17:01:41.698 | INFO | train:run:42 | epoch=3 step=1200 | loss improved
```

The API: `set_context(**fields)` merges, `clear_context("step")` removes one
field, `clear_context()` removes all, `get_context()` inspects, and
`with loggair.context(phase="validation"): ...` scopes fields to a block
(the entry snapshot is restored on exit). Behavior:

- The default formats render a compact `epoch=3 step=1200 | ` tag before the
  message (nothing at all when no context is set). Custom formats can place it
  via `{extra[context_tag]}`.
- In [JSON mode](#7-structured--json-logging) each field also lands
  individually in `record.extra` (`extra.epoch`, `extra.step`, ...) for log
  aggregators. An explicit `logger.bind(epoch=...)` wins over the ambient
  context.
- The context is **process-wide** (a checkpoint-saver thread's records carry
  the epoch the training thread set) and inherited by fork-started workers;
  spawn-started workers begin empty. Reads are lock-free — near-zero
  per-record cost.

### 15. Webhook & Alerting Sinks

Route high-severity records to external platforms so operators hear about
crashes immediately. Delivery is backed by
[Apprise](https://github.com/caronc/apprise) (optional dependency):

```bash
pip install "loggair[alerts]"      # pulls apprise
```

Configure one or more Apprise URLs — Slack, Teams, Discord, Telegram, email,
generic webhooks, [~100 services](https://github.com/caronc/apprise/wiki):

```yaml
# loggair.yaml
alert_urls:
  - "slack://TokenA/TokenB/TokenC/#alerts"
  - "mailto://user:pass@gmail.com"
alert_level: "ERROR"      # minimum severity that alerts (default ERROR)
alert_throttle: 60        # at most one delivery per N seconds (0 = every batch)
```

```bash
export LOGGAIR_ALERT_URLS="slack://TokenA/TokenB/TokenC/#alerts,mailto://ops@corp.com"
```

Behavior:

- **Never blocks the hot path**: the sink only enqueues; a background worker
  delivers. Everything queued since the last delivery is **batched into one
  notification** (title carries the highest severity + host + a `(+N more)`
  count), and `alert_throttle` rate-limits deliveries — an alert storm
  collapses into periodic digests instead of thousands of messages.
- Tracebacks from `logger.exception(...)` are included in the body.
- Invalid URLs and unknown levels **fail fast at configure time**; a missing
  `apprise` install raises with the `pip install loggair[alerts]` hint.
- A failing webhook is reported at WARNING in the normal sinks but can never
  feed back into another alert (internally-marked and filtered).
- `shutdown_logging()` flushes queued alerts; `reset_logging()` stops the
  worker. `get_active_config()` reports the URLs in Apprise's
  **privacy-redacted** form — tokens never leak into diagnostics.
- The sink is installed in **every** process, so a worker/rank crash alerts
  too; the per-process throttle keeps distributed storms bounded.

### 16. Per-Worker Log Files

By default all processes append to the shared `{name}.log` (rank-tagged). Set
`worker_files: true` (env `LOGGAIR_WORKER_FILES`) to give every non-main
writer its **own, exclusively-owned** file instead:

| Writer | File |
| --- | --- |
| main process, rank 0 / non-distributed | `app.log` |
| DDP rank 2 | `app.rank2.log` |
| spawned/forked child `Worker-1` | `app.worker-1.log` |
| rank 2's own child `Worker-1` | `app.rank2.worker-1.log` |

Because each process is the sole writer of its path, the multi-writer
restrictions disappear: per-worker files get **startup rotation and
[runtime rotation](#8-runtime-log-rotation-long-running-processes)** too, and
no rotation can ever split another process's stream. The
[startup sweep](#9-retention-semantics-shared-log-dirs-are-safe) still never
deletes the bare per-worker files (live-log-safe rule) — only their
timestamped archives, pruned per stem as usual.

Merge the streams at read time — [lnav](https://lnav.org/) interleaves the
whole directory chronologically:

```bash
lnav ./logs        # app.log + app.rank*.log + app.*worker*.log, one timeline
```

### 17. Interception Modes

By default (`intercept: full`) Loggair **owns** stdlib logging: the root
handler is replaced and existing named loggers are stripped so everything
propagates into Loggair. Two knobs relax this for embedding:

```yaml
# loggair.yaml
intercept: "coexist"              # full | coexist | off
intercept_exclude: ["uvicorn"]    # dotted prefixes Loggair must not touch
capture_warnings: true            # warnings.showwarning redirect (default on)
```

- **`coexist`** — Loggair *appends* its handler to the root and touches
  nothing else: a framework that configured its own logging (uvicorn,
  gunicorn) keeps its handlers, and both receive records. Trade-off: the root
  level is lowered to DEBUG so records reach Loggair (pre-existing handlers
  still apply their own levels), and a logger with `propagate=False` stays
  outside Loggair.
- **`intercept_exclude`** — even in `full` mode, the listed prefixes (and
  their children, e.g. `uvicorn` covers `uvicorn.error`) keep their handlers,
  `propagate` flags, and levels verbatim; the built-in third-party silences
  also skip them.
- **`off`** — no interception at all: no root rewiring, no warnings redirect,
  no third-party level defaults. Only code calling `loggair.get_logger()`
  directly is captured.
- **`capture_warnings: false`** — leave `warnings.showwarning` alone
  (independently of the mode).

An invalid `intercept` value raises `ValueError` at configure time.

## Log Inspection
For the best experience viewing Loggair logs (especially interleaving logs from multiple ranks/workers), we recommend using **[lnav](https://lnav.org/)** (The Log File Navigator).

`lnav` automatically detects Loggair's timestamp format and can merge multiple log files into a single, chronological view.

### Usage with lnav
```bash
# View all logs in the directory interleaved by time
lnav ./logs
```

For more information, see the **[lnav documentation](https://docs.lnav.org/)**.

## Distributed Training (DDP/SLURM)
Loggair handles ranks automatically. No need to wrap your log calls in `if rank == 0:`.
```python
# In a torchrun or SLURM environment
from loggair import get_logger

logger = get_logger(__name__)

# Only shows up once in console (Rank 0), but saved in file for all Ranks
logger.info("Initializing process group...")
```

## License
MIT
