Metadata-Version: 2.4
Name: promcsv
Version: 1.4.2
Summary: Scrape Prometheus endpoints and export per-target CSV files, with optional upload to S3 or SFTP
Author-email: Benny Brit <benny.brit@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/bennybrit/promcsv
Project-URL: Repository, https://github.com/bennybrit/promcsv.git
Keywords: prometheus,metrics,csv,export,exporter,s3,sftp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: No Input/Output (Daemon)
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking :: Monitoring
Classifier: Topic :: Utilities
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25
Requires-Dist: PyYAML>=5.4
Provides-Extra: s3
Requires-Dist: boto3>=1.26; extra == "s3"
Provides-Extra: sftp
Requires-Dist: paramiko>=3.4; extra == "sftp"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Dynamic: license-file

# promcsv

[![CI](https://github.com/bennybrit/promcsv/actions/workflows/ci.yml/badge.svg)](https://github.com/bennybrit/promcsv/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/promcsv)](https://pypi.org/project/promcsv/)
[![Python versions](https://img.shields.io/pypi/pyversions/promcsv)](https://pypi.org/project/promcsv/)
[![License](https://img.shields.io/badge/license-Apache--2.0-green)](https://github.com/bennybrit/promcsv/blob/v1.4.2/LICENSE)

Turn Prometheus metrics into CSV files - and deliver them to S3 or SFTP.

```sh
pip install promcsv
```

## What it does

promcsv turns Prometheus metrics into CSV files for systems that do not speak
Prometheus. It scrapes a configured set of `/metrics` endpoints on a fixed
interval and writes one CSV file per target per cycle - target outputs are
never combined. Files land atomically in a `ready/` directory, are optionally
gzip-compressed and uploaded to S3 or SFTP, and are cleaned up by a retention
sweep so disk usage stays bounded.

It is built to run unattended for months as a systemd service, but it runs
just as well in the foreground or from cron.

## Supported features

- **One CSV per target per cycle** with a fixed 5-column schema
  (`timestamp,metric,type,labels,value`) that never changes shape
- **Wall-clock-aligned scraping** (a 5m interval fires at :00, :05, :10, ...),
  targets scraped concurrently, one failing target never affects the others
- **Optional gzip compression** (`.csv.gz`), applied once at write time
- **Upload to S3** (including S3-compatible stores like MinIO; SSE/KMS and
  storage class supported) **or SFTP** (key or password auth, strict host-key
  verification), **or no upload** - consumers pick files up from `ready/`
- **Flat or per-target remote layout**, one config switch
- **Atomic file handling everywhere** - no partial files, local or remote
- **Automatic retention cleanup**; upload outages self-recover by draining
  the backlog once the endpoint returns
- **`status.json` health file** for external monitoring and alerting
- **systemd-native**: `Type=notify` watchdog, hardened unit, journald logging
- **Strict config validation** with did-you-mean hints (`--validate-config`)
- **Small footprint**: one RPM or wheel; only `requests` and `PyYAML` required
  (boto3/paramiko only for the upload target you actually use)

## Install

Two options - pick one, do not mix them on one host:

- **Option 1 - RPM** (Rocky/RHEL 8+): recommended for servers; one command
  installs everything, including the systemd service.
- **Option 2 - pip** (any Linux): from PyPI or a wheel file; systemd setup is
  a short manual step.

### Option 1 - install from RPM (Rocky/RHEL 8+, recommended)

Download the RPM from the
[latest release](https://github.com/bennybrit/promcsv/releases), then:

```sh
dnf install ./python3.12-promcsv-<version>-1.el8.noarch.rpm
vi /etc/promcsv/config.yaml
promcsv -c /etc/promcsv/config.yaml --validate-config
systemctl enable --now promcsv
```

For S3 or SFTP upload, also install the extra package (it has no python3.12
RPM): `python3.12 -m pip install boto3` (S3) or `paramiko` (SFTP).

### Option 2 - install with pip (any Linux)

Prerequisite: Python 3.12 or newer. On Rocky/RHEL 8:

```sh
dnf install python3.12 python3.12-pip
```

Install **as root** (system-wide), straight from PyPI:

```sh
sudo python3.12 -m pip install promcsv
# S3 / SFTP upload extras:
sudo python3.12 -m pip install boto3      # when upload.target is s3
sudo python3.12 -m pip install paramiko   # when upload.target is sftp
```

Offline hosts: download the wheel from the
[latest release](https://github.com/bennybrit/promcsv/releases) and
`sudo python3.12 -m pip install ./promcsv-<version>-py3-none-any.whl`.

Note: on hosts with a hardened root umask (`0027`/`0077`) pip creates
unreadable package directories and other users then get
`ModuleNotFoundError` - install with
`sudo sh -c 'umask 022 && python3.12 -m pip install promcsv'`.

Create the configuration from the built-in example and validate it:

```sh
install -d -m 0755 /etc/promcsv
promcsv --print-config > /etc/promcsv/config.yaml    # then edit
promcsv -c /etc/promcsv/config.yaml --validate-config
```

The tool is now fully usable from the command line
(`promcsv -c /etc/promcsv/config.yaml`, or `--once` from cron). To run it as
a systemd service, complete the one-time setup below - the RPM path does all
of this automatically.

### Run as a systemd service (pip installs)

One-time host setup:

```sh
# service user and directories
useradd -r -s /sbin/nologin promcsv
install -d -o promcsv -g promcsv /var/data/promcsv
chgrp promcsv /etc/promcsv /etc/promcsv/config.yaml
chmod 0750 /etc/promcsv; chmod 0640 /etc/promcsv/config.yaml

# verify the service user can load the package (catches permission problems)
sudo -u promcsv /usr/bin/python3.12 -c "import promcsv"   # must print nothing

# unit, log rotation and credentials template
promcsv --print-unit > /usr/lib/systemd/system/promcsv.service
promcsv --print-logrotate > /etc/logrotate.d/promcsv
promcsv --print-env > /etc/promcsv/env && chmod 0600 /etc/promcsv/env
systemctl daemon-reload
systemctl enable --now promcsv
```

The printed unit's `ExecStart=` assumes `/usr/local/bin/promcsv`; if
`command -v promcsv` shows a different path (e.g. a venv), edit `ExecStart=`
in the installed unit accordingly.

## CLI

```
promcsv -c /etc/promcsv/config.yaml [--validate-config | --once]
```

- `-c, --config FILE` - path to the YAML config (required).
- `--validate-config` - load and validate the config, print a report, and
  exit 0 (valid) or 2 (invalid). Read-only; touches nothing.
- `--once` - run a single scrape cycle and exit: 0 if at least one target was
  scraped successfully, 1 otherwise. Cron fallback / smoke test.
- `--print-config` - print the example configuration and exit.
- `--print-unit` - print the systemd unit and exit. The printed `ExecStart=`
  assumes `/usr/local/bin/promcsv`; if `command -v promcsv` shows a different
  path (e.g. a venv), edit `ExecStart=` accordingly.
- `--print-logrotate` - print the logrotate snippet and exit.
- `--print-env` - print the `/etc/promcsv/env` template and exit.
- `--version` - print version and exit.

The `--print-*` flags need no configuration file and are mutually exclusive
with each other and with `--once`/`--validate-config`.

Exit codes:

| Code | Meaning |
|------|---------|
| 0    | clean shutdown / successful `--once` cycle |
| 1    | runtime fatal (lock held, stuck scrape threads, unhandled error) |
| 2    | configuration error |

The unit sets `RestartPreventExitStatus=2`: a broken config exits 2 and is
**not** restarted, so systemd does not loop on an error no retry can fix.
Runtime failures exit 1 and are restarted after `RestartSec`.

## Documentation

- [Usage guide](https://github.com/bennybrit/promcsv/blob/v1.4.2/docs/USAGE.md) -
  the CSV format specification for consumers, the directory lifecycle, S3 and
  SFTP upload setup, monitoring and alerting, and operational best practices.
- [Annotated example configuration](https://github.com/bennybrit/promcsv/blob/v1.4.2/src/promcsv/data/config.example.yaml) -
  every parameter documented in place (also available via
  `promcsv --print-config`).

## Build

### Build the RPM (Rocky/RHEL 8+)

One-time build-host prerequisites:

```sh
dnf install rpm-build python3.12-devel python3.12-pip \
            python3.12-setuptools python3.12-wheel systemd
python3.12 -m pip install build
```

Then:

```sh
./packaging/build-rpm.sh
```

The artifacts are written to `dist/`:
`python3.12-promcsv-<version>-1.el8.noarch.rpm` (install this) and the
matching `.src.rpm` (for rebuilding on other EL releases).

### Build the wheel

No prerequisites beyond Python 3.12 - no venv, no test dependencies:

```sh
python3.12 -m pip install build
python3.12 -m build --wheel
```

The artifact is written to `dist/`:
`promcsv-<version>-py3-none-any.whl` - that single file is what you copy to
the target host.

## Development

Running the tests requires a one-time setup (a virtualenv with the package and
the test tools):

```sh
python3.12 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
```

Then:

```sh
pytest -q -m "not slow"      # default run (fast, skips the soak test)
pytest -q                    # everything, including the slow soak test
pytest -q -m "not slow" --cov=promcsv --cov-branch --cov-report=term-missing   # with coverage
```

To build the RPM or the wheel, see the Build section - neither needs the
virtualenv or the test dependencies.

### Releasing

Releases are fully automated by GitHub Actions on tag push:

```sh
# 1. bump the version in pyproject.toml, src/promcsv/__init__.py and the RPM
#    spec (new changelog entry), and update the two tag-pinned README links -
#    the test suite fails if any of these is missed
# 2. then:
git commit -am "promcsv vX.Y.Z: ..."
git tag vX.Y.Z
git push origin main vX.Y.Z
```

The release workflow verifies the tag (matches the version, commit is on
main), runs the full test suite, builds the wheel/sdist and the RPM (in a
Rocky Linux 8 container), publishes to PyPI via Trusted Publishing (after the
required environment approval), and creates the GitHub release with the wheel
and RPM attached. The manual Build instructions above remain as a fallback.
