Metadata-Version: 2.4
Name: promcsv
Version: 1.4.0
Summary: Scrape Prometheus endpoints and export per-target CSV files, with optional upload to S3 or SFTP
Author-email: Benny Brit <benny.brit@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/bennybrit/promcsv.git
Project-URL: Repository, https://github.com/bennybrit/promcsv.git
Keywords: prometheus,metrics,csv,export,exporter,s3,sftp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: No Input/Output (Daemon)
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking :: Monitoring
Classifier: Topic :: Utilities
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25
Requires-Dist: PyYAML>=5.4
Provides-Extra: s3
Requires-Dist: boto3>=1.26; extra == "s3"
Provides-Extra: sftp
Requires-Dist: paramiko>=3.4; extra == "sftp"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Dynamic: license-file

# promcsv

## What it does

promcsv turns Prometheus metrics into CSV files for systems that do not speak
Prometheus. It scrapes a configured set of `/metrics` endpoints on a fixed
interval and writes one CSV file per target per cycle - target outputs are
never combined. Files land atomically in a `ready/` directory, are optionally
gzip-compressed and uploaded to S3 or SFTP, and are cleaned up by a retention
sweep so disk usage stays bounded.

It is built to run unattended for months as a systemd service, but it runs
just as well in the foreground or from cron.

## Supported features

- **One CSV per target per cycle** with a fixed 5-column schema
  (`timestamp,metric,type,labels,value`) that never changes shape
- **Wall-clock-aligned scraping** (a 5m interval fires at :00, :05, :10, ...),
  targets scraped concurrently, one failing target never affects the others
- **Optional gzip compression** (`.csv.gz`), applied once at write time
- **Upload to S3** (including S3-compatible stores like MinIO; SSE/KMS and
  storage class supported) **or SFTP** (key or password auth, strict host-key
  verification), **or no upload** - consumers pick files up from `ready/`
- **Flat or per-target remote layout**, one config switch
- **Atomic file handling everywhere** - no partial files, local or remote
- **Automatic retention cleanup**; upload outages self-recover by draining
  the backlog once the endpoint returns
- **`status.json` health file** for external monitoring and alerting
- **systemd-native**: `Type=notify` watchdog, hardened unit, journald logging
- **Strict config validation** with did-you-mean hints (`--validate-config`)
- **Small footprint**: one RPM or wheel; only `requests` and `PyYAML` required
  (boto3/paramiko only for the upload target you actually use)

## Build

### Build the RPM (Rocky/RHEL 8+)

One-time build-host prerequisites:

```sh
dnf install rpm-build python3.12-devel python3.12-pip \
            python3.12-setuptools python3.12-wheel systemd
python3.12 -m pip install build
```

Then:

```sh
./packaging/build-rpm.sh
```

The artifacts are written to `dist/`:
`python3.12-promcsv-<version>-1.el8.noarch.rpm` (install this) and the
matching `.src.rpm` (for rebuilding on other EL releases).

### Build the wheel

No prerequisites beyond Python 3.12 - no venv, no test dependencies:

```sh
python3.12 -m pip install build
python3.12 -m build --wheel
```

The artifact is written to `dist/`:
`promcsv-<version>-py3-none-any.whl` - that single file is what you copy to
the target host.

## Install

Two supported paths: the RPM (recommended on Rocky/RHEL 8+) or the wheel
(any Linux). Do not mix them on one host.

### Install from RPM (Rocky/RHEL 8+, recommended)

```sh
dnf install ./python3.12-promcsv-<version>-1.el8.noarch.rpm
vi /etc/promcsv/config.yaml
promcsv -c /etc/promcsv/config.yaml --validate-config
systemctl enable --now promcsv
```

For S3 or SFTP upload, also install the extra package (it has no python3.12
RPM): `python3.12 -m pip install boto3` (S3) or `paramiko` (SFTP).

### Install from wheel (any Linux)

Prerequisite: Python 3.12 or newer on the host. On Rocky/RHEL 8:

```sh
dnf install python3.12 python3.12-pip
```

Copy the wheel over and install it **as root** (system-wide):

```sh
sudo python3.12 -m pip install promcsv-*.whl
# When upload.target is s3, also install boto3:
sudo python3.12 -m pip install boto3
# When upload.target is sftp, also install paramiko:
sudo python3.12 -m pip install paramiko
```

On hosts with a hardened root umask (`0027`/`0077`), pip creates the package
directories without world-read, and other users then cannot import the
package (`ModuleNotFoundError: No module named 'promcsv'`). Install with a
safe umask:

```sh
sudo sh -c 'umask 022 && python3.12 -m pip install promcsv-*.whl'
```

Create the configuration from the built-in example and validate it (no git
checkout needed - the files ship inside the package):

```sh
install -d -m 0755 /etc/promcsv
promcsv --print-config > /etc/promcsv/config.yaml    # then edit
promcsv -c /etc/promcsv/config.yaml --validate-config
```

At this point the tool is fully usable from the command line
(`promcsv -c /etc/promcsv/config.yaml`, or `--once` from cron). To run it as
a systemd service, complete the one-time setup below - the RPM path does all
of this automatically.

### Run as a systemd service (wheel installs only)

One-time host setup:

```sh
# service user and directories
useradd -r -s /sbin/nologin promcsv
install -d -o promcsv -g promcsv /var/data/promcsv
chgrp promcsv /etc/promcsv /etc/promcsv/config.yaml
chmod 0750 /etc/promcsv; chmod 0640 /etc/promcsv/config.yaml

# verify the service user can load the package (catches permission problems)
sudo -u promcsv /usr/bin/python3.12 -c "import promcsv"   # must print nothing

# unit, log rotation and credentials template
promcsv --print-unit > /usr/lib/systemd/system/promcsv.service
promcsv --print-logrotate > /etc/logrotate.d/promcsv
promcsv --print-env > /etc/promcsv/env && chmod 0600 /etc/promcsv/env
systemctl daemon-reload
systemctl enable --now promcsv
```

The printed unit's `ExecStart=` assumes `/usr/local/bin/promcsv`; if
`command -v promcsv` shows a different path (e.g. a venv), edit `ExecStart=`
in the installed unit accordingly.

## CLI

```
promcsv -c /etc/promcsv/config.yaml [--validate-config | --once]
```

- `-c, --config FILE` - path to the YAML config (required).
- `--validate-config` - load and validate the config, print a report, and
  exit 0 (valid) or 2 (invalid). Read-only; touches nothing.
- `--once` - run a single scrape cycle and exit: 0 if at least one target was
  scraped successfully, 1 otherwise. Cron fallback / smoke test.
- `--print-config` - print the example configuration and exit.
- `--print-unit` - print the systemd unit and exit. The printed `ExecStart=`
  assumes `/usr/local/bin/promcsv`; if `command -v promcsv` shows a different
  path (e.g. a venv), edit `ExecStart=` accordingly.
- `--print-logrotate` - print the logrotate snippet and exit.
- `--print-env` - print the `/etc/promcsv/env` template and exit.
- `--version` - print version and exit.

The `--print-*` flags need no configuration file and are mutually exclusive
with each other and with `--once`/`--validate-config`.

Exit codes:

| Code | Meaning |
|------|---------|
| 0    | clean shutdown / successful `--once` cycle |
| 1    | runtime fatal (lock held, stuck scrape threads, unhandled error) |
| 2    | configuration error |

The unit sets `RestartPreventExitStatus=2`: a broken config exits 2 and is
**not** restarted, so systemd does not loop on an error no retry can fix.
Runtime failures exit 1 and are restarted after `RestartSec`.

## CSV format (for consumers)

Every file starts with the header:

```
timestamp,metric,type,labels,value
```

Example:

```csv
timestamp,metric,type,labels,value
2026-07-21T10:15:00Z,connections_active,gauge,,412
2026-07-21T10:15:00Z,messages_total,counter,direction=tx;interface=eth0,908311
2026-07-21T10:15:00Z,request_duration_seconds_bucket,histogram,le=0.5,17734
2026-07-21T10:15:00Z,request_duration_seconds_sum,histogram,,5401.25
2026-07-21T10:15:00Z,setup_seconds,summary,quantile=0.99,0.087
```

Rules:

- **timestamp** - the cycle's scrape time in UTC, identical for every row in a
  file. Format per `csv.timestamp_format`: `iso8601`
  (`2026-07-21T10:15:00Z`) or `epoch` (integer Unix seconds).
- **metric** - the sample name exactly as exposed (e.g. `foo_bucket`,
  `foo_sum`).
- **type** - from the endpoint's `# TYPE` metadata: `counter`, `gauge`,
  `histogram`, `summary` or `untyped`. Child samples (`_bucket`, `_sum`,
  `_count`, `_created`) inherit their family's type. Samples without TYPE
  metadata are `untyped`.
- **labels** - `key=value;key=value`, sorted by key; empty when the sample has
  no labels. Within label *values*, backslash, `;` and `=` are
  backslash-escaped (`\\`, `\;`, `\=`).
- **CSV quoting** - standard RFC 4180: fields containing the delimiter, `"` or
  newlines are quoted, embedded quotes doubled. Consume the files with a real
  CSV parser, or configure a delimiter that cannot appear in your data.
- **value** - floats with an integral value are rendered as bare integers
  (up to 2^53 in magnitude); `+Inf`, `-Inf` and `NaN` are passed through
  literally.

Filenames are `<label>_<YYYYMMDDTHHMMSSZ>.csv` (UTC cycle time) - `.csv.gz`
with compression on. On the rare name collision a `_1`, `_2`, ... suffix is
inserted before the full extension (`app_..._1.csv`, `app_..._1.csv.gz`).

### Compression

`csv.compress: gzip` writes `.csv.gz` files instead of `.csv`. The gzip stream
is deterministic (mtime=0 in the header), so identical content produces
identical bytes. Consumers use any gzip-capable reader (`zcat`, Python
`gzip`, ...). Flipping the flag between restarts is safe - `ready/` may
transiently hold both extensions, and upload and retention handle both. The
remote object keeps the exact local name and bytes; nothing is re-compressed
or renamed on upload.

## Directory lifecycle

`output_dir` (default layout under `/var/data/promcsv`):

```
output_dir/
├── tmp/         in-progress files (*.part); purged at startup
├── ready/       completed CSVs awaiting upload or pickup
├── uploaded/    CSVs successfully uploaded
├── status.json  machine-readable health, rewritten atomically each cycle
└── .lock        single-instance flock (never deleted)
```

Flow: each CSV is written to `tmp/`, fsynced, then atomically renamed into
`ready/` - readers never see a partial file. When an upload succeeds, the file
moves from `ready/` to `uploaded/`. The retention sweep deletes files older
than `retention` from **both** `ready/` and `uploaded/`; deletion from `ready/`
is the disk-safety bound (it is logged as a WARNING when an uploader is
configured, since it means undelivered data was lost). With
`upload.target: none`, `ready/` is the hand-off point: consumers collect files
from there themselves before retention removes them.

## S3 upload support

Requires boto3 (`sudo python3.12 -m pip install boto3`). Config:

```yaml
upload:
  target: s3
  s3:
    bucket: my-metrics-bucket
    prefix: metrics/                   # optional; object keys become <prefix><filename>
    region: eu-west-1
    # endpoint_url: http://minio:9000  # optional: non-AWS S3-compatible storage (region then optional)
    # sse: aws:kms                     # optional server-side encryption: AES256 | aws:kms
    # kms_key_id: <arn-or-id>          # optional; requires sse: aws:kms
    # storage_class: STANDARD_IA       # optional; passed through verbatim
```

Credentials never go in the YAML - the S3 client uses the standard boto3
credential chain: environment variables, `~/.aws/credentials`, or an instance
profile / IAM role. For environment variables, uncomment the placeholders in
`/etc/promcsv/env` (installed by the RPM; on wheel installs create it with
`promcsv --print-env`; keep it mode 0600, root-owned):

```sh
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
```

The unit already wires it in via `EnvironmentFile=-/etc/promcsv/env` (the `-`
makes the file optional). For non-AWS S3-compatible storage (MinIO, Ceph RGW,
...), set `upload.s3.endpoint_url`; `region` then becomes optional.

The uploaded object keeps the exact local name and bytes (`ContentType` is
`text/csv`, or `application/gzip` for compressed files). Pruning the remote
side is the consumer's responsibility - at 6 targets on a 5m interval the
prefix grows by ~1,700 files/day.

Retry semantics: connection failures, HTTP 5xx and throttling abort the upload
pass and retry next cycle; per-file permanent errors (AccessDenied, invalid
KMS key) are skipped and surface as `permanent_failures_pending` in
`status.json`.

## SFTP upload support

Requires paramiko (`sudo python3.12 -m pip install paramiko`). Config:

```yaml
upload:
  target: sftp
  sftp:
    host: sftp.example.com
    port: 22                                  # optional, default 22
    username: metrics
    key_file: /etc/promcsv/sftp_key           # private key auth (recommended)
    remote_dir: /upload/metrics               # files land in <remote_dir>
    known_hosts: /etc/promcsv/known_hosts     # strict host-key verification (required)
```

Key setup (then install `sftp_key.pub` on the server):

```sh
ssh-keygen -t ed25519 -f /etc/promcsv/sftp_key -N '' -C promcsv
chown root:promcsv /etc/promcsv/sftp_key && chmod 0640 /etc/promcsv/sftp_key
```

Host key verification is strict; capture the server's key into `known_hosts`:

```sh
ssh-keyscan -p 22 sftp.example.com > /etc/promcsv/known_hosts
```

**Warning - trust on first use.** `ssh-keyscan` records whatever key answered
at that moment. Verify the fingerprint out-of-band with the server owner
(`ssh-keygen -lf /etc/promcsv/known_hosts`) before trusting it - strict
verification is only as strong as this first capture.

Password auth alternative: omit `key_file` and uncomment
`PROMCSV_SFTP_PASSWORD=...` in `/etc/promcsv/env` (installed by the RPM; on
wheel installs create it with `promcsv --print-env`; mode 0600, root-owned;
systemd's `EnvironmentFile` delivers it) - never in the YAML.

The uploaded file keeps the exact local name and bytes. Each upload is staged
as `<filename>.part` and atomically renamed on completion, so the server never
exposes a partial file. `remote_dir: /` is supported for chrooted SFTP
accounts. Pruning the remote side is the consumer's responsibility (see the S3
section for the growth math).

Retry semantics: connection failures (host unreachable, authentication,
host-key mismatch) abort the upload pass and retry next cycle; per-file
permanent errors (remote permissions, quota) are skipped and surface as
`permanent_failures_pending` in `status.json`.

## Monitoring and alerting

`<output_dir>/status.json` is rewritten atomically after every cycle:

| Field | Meaning |
|-------|---------|
| `version` | promcsv version |
| `state` | `running` or `stopped`; written `running` at startup and every cycle, `stopped` by a clean shutdown. A crash/SIGKILL leaves `running` behind - that is the signal that the process was NOT stopped properly |
| `started_at` | UTC start time of the current (or last) process |
| `stopped_at` | UTC time of the clean shutdown, `null` while running |
| `stop_reason` | `null` while running; else e.g. `signal SIGTERM`, `once complete`, `fatal: ...` |
| `last_cycle` | UTC timestamp of the last completed cycle (`null` before the first cycle of this process) |
| `last_cycle_duration_seconds` | wall time of that cycle |
| `targets.<label>.last_success` | UTC time of the target's last successful scrape (`null` if never) |
| `targets.<label>.consecutive_failures` | failed cycles in a row for this target (0 = healthy) |
| `upload` | `null` when `upload.target` is `none`, else: |
| `upload.last_success` | UTC time of the last successful upload |
| `upload.consecutive_retryable_failures` | upload passes in a row aborted by a retryable error (endpoint down, throttling) |
| `upload.permanent_failures_pending` | files in `ready/` whose upload fails permanently (e.g. AccessDenied) |
| `upload.last_failure` | `{kind, detail, at}` of the most recent failed upload, or `null`; set on every failed upload, cleared on the next success |
| `ready_backlog` | CSV files currently waiting in `ready/` |
| `uploaded_this_cycle` | files uploaded during the last cycle |

Staleness rule: define staleness as `now - max(last_cycle, started_at)`
exceeding your threshold (treat a `null` `last_cycle` as absent) - a freshly
restarted daemon has `last_cycle: null` for the whole first alignment wait,
and a naive `last_cycle`-only check would page on every restart.

Lifecycle decision table: **stale + `state: running`** -> the daemon died or
hung -> page. **`state: stopped`** -> deliberate shutdown (or between `--once`
cron runs) -> suppress the staleness page; apply your "service down" policy
instead. The file is never deleted - the last-known state is kept for
forensics.

Suggested alerts: staleness per the rule above (daemon stalled or down),
`ready_backlog` growing over time (uploads or consumers falling behind), any
`consecutive_failures` or `permanent_failures_pending` above 0 for sustained
periods.

Upload alerting rule: if `upload.consecutive_retryable_failures` keeps
climbing **and** `upload.last_failure.kind` is one of
`AuthenticationException`, `BadHostKeyException` (SFTP) or `AccessDenied`,
`InvalidAccessKeyId`, `SignatureDoesNotMatch` (S3), page a human - these are
credential, host-key or bucket-policy problems and never self-recover. Other
retryable kinds are self-recovering outages (endpoint down, throttling); the
backlog drains on its own once the endpoint returns.

Logging: journald by default (`journalctl -u promcsv`). Errors are logged on
state change - first failure of a target/upload, then a reminder every 10th
consecutive failing cycle, then a recovery line - plus exactly one INFO
summary line per cycle
(`cycle: targets 6/6, rows 5321, written 6, uploaded 6, backlog 0, 1.2s`).

Watchdog: the unit sets `WatchdogSec=900`. Watchdog pings are time-driven
(sent on a fixed cadence during scraping, uploading and idle waits), so the
900-second deadline works with **any** `scrape_interval` - no tuning needed.

## Operational notes and best practices

- **Validate before every restart.** The config is read once at startup;
  changes require `systemctl restart promcsv`. Run
  `promcsv -c ... --validate-config` first - a broken config exits 2 and the
  service stays down until fixed (deliberately, to avoid a restart loop).
- **Smoke-test with `--once`.** After installing or changing the config, one
  `--once` run against a scratch `output_dir` shows exactly what will be
  produced, without touching the service.
- **Single instance per output_dir.** An exclusive lock on
  `<output_dir>/.lock` guards the directory. Running `promcsv --once` beside
  the daemon exits 1 with a clear message - by design; point it at a different
  `output_dir` if you need a parallel run.
- **Target isolation.** Targets are scraped concurrently; one target being
  down (or slow, or hung) never affects the others' CSVs. A scrape thread
  stuck beyond the timeout is abandoned; if stuck threads persist, the daemon
  exits 1 for a clean systemd restart.
- **Upload outages self-recover.** Files stay in `ready/` and the backlog
  drains automatically once the endpoint returns - time-boxed to half the
  scrape interval per cycle, in filename order (per-target oldest first), so
  scraping cadence is never starved.
- **Permanent upload errors** (S3: AccessDenied, invalid KMS key; SFTP: remote
  permissions, quota) are skipped and retried each cycle until retention
  removes the file; they are visible as `upload.permanent_failures_pending` in
  `status.json` and in the logs.
- **Upgrade the RPM, never reinstall it.** `dnf upgrade` preserves your edited
  `/etc/promcsv/config.yaml` and `env` (they are `%config(noreplace)`), but
  `dnf reinstall` of the same version resets them to the packaged defaults and
  moves your files to `.rpmsave` - standard RPM behavior, easy to trip over.

## Development

Running the tests requires a one-time setup (a virtualenv with the package and
the test tools):

```sh
python3.12 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
```

Then:

```sh
pytest -q -m "not slow"      # default run (fast, skips the soak test)
pytest -q                    # everything, including the slow soak test
pytest -q -m "not slow" --cov=promcsv --cov-branch --cov-report=term-missing   # with coverage
```

To build the RPM or the wheel, see the Build section - neither needs the
virtualenv or the test dependencies.
