Metadata-Version: 2.4
Name: agentsor-file
Version: 0.2.3
Summary: Validate local CSV and Parquet schemas and monitor scheduled outputs
Author: Agentsor
License-Expression: MIT
Project-URL: Homepage, https://agentsor.ai/file-contracts
Project-URL: Repository, https://github.com/linkoinsight/agentsor-file
Project-URL: Issues, https://github.com/linkoinsight/agentsor-file/issues
Keywords: csv,parquet,data-contracts,schema-validation,file-monitoring,data-freshness
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyarrow>=16
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: jsonschema<5,>=4.23; extra == "dev"
Requires-Dist: pytest<10,>=8; extra == "dev"
Requires-Dist: ruff<0.14,>=0.12; extra == "dev"
Dynamic: license-file

# Agentsor File Contracts

**Check and validate a Parquet or CSV file schema locally, then monitor every
scheduled output.**

Agentsor File Contracts is a free, MIT-licensed command-line tool for checking
one local CSV or Parquet file against an explicit TOML contract. It checks:

- file readability;
- required columns and Arrow types;
- unexpected columns;
- row and byte bounds;
- optional event-time freshness; and
- optional duplicate history.

`init`, `check`, and `schema` run offline. The separate `report` command checks
the file locally and sends only the fixed redacted result envelope to the
single Agentsor collector. The package has no file-upload or telemetry path.

[Create one free missed-run monitor](https://agentsor.ai/file-contracts) after
the local check works.

## Local quickstart

Linux or macOS and Python 3.11 or newer are required. Windows is not supported
in this release because local credential safety relies on POSIX owner-only
file modes. Install the published package:

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install agentsor-file==0.2.3
agentsor-file init \
  --format parquet \
  --contract file-contract.toml \
  --fingerprint-key-file .agentsor-file.key
```

`init` refuses to replace either target or follow a target symlink. It creates
a random 32-byte fingerprint key with mode `0600` and never prints the key.
Edit the generated `[schema]` table to describe the file, then run:

```bash
agentsor-file check YOUR_FILE.parquet \
  --contract file-contract.toml \
  --fingerprint-key-file .agentsor-file.key \
  --state .agentsor-file-state.json
```

The `init`, `check`, and `schema` commands make no network requests. The CLI
requires the fingerprint key to be an owner-only regular file and refuses
links or group/world-readable modes. Keep it private and stable within one
project. Back it up if fingerprint continuity matters.

The CLI exits `0` for a passed contract, `1` for a failed or inconclusive
result, and `2` for a usage, contract, credential, input, transport, or hosted
receipt error.

This validates an expected schema rather than merely printing the schema that
happens to be present. `init` creates the contract once; after you review its
`[schema]`, each `check` compares required columns and Arrow types, unexpected
columns, freshness, and row/byte bounds against that explicit contract.

## Hosted deadline reporting

After creating a free monitor at
[agentsor.ai/file-contracts](https://agentsor.ai/file-contracts), put the
one-time ingest token in an owner-only credential file without placing it in a
shell argument:

```bash
mkdir -m 700 -p ~/.config/agentsor
umask 077
${EDITOR:-vi} ~/.config/agentsor/file-token
chmod 600 ~/.config/agentsor/file-token
```

Then check and submit one redacted result:

```bash
agentsor-file report YOUR_FILE.parquet \
  --contract file-contract.toml \
  --fingerprint-key-file .agentsor-file.key \
  --token-file ~/.config/agentsor/file-token
```

`report` has a fixed HTTPS destination, disables inherited proxies and
redirects, verifies TLS, bounds request/response sizes, validates the complete
result before network I/O, and never prints the token. It prints a fixed
acknowledgement containing the run ID, local overall result, duplicate-receipt
flag, and next hosted deadline.

Hosted reporting does not currently accept `--state`; configure
`reject_duplicates = false` for that command. Use offline `check --state` when
local duplicate-output detection is required. This avoids mutating duplicate
state before a network failure can be retried safely.

## Monitor a scheduled export from cron

For a start-to-finish setup using one consistent unprivileged path layout, see
[validate a Parquet schema and monitor cron outputs](https://agentsor.ai/file-contracts/guides/validate-parquet-schema-cron).

Put the producer and the receipt in one fail-closed script:

```sh
#!/bin/sh
set -eu

/opt/orders/bin/export-daily-orders
/opt/orders/.venv/bin/agentsor-file report \
  /srv/orders/daily-orders.parquet \
  --contract /opt/orders/file-contract.toml \
  --fingerprint-key-file /opt/orders/credentials/file-fingerprint.key \
  --token-file /opt/orders/credentials/agentsor-file-token
```

Then schedule that script with the cadence selected for the free monitor:

```cron
5 7 * * * /opt/orders/bin/run-daily-export >>/var/log/orders-export.log 2>&1
```

If the producer never reaches `report`, the hosted deadline detects the
missing receipt. If it creates an unreadable, stale, empty, oversized, or
schema-changed file, `report` submits the fixed redacted failure result and
exits nonzero. Keep the script and all credential files owner-only. The same
pattern works for CSV by changing the contract format and input path.

## Contract format

```toml
[contract]
format = "parquet"
allow_extra_columns = false
min_rows = 1
max_rows = 1000000
min_bytes = 1
max_bytes = 104857600
csv_delimiter = ","
reject_duplicates = false

# Optional; configure both together.
# event_time_column = "event_time"
# max_age_seconds = 86400

[schema]
id = "string"
amount = "double"
# event_time = "timestamp[ms]"
```

`format` may be `csv` or `parquet`, or omitted when one contract intentionally
supports both. Schema values use PyArrow type aliases. Parquet physical types
must match exactly. CSV has no physical type metadata, so required CSV columns
must safely cast to their configured Arrow types.

Freshness uses the newest non-null event time. Timestamp columns without a
timezone are interpreted as UTC. `reject_duplicates = true` requires
`--state`; without usable state, the result is inconclusive. State contains
only project-keyed output fingerprints and retains the latest 4,096 unique
values.

## Redacted result envelope

`check` writes exactly one JSON object to stdout. The v1 envelope contains a
canonical run UUID and UTC timestamps, format, bounded row/byte counts, six
fixed check statuses, fixed reason codes, and three fingerprints. It does not
contain file paths, file names, column names, raw values, exception messages,
or arbitrary text.

The contract, schema, and output fingerprints are domain-separated
HMAC-SHA256 values keyed by the supplied project key. They support equality
and drift checks inside that project without exposing a global raw SHA-256
that could correlate low-entropy files or schemas across projects.

Redaction is not zero knowledge: row/byte counts, timing, outcomes, and
within-project equality can still reveal metadata. Review that boundary
before sharing a result.

Print the installed formal JSON Schema with:

```bash
agentsor-file schema
```

The fixed checks are `readability`, `schema`, `rowBounds`, `byteBounds`,
`eventFreshness`, and `duplicate`. Required checks use
`passed|failed|inconclusive`; the two optional checks may also use
`not_configured`. Overall status is:

- `passed` when every check is passed or not configured and reasons are empty;
- `failed` when at least one check failed; or
- `inconclusive` when none failed and at least one could not complete.

## Python API

```python
from pathlib import Path

from parquet_guard import check_file, load_file_contract

contract = load_file_contract("file-contract.toml")
key = Path(".agentsor-file.key").read_bytes()
result = check_file("output.parquet", contract, fingerprint_key=key)
print(result.as_dict())
```

Applications should protect the key at least as carefully as the local CLI
does and should not log it.

## Deliberate limits

- Files larger than 1 TiB and row counts above 1 trillion are outside the v1
  result envelope.
- Files are read locally by PyArrow; set a realistic `max_bytes` to establish
  a resource boundary before parsing.
- Duplicate state is atomic and fail-closed but assumes one writer at a time.
- This release checks completed files; it does not watch producer directories
  or establish that a producer has finished writing.
- A same-size file that changes during a run is detected by a second keyed
  fingerprint and produces an inconclusive result.
- Hosted reporting sends only aggregate result metadata; it does not send the
  file, its name or path, column names, values, storage location, or arbitrary
  notes.

## Historical Parquet demonstration

This repository began as an owned engineering demonstration of a fail-closed,
idempotent Parquet batch pipeline. That truthful history remains available
through the compatibility command:

```bash
python examples/generate_sample.py
parquet-guard --config examples/config.toml
```

It demonstrates validation, deterministic decimal transformation, quarantine,
atomic output/state, and rerun integrity. It is not client work and is not
evidence that customer data has been processed. Agentsor File Contracts does
not silently invoke or quarantine files through that historical command.

## Development

```bash
python -m pip install -e '.[dev]'
pytest
ruff check src tests
python -m build
```

The tests cover both formats, config validation, redaction, formal envelope
validation, project-key isolation, schema stability, bounds, freshness,
duplicate state, fail-closed state handling, CLI exit behavior, secure init,
and the historical batch demonstration.

For bounded implementation work, see
[Agentsor Automation Reliability](https://agentsor.ai/automation-reliability).
Do not send
credentials, source code, personal data, production records, or confidential
material by email.

License: MIT.
