Metadata-Version: 2.5
Name: agnue-push
Version: 0.5.0
Summary: SDK and CLI for pushing data to the AgNUE platform
Project-URL: Homepage, https://agnue.com/
Project-URL: Source, https://github.com/teknologisk-institut/AgNUE/tree/main/sdk/python
Project-URL: Issues, https://github.com/teknologisk-institut/AgNUE/issues
Project-URL: Changelog, https://github.com/teknologisk-institut/AgNUE/blob/main/sdk/python/CHANGELOG.md
Requires-Python: >=3.10
Requires-Dist: boto3>=1.34
Requires-Dist: httpx>=0.28
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: convert
Description-Content-Type: text/markdown

# agnue-push

CLI and Python SDK for uploading sensor data to the [AgNUE](https://agnue.com/) platform.

**Step-by-step guide** (install, configure, first upload, scheduling with cron or Windows Task Scheduler, troubleshooting): [Automatic uploads with agnue-push](https://docs.nuedata.eu/docs/getting-started/agnue-push).

## Installation

```bash
pip install agnue-push
```

## Configuration

When your site is approved, you receive an `agnue.toml` file by email. Place it in the directory you run `agnue` from — the SDK reads `./agnue.toml`, or pass `--config PATH`.

You can also configure via environment variables: `AGNUE_PLATFORM_URL`, `AGNUE_CLIENT_ID`, `AGNUE_CLIENT_SECRET`, and optionally `AGNUE_KEYCLOAK_URL` (defaults to `<platform origin>/realms/agnue`) and `AGNUE_S3_*`. Precedence: CLI flags / explicit args > env vars > `agnue.toml`.

## CLI

```bash
# Upload a single file
agnue upload data.csv --data-source DK-Foo-SOIL-001

# Upload a directory
agnue upload ./sensor-exports/ --data-source DK-Foo-SOIL-001

# Upload with glob, excluding backups
agnue upload "*.csv" --data-source DK-Foo-SOIL-001 --exclude "*_backup*"

# Several patterns; a leading ! excludes (single-quote it in bash)
agnue upload "*.csv" "*.txt" '!*.locked' --data-source DK-Foo-SOIL-001

# Scheduled runs: send everything, delete what arrived, retry the rest next run
agnue upload "*.dat" '!*.locked' --data-source DK-Foo-SOIL-001 --delete-after-upload
```

The `--data-source` ID follows the format `CC-SSS-DOMAIN-SEQ` (e.g. `DK-Foo-SOIL-001`).

**Compression:** text files (`.csv`, `.txt`, `.dat`, `.tsv`, `.json`) are gzip-compressed before upload; the platform keeps the original content. Pass `--no-compress` (or `upload(..., compress=False)`) to send files as-is.

**Deleting after upload:** pass `--delete-after-upload` (or `upload(..., delete_after_upload=True)`) to delete each local file once the platform or S3 confirms it has it. A file that changed after it was read or can't be deleted is kept and reported, but a write landing in the instant before deletion is lost, so leave out the file the logger is still writing to (`'!*.locked'`, or have the logger start a new file e.g. every 30 minutes and exclude the current one); a file only queued locally in `agnue-staging/` is kept too. Run the same command again later and those are sent (or recognised) and deleted then.

**Fallback behaviour:** if the platform API is unreachable after 3 attempts, files are uploaded directly to S3 only if the resolved S3 settings are complete (from the `[s3]` block, `AGNUE_S3_*` env vars or `--s3-*` flags), otherwise copied to a local `agnue-staging/` folder. In both cases `upload()` returns `None`.

### Registering data sources (for AI agents)

`agnue sources` registers data sources from JSON spec files, with the site client in `agnue.toml`. It is built for coding agents; `agnue skill --install` installs the instructions for Claude Code (`agnue skill` prints them).

```bash
agnue sources list                                 # this site's sources and status
agnue sources template --data-type LAB_ANALYSIS > specs/nitrate.json
agnue sources validate specs/*.json                # server dry run, nothing saved
agnue sources register specs/*.json                # writes the assigned ID into each spec
agnue domains | badm-variables nitrate | locations | credentials list
agnue api GET api/v1/data-sources/domains          # any endpoint, authenticated
```

Output is JSON. A failed spec doesn't stop the rest; the exit code is 1 if any failed. Registration stops at DRAFT: an operator approves the schema. These commands are never loaded by `agnue upload`.

## Python API

```python
from agnue_push import AgNUEClient

client = AgNUEClient.from_config()  # reads ./agnue.toml or env vars
# client = AgNUEClient.from_config("/path/to/agnue.toml")

# Upload a file — returns ingestion_id, or None after the direct-S3 or local-staging fallback
ingestion_id = client.upload("data.csv", data_source_id="DK-Foo-SOIL-001")

# Re-uploading the same file is safe: the platform recognises it and returns the original ingestion_id
```

Pass explicit config instead of a file:

```python
client = AgNUEClient.from_config(
    platform_url="https://nuedata.eu",
    client_id="site-DK-Foo",
    client_secret="...",
)
```


# Changelog

## Unreleased

### Added

- `agnue sources template | validate | register | list`: register data sources from JSON spec files, meant for AI coding agents. `validate` is a server dry run; `register` uploads the sample (or stores the ZentraCloud credential), sends an optional help request, and writes the assigned ID back into the spec so re-runs are safe. Needs a platform that assigns data source IDs (`domain` in the create request).
- Lookups `agnue domains`, `agnue badm-variables`, `agnue locations`, `agnue credentials list`, and `agnue api METHOD PATH [--data @body.json]` for any other endpoint.
- `agnue skill [--install]` prints the agent instructions shipped in the package, or installs them to `~/.claude/skills/agnue-cli/SKILL.md`.
- `--delete-after-upload` / `upload(..., delete_after_upload=True)` deletes the local file once the platform or S3 confirms it has it. Never deletes a file that was only queued locally, or one seen to change after it was read. A write in the instant before deletion is not caught, so exclude files the logger is still writing to.
- `agnue upload --config PATH` reads a config file other than `./agnue.toml`, like the `sources` commands.
- `agnue upload` takes several paths or patterns; a pattern starting with `!` excludes matching files (same as `--exclude`). A file matched by more than one pattern is sent once.

## 0.4.0

### Breaking

- Removed Parquet conversion and the `convert` argument of `upload()`. It guessed each file's layout and could change values (headerless files lost their first row, some files failed to upload). Files are now sent exactly as they are. `--no-convert` and `pip install 'agnue-push[convert]'` still work but do nothing.
- Requires a platform that accepts compressed uploads (rolled out before this release).

### Added

- Text files (`.csv`, `.txt`, `.dat`, `.tsv`, `.json`) are gzip-compressed before upload, typically 4–25× smaller. The platform stores the original content. Turn it off with `--no-compress` or `upload(..., compress=False)`.

## 0.3.0

### Breaking

- Removed `AgNUEClient.verify_upload()` and `AgNUEClient.list_uploads()`. They could only see files sent through the direct-S3 fallback, not files sent through the platform. To check whether a file arrived, upload it again: the platform recognises it and returns the original ingestion ID.