Metadata-Version: 2.4
Name: daggydata
Version: 0.1.1
Summary: Upload datasets to Daggy with an API key: files, hive-partitioned folders, or pandas/polars DataFrames, up to 25 GB, in resumable parallel parts.
Keywords: daggy,daggydata,datasets,upload,parquet,csv,dataframe
Author: Daggy
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Requires-Dist: requests>=2.31
Requires-Dist: pyarrow>=15 ; extra == 'pandas'
Requires-Python: >=3.10
Project-URL: Homepage, https://daggydata.com
Provides-Extra: pandas
Description-Content-Type: text/markdown

# daggydata

Upload datasets to [Daggy](https://daggydata.com) from a script, a notebook or cron, with an API
key: one file, several files, a hive-partitioned folder, or a pandas or polars DataFrame straight
from memory, up to 25 GB. CSV, TSV, JSON and Parquet are supported. Bytes go straight to storage
in parallel parts, interrupted uploads resume where they stopped, and only `requests` is needed,
no cloud credentials.

```bash
pip install daggydata
```

The package installs the `daggy` command and imports as `daggy`.

Create an API key on the **API keys** page of the Daggy app, then:

```bash
export DAGGY_API_KEY=daggy_prod_...
daggy upload sales ./sales.parquet
daggy upload events ./events/ --overwrite --description "Clickstream"
```

```python
import daggy  # reads DAGGY_API_KEY

ds = daggy.upload("sales", "./sales.parquet")
print(ds.name, ds.num_obs, "rows")
```

## What gets uploaded

- **A file** is uploaded under its name, with anything outside `[A-Za-z0-9_.-]` replaced
  (`Q3 sales.csv` → `Q3_sales.csv`).
- **A folder** becomes one dataset. Hive folders like `region=west/` become columns. Marker
  files (`_SUCCESS`, `.crc`, `_temporary/`) are skipped.
- **Formats:** `.parquet .pq .csv .tsv .txt .json .ndjson .jsonl`, one format per upload,
  uncompressed.
- **`--overwrite`** (`overwrite=True`) replaces an existing dataset; without it, an existing
  name is refused. Appends are not supported.

| Limit | Value |
|---|---|
| Size per upload | 25 GiB |
| Files per upload | 200 |
| Open uploads at once | 5 |
| Paths in a folder | `[A-Za-z0-9_.=/-]`, at most 8 levels |

## From a DataFrame

```bash
pip install "daggydata[pandas]"   # adds pyarrow; polars needs nothing extra
```

```python
ds = daggy.upload("sales", df)   # pandas or polars DataFrame, or a pyarrow Table
```

Any object with `__arrow_c_stream__` (DuckDB relations, ibis tables, ...) works too, with
pyarrow installed. The table is written to a temporary Parquet file (it needs free space in
`$TMPDIR`), uploaded, and deleted. A pandas index is kept as a column only when every level is
named (`df.set_index("date")` keeps `date`; the leftover integer index of a filtered frame is
dropped).

A table can't be resumed. If an upload fails, call `daggy.upload` again, with
`overwrite=True` if the dataset was already created.

## Resuming

The session id is printed (logged, from Python) before any data is sent. If a run is killed or
a part keeps failing, run it again with the id; only the missing parts are sent:

```bash
daggy upload sales ./sales.parquet --resume <session_id>
```

```python
try:
    daggy.upload("sales", "./sales.parquet")
except daggy.UploadError as e:
    print(e.code, e.session_id, e.final_state)
    # final_state None or "uploading": daggy.upload(..., resume=e.session_id)
```

`UploadError` has `status`, `code`, `message`, `session_id` and `final_state` (`None` for a
refused request or a failed part, otherwise `failed`, `aborted` or `uploading`). From Python,
progress goes to the `daggy` logger: `logging.basicConfig(level=logging.INFO)` shows it.

## Options

| CLI | Python | |
|---|---|---|
| `--overwrite` | `overwrite=True` | replace an existing dataset |
| `--description D` | `description="D"` | dataset description (≤ 10,000 characters) |
| `--resume SID` | `resume="SID"` | finish an earlier upload |
| `--workers N` | `workers=N` | parallel part uploads (default 8) |
| `--api URL` | `api_url="URL"` | API address; default from the key (`DAGGY_API_URL` also works) |
| `DAGGY_API_KEY` | `api_key="..."` | the key |

CLI exit codes: 0 done; 1 the upload ended `failed`/`aborted` or needs `--resume`; 2 bad usage,
a refused request or a part that kept failing; 130 interrupted.

## License

MIT
