Metadata-Version: 2.4
Name: flowfile-duckdb
Version: 0.1.0
Summary: Query a Flowfile catalog from DuckDB / Superset — zero-copy, read-only.
Project-URL: Homepage, https://github.com/edwardvaneechoud/flowfile-duckdb
Project-URL: Repository, https://github.com/edwardvaneechoud/flowfile-duckdb
Project-URL: Issues, https://github.com/edwardvaneechoud/flowfile-duckdb/issues
Author-email: Edward van Eechoud <evaneechoud@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: catalog,delta-lake,duckdb,etl,flowfile,superset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: duckdb>=1.1
Provides-Extra: superset
Requires-Dist: duckdb-engine>=0.13; extra == 'superset'
Description-Content-Type: text/markdown

# flowfile-duckdb

Query a [Flowfile](https://github.com/edwardvaneechoud/Flowfile) catalog from **DuckDB** (and anything that speaks DuckDB, like **Apache Superset**) — zero-copy, read-only.

Flowfile stores catalog tables as open **Delta Lake** tables (Parquet + `_delta_log`) and tracks them in a small SQLite database (`flowfile_catalog.db`). DuckDB reads Delta natively via `delta_scan`, so this package is a thin metadata bridge: it reads the SQLite catalog and exposes every table to DuckDB as a **view** over `delta_scan(...)` (or `read_parquet(...)` for legacy Parquet tables). No data is copied or moved; queries read the underlying files live.

## Install

With [uv](https://docs.astral.sh/uv/) (recommended):

```bash
uv pip install flowfile-duckdb          # once published
uv pip install -e .                     # from a checkout
```

Or plain pip:

```bash
pip install flowfile-duckdb
```

## CLI

```bash
flowfile-duckdb list                          # sanity-check discovered tables
flowfile-duckdb build --out catalog.duckdb    # persist a DuckDB file with the views
flowfile-duckdb attach > init.sql             # emit the CREATE VIEW SQL as an init script
```

Common flags: `--catalog-db PATH`, `--schema-template`, `--include-cloud`, `--owner-id N`.

## Python

```python
import duckdb
from flowfile_duckdb import attach_flowfile_catalog

con = duckdb.connect()
attach_flowfile_catalog(con)                  # schemas render as e.g. Demo_sales_analytics
con.sql('SELECT * FROM "Demo_sales_analytics"."sales" LIMIT 5')
```

Schema names default to `{catalog}_{schema}` (single underscore); override via
`schema_template=` (e.g. `"{schema}"` to drop the catalog prefix).

### Where is the catalog DB?

Resolved the same way Flowfile resolves it:

1. `FLOWFILE_DB_PATH` — explicit path to `flowfile_catalog.db`.
2. `<FLOWFILE_STORAGE_DIR or ~/.flowfile>/database/flowfile_catalog.db` (default).

The SQLite connection is opened read-only.

## Apache Superset

```bash
pip install duckdb-engine
flowfile-duckdb build --out /data/flowfile_catalog.duckdb
```

Add a Superset database with SQLAlchemy URI `duckdb:////data/flowfile_catalog.duckdb`.
See [`examples/superset.md`](examples/superset.md).

## Publishing (maintainers)

Distribution is automated via GitHub Actions ([`.github/workflows/publish.yml`](.github/workflows/publish.yml)),
using PyPI **Trusted Publishing** (OIDC — no API token stored):

1. On PyPI, add a *trusted publisher* for this project: owner
   `edwardvaneechoud`, repo `flowfile-duckdb`, workflow `publish.yml`,
   environment `pypi`.
2. Create a GitHub Environment named `pypi`.
3. Push a `v*` tag (or publish a GitHub Release) — the workflow builds with `uv`
   and publishes to PyPI.

```bash
git tag v0.1.0 && git push origin v0.1.0
```

## Known limitations / next steps

- **Cloud tables** (`s3://`, `az://`, `gs://`) are skipped unless `--include-cloud`,
  which also needs a DuckDB `CREATE SECRET`. `flowfile_duckdb/secrets.py` maps a
  decrypted options dict to `CREATE SECRET` SQL but deliberately does **not**
  decrypt Flowfile's `$ffsec$` secrets (that needs Flowfile's master key + HKDF).
- **Virtual / SQL tables** have no physical files and are skipped (they can't be
  read without Flowfile's engine).
- **Refresh:** views read live data, but the table *list* is a snapshot —
  re-run `build`/`attach` to pick up newly created tables.
- On multi-user (docker-mode) installs the catalog holds every user's tables;
  use `--owner-id` to scope to one user.

## License

MIT
