Metadata-Version: 2.4
Name: datacoolie-studio
Version: 0.1.2
Summary: Local web studio for inspecting DataCoolie metadata, lineage, and ETL logs.
Project-URL: Homepage, https://github.com/datacoolie/datacoolie-studio
Project-URL: Repository, https://github.com/datacoolie/datacoolie-studio
Project-URL: Issues, https://github.com/datacoolie/datacoolie-studio/issues
Author: DataCoolie Team
License-Expression: AGPL-3.0-or-later
Keywords: data-engineering,etl,lineage,metadata,observability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: databricks-sdk<0.122,>=0.121
Requires-Dist: duckdb>=1.1
Requires-Dist: fastapi>=0.115
Requires-Dist: keyring>=25.0
Requires-Dist: libcst>=1.8
Requires-Dist: openpyxl>=3.1
Requires-Dist: pydantic>=2.8
Requires-Dist: pytz>=2024.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: ruamel-yaml>=0.18
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: sqlglot>=27.0
Requires-Dist: tzdata>=2024.1
Requires-Dist: uvicorn[standard]>=0.30
Provides-Extra: adls
Requires-Dist: adlfs>=2024.12; extra == 'adls'
Requires-Dist: azure-identity>=1.19; extra == 'adls'
Requires-Dist: azure-storage-blob>=12.24; extra == 'adls'
Provides-Extra: cloud
Requires-Dist: adlfs>=2024.12; extra == 'cloud'
Requires-Dist: azure-identity>=1.19; extra == 'cloud'
Requires-Dist: azure-storage-blob>=12.24; extra == 'cloud'
Requires-Dist: azure-storage-file-datalake>=12.18; extra == 'cloud'
Requires-Dist: boto3>=1.35; extra == 'cloud'
Requires-Dist: gcsfs>=2025.3; extra == 'cloud'
Requires-Dist: google-cloud-storage>=3.0; extra == 'cloud'
Requires-Dist: s3fs>=2025.3; extra == 'cloud'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: twine>=6.0; extra == 'dev'
Provides-Extra: gcs
Requires-Dist: gcsfs>=2025.3; extra == 'gcs'
Requires-Dist: google-cloud-storage>=3.0; extra == 'gcs'
Provides-Extra: minio
Requires-Dist: boto3>=1.35; extra == 'minio'
Requires-Dist: s3fs>=2025.3; extra == 'minio'
Provides-Extra: onelake
Requires-Dist: azure-identity>=1.19; extra == 'onelake'
Requires-Dist: azure-storage-file-datalake>=12.18; extra == 'onelake'
Provides-Extra: s3
Requires-Dist: boto3>=1.35; extra == 's3'
Requires-Dist: s3fs>=2025.3; extra == 's3'
Description-Content-Type: text/markdown

# DataCoolie Studio

DataCoolie Studio is a local web app for exploring DataCoolie projects. Use it to manage sources, edit metadata, inspect lineage, and monitor extract, transform, and load (ETL) runs.

## What you can do

- Organize sources by Project and Environment
- Read and edit metadata in `JSON`, `YAML`, or `XLSX` format
- Inspect lineage from metadata, SQL queries, and Python code
- Monitor Dataflow, Job, and System logs
- Connect to Local, S3, MinIO, ADLS, OneLake, GCS, and Databricks storage
- Store cloud credentials in the operating system credential store

Studio keeps source files as the source of truth. Metadata saves validate the document and create a backup before replacing the original file. Lineage combines evidence for display without creating a merged metadata file.

## Install and run

DataCoolie Studio requires Python 3.11 or later.

```powershell
pip install datacoolie-studio
datacoolie-studio
```

The launcher starts Studio at `http://127.0.0.1:8765`, creates its local workspace on first run, and opens your browser.

Install only the cloud integrations you need:

```powershell
pip install "datacoolie-studio[s3]"
pip install "datacoolie-studio[minio]"
pip install "datacoolie-studio[adls]"
pip install "datacoolie-studio[onelake]"
pip install "datacoolie-studio[gcs]"
```

Databricks SDK support is included in the base installation. Use
`pip install "datacoolie-studio[cloud]"` to install every other cloud
integration.

## Create your first workspace

1. Create a **Project**
2. Add an **Environment** such as `dev`, `test`, or `prod`
3. Add a metadata file or scan a DataCoolie project
4. Add ETL logs for Monitoring
5. Add Python code artifacts when metadata references Python functions
6. Open **Metadata**, **Assets**, **Lineage**, or **Monitoring**

Metadata is required. Logs and code artifacts are optional.

## Configure Studio

Studio stores local state under `~\.datacoolie\datacoolie-studio\`:

```text
db\studio.db
backups\
cache\
logs\
```

Common launcher options:

```powershell
datacoolie-studio --port 8765
datacoolie-studio --host 127.0.0.1
datacoolie-studio --db .\.scratch\studio.db
datacoolie-studio --database-url "postgresql+psycopg://user:password@host:5432/datacoolie_studio"
datacoolie-studio --no-open
```

You can also configure storage with environment variables:

| Variable | Purpose |
|---|---|
| `DATACOOLIE_STUDIO_DB` | SQLite workspace database path |
| `DATACOOLIE_STUDIO_DATABASE_URL` | SQLAlchemy database URL; overrides the SQLite path |
| `DATACOOLIE_STUDIO_RESULT_CACHE_URL` | Result-cache SQLite URL |

Studio binds to `127.0.0.1` by default. Choose a shared database and review network access before hosting it for multiple users.

## Develop from source

Run the backend directly from `src`. This assumes the active Python environment already contains the dependencies declared in `pyproject.toml`.

```powershell
$env:PYTHONPATH = "$PWD\src"
python -m uvicorn datacoolie_studio.main:app `
  --reload `
  --host 127.0.0.1 `
  --port 8765
```

Run the frontend in another terminal:

```powershell
cd frontend
npm install
npm run dev
```

Open `http://127.0.0.1:5173`. Vite sends API requests to the backend at `http://127.0.0.1:8765`.

Build the frontend into the Python package:

```powershell
cd frontend
npm run build
```

Run repository checks:

```powershell
.\scripts\verify.ps1
.\scripts\verify.ps1 -Mode Full
```

The default check covers architecture, packaged static assets, security, API contracts, frontend tests, and the production build. Full mode also runs the complete backend test suite.
