Metadata-Version: 2.5
Name: sunstone-data-platform
Version: 1.2.0
Summary: Sunstone Data Platform - Data lakehouse with semantic metadata
Author-email: Sunstone Institute <stig@sunstone.institute>
License-Expression: MIT
Requires-Python: >=3.12
Requires-Dist: charset-normalizer>=3.4.0
Requires-Dist: duckdb>=1.5.3
Requires-Dist: fastexcel>=0.20.2
Requires-Dist: httpx>=0.27.0
Requires-Dist: publicsuffixlist>=1.0.2.20261003
Requires-Dist: pyiceberg[duckdb,gcsfs,s3fs]>=0.10.0
Requires-Dist: python-dateutil>=2.9.0.post0
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: python-slugify>=8.0.0
Requires-Dist: rich>=14.0.0
Requires-Dist: sunstone-py>=1.15.0
Requires-Dist: typer>=0.27.1
Provides-Extra: pipelines
Requires-Dist: dagster-docker>=0.29.0; extra == 'pipelines'
Requires-Dist: dagster-postgres>=0.29.0; extra == 'pipelines'
Requires-Dist: dagster-webserver>=1.13.0; extra == 'pipelines'
Requires-Dist: dagster>=1.13.9; extra == 'pipelines'
Description-Content-Type: text/markdown

# Sunstone Data Platform

A data lakehouse for importing, versioning, and querying external datasets with full provenance tracking. Built on Apache Iceberg, an Iceberg REST catalog, a versioned RDF graph in Postgres, RustFS, and DuckDB.

Data scientists import CSV/Excel files from local paths or URLs into Iceberg tables, query them with DuckDB using git-like version references, and publish dataset metadata and W3C PROV-DM provenance into the graph. The graph is the system of record for metadata, branches and tags.

## Architecture

```
              ┌─────────────────────────────────┐
              │ sunstone_data (Python)          │
              │ sunstone data CLI, sunstone:    │
              │ URLs, import pipeline, DuckDB   │
              └───────┬─────────────────┬───────┘
         Iceberg REST │                 │ GSP, refs, SPARQL
                      ▼                 ▼
          ┌─────────────────┐   ┌─────────────────┐
          │ iceberg-catalog │──▶│ graph-server    │
          │ :8181           │   │ :8080           │
          └────────┬────────┘   └────────┬────────┘
                   ▼                     ▼
          ┌─────────────────┐   ┌─────────────────┐
          │ RustFS (S3)     │   │ Postgres graph  │
          │ Iceberg tables, │   │ (SoR), copied   │
          │ original files  │   │ to Oxigraph     │
          └─────────────────┘   └─────────────────┘
```

On every table commit, iceberg-catalog also writes the table's descriptor into
the graph. See [docs/architecture.md](docs/architecture.md) for the
full data flow.

**Stack**: Python 3.11+ | PyIceberg | DuckDB | iceberg-catalog | graph-server | Oxigraph | RustFS | PostgreSQL 16

## Endpoints

- `/catalog` — DCAT JSON-LD view of the lakehouse, served by graph-server views
  (see [the views design](docs/specs/2026-08-02-graph-server-views-design.md))
- `/ui/` — the data explorer, a 3D view of the data graph, served by
  graph-server (see [docs/data-explorer.md](docs/data-explorer.md))

## Local Development Setup

### Prerequisites

- Python 3.11+
- [uv](https://docs.astral.sh/uv/) package manager
- Docker and Docker Compose

### 1. Clone and install

```bash
git clone <repo-url>
cd data-platform
uv sync
```

### 2. Configure environment

```bash
cp .env.example .env
# Edit .env if you want to change default credentials
```

### 3. Start the infrastructure

```bash
docker compose --profile s3 up -d
```

This starts the graph stack (Postgres, `graph-migrate`, `oxigraph`,
`graph-server`, `materializer` — see
[docs/architecture.md](docs/architecture.md#infrastructure)) plus:
- **iceberg-catalog** on port 8181 (Iceberg REST catalog, S3-backed)
- **object-store** (RustFS) on port 9000 (S3-compatible storage, console on 9001)

For GCS testing, use `docker compose --profile gcs up -d` instead, which
starts `fake-gcs-server` — note there's no GCS-backed iceberg-catalog wired
up yet, so this profile provides the emulator but not a working GCS
catalog.

### Cloudflare R2 backend

The `r2` profile hasn't moved to the graph stack yet — it still runs Nessie
directly, against a real Cloudflare R2 bucket (no local object store):

1. Create an R2 bucket in the Cloudflare dashboard.
2. Create an R2 API token (R2 > Manage R2 API Tokens) with **Object Read & Write** scoped to that bucket. Copy the Access Key ID and Secret Access Key.
3. Copy `.env.r2.example` values into your `.env` and fill in `R2_ACCOUNT_ID`, `R2_BUCKET`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`.
4. If your Postgres volume already exists from an earlier `s3`/`gcs` run, create the R2 version-store database once (the init script in `docker/init-databases.sql` only runs on a *fresh* `iceberg-db` volume):

   ```bash
   docker compose up -d postgres
   docker compose exec postgres \
     psql -U iceberg -d iceberg \
     -c "CREATE DATABASE nessie_r2; GRANT ALL PRIVILEGES ON DATABASE nessie_r2 TO iceberg;"
   ```

   (A fresh stack — or `docker compose down -v && docker compose --profile r2 up -d` — creates it automatically. The reset destroys all local data.)
5. Start the R2 profile:

   ```bash
   docker compose --profile r2 up -d
   ```

   Nessie listens on port 19122. Point your client at `http://localhost:19122`.

R2-specific Nessie settings the profile sets for you:

- `region=auto` (R2 ignores region but the SDK requires a value).
- `endpoint=https://<R2_ACCOUNT_ID>.r2.cloudflarestorage.com` (account-style host).
- `path-style-access=true` so requests are `<endpoint>/<bucket>/<key>` — R2 accepts both styles but path-style avoids virtual-host DNS issues with some older clients.

The local PyIceberg/DuckDB client reads Iceberg data files straight from R2, so it needs the same bucket and credentials. It derives those from the same four `R2_*` vars automatically — warehouse (`s3://<R2_BUCKET>/`), endpoint, access keys, `region=auto`, and a default `CATALOG_URI` of `http://localhost:19122`. No separate `S3_*`/`ICEBERG_WAREHOUSE` settings are required; if you set them explicitly they take precedence.

### 4. Verify the stack

```bash
uv run scripts/smoke_test.py
```

### 5. Set up your environment

The `sunstone` CLI comes from sunstone-py. data-platform registers as a plugin and adds the `data` subcommand group. You configure one value per environment, the base IRI (`base_iri`). graph-server is served at the base, the Iceberg catalog under `{base_iri}iceberg/`, and datasets and organizations are named under it. The client discovers sign-in settings from graph-server.

```bash
# Add your local Docker stack as an environment
sunstone env add local data-platform.base_iri=https://data.example.org/
sunstone env use local
```

`base_iri` must equal the iceberg-catalog service's `BASE_IRI`. In the local Docker stack, graph-server (port 8080) and the catalog (port 8181) listen on different ports, so also set the service overrides `GRAPH_URL=http://localhost:8080` and `CATALOG_URI=http://localhost:8181` (see `.env.example`). They never affect IRIs.

### 6. Try it out

```bash
# Import a local CSV
sunstone data import path/to/data.csv

# Import from a URL
sunstone data import https://example.com/dataset.csv

# List namespaces and tables
sunstone data ls

# View dataset metadata from the graph
sunstone data show namespace.table_name

# Snapshot history
sunstone data history namespace.table_name

# Publish the current snapshot as a named version
sunstone data publish namespace.table_name --version v1.0
```

## Running Tests

```bash
# Unit tests (fast, no Docker required)
uv run pytest tests/unit/

# Integration tests (run inside Docker automatically; needs Docker)
uv run pytest tests/integration/

# All tests
uv run pytest

# Specific test file with verbose output
uv run pytest tests/unit/test_query.py -v
```

## CLI Commands

| Command | Description |
|---------|-------------|
| `sunstone data login`, `logout`, `whoami` | Sign in to graph-server (Keycloak) |
| `sunstone data ls [namespace]` | List namespaces and tables |
| `sunstone data show <table>` | Dataset metadata from the graph (table, Turtle or JSON-LD) |
| `sunstone data info [table]` | Iceberg table properties for one or all datasets |
| `sunstone data import <source>` | Import CSV/Excel from a path or URL |
| `sunstone data publish <table>` | Publish dataset metadata to the graph; `--version` names the current snapshot |
| `sunstone data history <table>` | Iceberg snapshot history |
| `sunstone data lineage <table>` | Provenance from the published PROV facts |
| `sunstone data sparql <query>` | Run a SPARQL query through graph-server |
| `sunstone data branches`, `branch create\|delete` | List, create and delete graph branches |
| `sunstone data tags`, `tag create\|delete` | List, create and delete graph tags |

Every command takes `--json` for machine-readable output. `import`, `publish`, `show`, `history` and `tag create` take `--branch`. Run `sunstone data <command> --help` for full options.

## Key Concepts

- **Table references** use the form `namespace.table@ref`. A ref is checked in this order:
  1. a branch name
  2. a snapshot id (13 or more digits)
  3. a date or timestamp (UTC if it has no timezone), which selects the last snapshot at or before that time
  4. a named version created with `sunstone data publish --version`

  Snapshot ids, timestamps and named versions resolve against `main`. Graph tags (`sunstone data tag create`) are not valid refs.
- **Deduplication**: Imports are checksum-based (SHA-256). Re-importing the same file is a no-op.
- **Schema evolution**: Safe type promotions (e.g. int32 to int64) are allowed automatically. Unsafe changes require `--force-schema`.
- **Provenance**: Every import records source metadata and schema info as Iceberg table properties, and asserts W3C PROV-DM provenance into the dataset's graph.
- **Original files**: Source files are stored content-addressed in object storage next to the Iceberg data.

## sunstone: URLs

Datasets in the lakehouse can be referenced as URLs anywhere sunstone-py accepts a data source:

```python
import sunstone

df = sunstone.read("sunstone:namespace/table@v1.0")
```

The `@ref` part follows the same rules as table references above. Here `v1.0` is a named version.

## Project Structure

```
src/sunstone_data/
├── plugin.py          # sunstone-py plugin (CLIProvider + URLHandler)
├── url.py             # sunstone: URL parser
├── resolve.py         # Resolves a @ref to a branch or snapshot
├── config.py          # Settings from DataEnvironment or .env
├── discovery.py       # Sign-in settings from graph-server /ui/config.json
├── auth/              # Keycloak login and token handling
├── graph/             # graph-server client and RDF serialization
├── publish.py         # Publishes dataset metadata to the graph
├── catalog.py         # PyIceberg RestCatalog connection to the Iceberg REST catalog
├── query.py           # DuckDB QuerySession with time-travel support
├── cli/               # Typer CLI commands
├── import_/           # Pipeline: fetch → parse → store → metadata
├── metadata/          # Frozen dataclasses, MetadataBuilder, W3C PROV-DM
├── versioning/        # Schema evolution checks
└── utils/             # Checksums, encoding detection, naming
```

## Documentation

- [docs/README.md](docs/README.md) - Index of all documentation
- [docs/architecture.md](docs/architecture.md) - System overview and data flow
- [docs/specs2/](docs/specs2/) - Current design, one file per subsystem. The Python client and CLI are in [05-python-client.md](docs/specs2/05-python-client.md).

## Resetting the Stack

To wipe all data and start fresh:

```bash
docker compose --profile s3 --profile gcs down -v
docker compose --profile s3 up -d
```

## License

MIT

## Graph server authentication

`graph-server` requires a Keycloak bearer token on every route except
`/healthz`/`/readyz`. Access is role-based: the client roles in your token
determine which environment you can use. See
`docs/specs/2026-07-05-graph-server-auth-design.md`.

No hostnames or client ids ship in this package. Each environment's
`base_iri` comes from config: Sunstone-managed machines get
`/etc/sunstone/data_platform.toml` from the `fleet-config` repo, and other
machines set `BASE_IRI` or add an environment with `sunstone env add`.
graph-server publishes the OIDC issuer and public client id at
`{base_iri}ui/config.json`; the CLI reads them from there.

### Humans (CLI, local agents)

```bash
sunstone env init          # cache discovery for configured environments
sunstone data login        # browser sign-in (authorization code + PKCE)
```

A browser window opens for Keycloak SSO; the refresh token renews silently
until the SSO session expires.

### Services (CI, automation)

Set `SUNSTONE_CLIENT_ID` and `SUNSTONE_CLIENT_SECRET` (or `client_id` and
`client_secret` in the environment's `data-platform` section). The client then
uses the OAuth2 client-credentials flow against the discovered issuer instead
of a cached human login.
