Metadata-Version: 2.4
Name: fw-dataset
Version: 0.4.2
Summary: A library for working with Flywheel datasets
Author: joshicola
Author-email: joshicola <joshuajacobs@flywheel.io>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: pandas>=3
Requires-Dist: adlfs>=2024.7.0,<2025
Requires-Dist: pyarrow>=23.0.1,<26
Requires-Dist: duckdb>=1.1.1,<2
Requires-Dist: s3fs>=2024.9.0,<2025
Requires-Dist: gcsfs>=2024.9.0.post1,<2025
Requires-Dist: httpcore>=1.0.0,<2
Requires-Dist: httpx>=0.28.1,<0.29
Requires-Dist: h11>=0.16.0,<0.17
Requires-Dist: fw-client>=2.3.1,<3
Requires-Dist: flywheel-sdk>=22,<23
Requires-Dist: pydantic>=2.9.2,<3
Requires-Dist: orjson>=3.10.7,<4
Requires-Dist: deepdiff>=8.4,<9
Requires-Dist: fsspec>=2024.9.0,<2025
Requires-Dist: jinja2>=3.1.4,<4
Requires-Python: >=3.12, <4.0
Description-Content-Type: text/markdown

# fw-dataset

`fw-dataset` provides the classes and functions used to create, manage, and serve
**Flywheel Datasets** — a way to organize, share, and query data derived from the
Flywheel Data Model.

> [!WARNING]
> This package is under active development and should be considered unstable. It is
> provided as-is, with no guarantee of support or maintenance at this stage. Features
> may be incomplete, change without notice, or be removed in future versions. Use it
> at your own risk, for experimental or development purposes only.

## Overview

A Flywheel Dataset is a versioned, columnar snapshot of a Flywheel project's data,
stored in cloud or local object storage and queryable with standard tabular tools.
The package exposes three primary entry points:

- **`DatasetBuilder`** — renders a dataset from a Flywheel project snapshot.
- **`FWDatasetClient`** — accesses, queries, and manages existing datasets.
- **`Dataset` / `Table`** — object models representing a dataset and its tables.

## Documentation

The [Dataset Definition](docs/dataset_definition.md) document is the authoritative
reference for the dataset format. It covers:

- [Dataset components and on-disk structure](docs/dataset_definition.md#dataset-components)
- [How datasets are rendered, including primary and derived
  tables](docs/dataset_definition.md#dataset-rendering)
- [Dataset storage layout](docs/dataset_definition.md#dataset-storage)
- [A worked example structure and the current primary table
  schemas](docs/dataset_definition.md#example-dataset-structure)

## Installation

`fw-dataset` requires **Python 3.12 or later**. Install it with pip:

```bash
pip install fw-dataset
```

## Usage

### Rendering datasets

Use `DatasetBuilder` to render a Flywheel dataset from a project. See
[notebooks/quickstart_dataset_creation.ipynb](notebooks/quickstart_dataset_creation.ipynb)
for a complete walkthrough.

### Accessing and managing datasets

Use `FWDatasetClient` to access and query an existing dataset. See
[notebooks/quickstart_dataset_management.ipynb](notebooks/quickstart_dataset_management.ipynb)
for a complete walkthrough.

### Working with unassociated datasets

A valid dataset that is not associated with a Flywheel project can still be accessed
directly. Provide the `type`, `bucket`, `prefix`, and `credentials` of the cloud or
local filesystem to instantiate and query it — no API key or client instantiation
required:

```python
from fw_dataset import FWDatasetClient

fs_type = "s3"  # or "gcs", "azure", "fs", "local"
bucket = "your-bucket"
prefix = "your-prefix"
credentials = {"url": "{bucket-specific-credential-string}"}

dataset = FWDatasetClient.get_dataset_from_filesystem(fs_type, bucket, prefix, credentials)
```

### Merging related datasets

Multiple datasets with related tables can be merged into a single dataset so their
tables can be queried together.

> [!NOTE]
> Federated querying across datasets is not yet enabled; this is a work in progress.

Merging requires that both datasets satisfy the following:

1. A valid `tables` directory structure.
2. A valid `schemas` directory structure, where:
   - every table in `tables` has a corresponding schema file in `schemas`;
   - each schema file is named `{table_name}.schema.json`; and
   - each schema file is valid JSON with at least the following structure:

     ```json
     {
         "schema": "http://json-schema.org/draft-07/schema#",
         "id": "{table_name}",
         "description": "",
         "properties": {},
         "required": [],
         "type": "object"
     }
     ```

3. Tables and schemas selected from the `source` must not share names with existing
   tables or schemas in the `destination`.

Once these requirements are met, merge the datasets by copying or moving the selected
tables and schemas from the `source` dataset into the `destination` dataset.

## Flywheel project requirements

For the Flywheel Dataset client and the `Dataset` objects to function, a project must
provide valid custom-information metadata and a well-formed storage layout.

### Project metadata

The Flywheel project must carry the following custom information:

```json
{
    "dataset": {
        "type": "s3",
        "bucket": "{bucket-name}",
        "prefix": "{path/to/dataset}",
        "storage_id": "storage-id-of-fw-storage-object"
    }
}
```

| Field | Description |
| --- | --- |
| `type` | Storage backend: `s3`, `gcs`, `azure`, or `fs`/`local`. |
| `bucket` | Name of the bucket or container holding the dataset. |
| `prefix` | Path to the dataset within the bucket or container. |
| `storage_id` | Flywheel ID of the storage record for the bucket or filesystem. |

The directory structure beneath `prefix` must follow the [Dataset
structure](#dataset-structure) described below.

### Dataset structure

A dataset is stored beneath its `prefix` with the following layout:

```text
{bucket}/{prefix}/
└── versions/
    └── {version}/
        ├── provenance/
        │   ├── dataset_description.json
        │   ├── snapshot.db.gz
        │   ├── snapshot_info.json
        │   └── project.json
        ├── tables/
        │   └── {table_name}/
        │       └── {hash}.parquet
        └── schemas/
            └── {table_name}.schema.json
```

Each version lives in its own subdirectory, named with its version identifier
(typically a BSON ID such as `66cf6701af1c6f3855f1ee61`). The "latest" version is
determined dynamically by comparing the creation dates recorded in each version's
`dataset_description.json`.

This layout is described more completely in the [Dataset
definition](docs/dataset_definition.md#dataset-components) document.

#### Schema files

Schema files describe the schema of each table and live in the `schemas` directory.
Each is named `{table_name}.schema.json`. Schemas should ideally be fully descriptive,
but a minimal schema is sufficient to make a table queryable:

```json
{
    "schema": "http://json-schema.org/draft-07/schema#",
    "id": "{table_name}",
    "description": "Table derived from tabular data file: conditions.csv",
    "properties": {},
    "required": [],
    "type": "object"
}
```
