Metadata-Version: 2.4
Name: dagster-ncsa
Version: 0.1.0
Summary: A great package.A Python library providing useful components for using [Dagster](https://dagster.io/) to create academic research cloud datalakes
Project-URL: Homepage, https://github.com/ncsa/dagster-ncsa
Project-URL: Bug Tracker, https://github.com/ncsa/dagster-ncsa/issues
Project-URL: Discussions, https://github.com/ncsa/dagster-ncsa/discussions
Project-URL: Changelog, https://github.com/ncsa/dagster-ncsa/releases
Author-email: Ben Galewsky <bengal1@illinois.edu>
License-File: LICENSE
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: boto3
Requires-Dist: dagster
Requires-Dist: dagster-aws
Requires-Dist: pyairtable
Provides-Extra: dev
Requires-Dist: pytest-cov>=3; extra == 'dev'
Requires-Dist: pytest>=6; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo>=2023.08.17; extra == 'docs'
Requires-Dist: myst-parser>=0.13; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints; extra == 'docs'
Requires-Dist: sphinx-copybutton; extra == 'docs'
Requires-Dist: sphinx>=7.0; extra == 'docs'
Provides-Extra: test
Requires-Dist: pytest-cov>=3; extra == 'test'
Requires-Dist: pytest>=6; extra == 'test'
Description-Content-Type: text/markdown

# dagster-ncsa

A Python library providing useful components for using
[Dagster](https://dagster.io/) to create academic research cloud data lakes for
the National Center for Supercomputing Applications (NCSA).

## Overview

`dagster-ncsa` extends Dagster's capabilities with specialized tools designed
specifically for academic research workflows and data management at scale. It
provides abstractions and utilities to simplify building, managing, and
monitoring data pipelines in research-oriented cloud data lake environments.

## Components

- **S3ResourceNCSA**: Extends the Dagster S3 resource to add some useful helper
  functions for working with S3 objects in a research data pipeline.
- **AirTableCatalogResource**: A resource for interacting with AirTable tables
  as a catalog for data assets in a research data pipeline.

## Installation

### Basic Installation

```bash
pip install dagster-ncsa
```

### Development Installation

```bash
pip install -e ".[dev]"
```

## Development

### Setup Development Environment

```bash
# Clone the repository
git clone https://github.com/your-organization/dagster-ncsa.git
cd dagster-ncsa

# Install development dependencies
pip install -e ".[dev]
```
