Metadata-Version: 2.4
Name: elt-dagster-meltano
Version: 1.0.0
Summary: A dagster plugin that allows you to run your Meltano project inside Dagster.
License: Apache 2.0
License-File: LICENSE
Author: Camis Inc
Requires-Python: >=3.9,<3.14
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: dagster (>=1.10,<2)
Project-URL: Homepage, https://github.com/camisinc/elt-dagster-meltano
Description-Content-Type: text/markdown

# Dagster-Meltano

A dagster plugin that allows you to run Meltano using Dagster. Forked from the original repo that was from [Quantile Devlopment](https://github.com/quantile-development/dagster-meltano); with a special thanks to [Jules Huisman](https://github.com/JulesHuisman) for the maintenance of that repo.

[![Downloads](https://pepy.tech/badge/elt-dagster-meltano/month)](https://pepy.tech/project/elt-dagster-meltano)

## Installation

You can install using `pip install elt-dagster-meltano`.

## Examples

An example of automatically loading all jobs and schedules from your Meltano project.

```python
from dagster import repository
from dagster_meltano import load_jobs_from_meltano_project

meltano_jobs = load_jobs_from_meltano_project("<path-to-meltano-root>")

@repository
def repository():
    return [meltano_jobs]
```

An example of running an abitrary `meltano run` command.

```python
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]
```

You can inject Meltano config with the following Dagster config.

```yaml
ops:
  tap_smoke_test_target_jsonl:
    config:
      env:
        TAP_SMOKE_TEST_STREAMS: '[{"stream_name": "new-stream", "input_filename": "demo.json"}]'
```

An example of running an arbitrary Meltano command.

```python
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_command_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_command_job():
    meltano_command_op("install loader tap-smoke-test")()

@repository()
def repository():
    return [meltano_command_job]
```

## Development using VSCode

1. Open this repository in Visual Studio Code.
2. Install the [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) plugin for Visual Studio Code.
3. Go to the example Meltano project root `cd meltano_project`
4. Install all plugins `meltano install`
5. Start dagit `meltano invoke dagster:start`
6. Visit `localhost:3000` to access Dagit.

