Metadata-Version: 2.1
Name: pipen-gcs
Version: 0.0.2
Summary: A plugin for pipen to handle file metadata in Google Cloud Storage
License: MIT
Author: pwwang
Author-email: 1188067+pwwang@users.noreply.github.com
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: google-cloud-storage (>=2.17.0,<3.0.0)
Requires-Dist: pipen (>=0.15.0,<0.16.0)
Description-Content-Type: text/markdown

# pipen-gcs

A plugin for [pipen][1] to handle files in Google Cloud Storage

## Installation

```bash
pip install -U pipen-gcs

# uninstall to disable
pip uninstall pipen-gcs
```

## Usage

```python
from pipen import Proc, Pipen

class MyProc(Proc):
    input = "infile:file"
    input_data = ["gs://bucket/path/to/file"]
    output = "outfile:file:gs://bucket/path/to/output"
    script = "cat {{in.infile}} > {{out.outfile}}"

class MyPipen(Pipen):
    starts = MyProc
    # input files/directories will be downloaded to /tmp
    # output files/directories will be generated in /tmp and then uploaded
    #   to the cloud storage
    plugin_opts = {"gcs_localize": "/tmp"}

if __name__ == "__main__":
    MyPipen().run()
```

You can also disable localization, then you will have to handle the
cloud storage files yourself.

```python
from pipen import Proc, Pipen

class MyProc(Proc):
    input = "infile:file"
    input_data = ["gs://bucket/path/to/file"]
    output = "outfile:file:gs://bucket/path/to/output"
    script = "gsutil cp {{in.infile}} {{out.outfile}}"

class MyPipen(Pipen):
    starts = MyProc
    plugin_opts = {"gcs_localize": False}

if __name__ == "__main__":
    MyPipen().run()
```

## Configuration

- `gcs_localize`: The directory to localize the cloud storage files. If
  set to `False`, the files will not be localized. Default is `False`.
- `gcs_credentials`: The path to the Google Cloud Service Account
  credentials file.

[1]: https://github.com/pwwang/pipen

