Metadata-Version: 2.1
Name: shub-workflow
Version: 1.14.43
Summary: Workflow manager for Zyte ScrapyCloud tasks.
Home-page: https://github.com/scrapinghub/shub-workflow
Maintainer: Scrapinghub
License: Apache-2.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=3.12
Requires-Dist: scrapinghub[msgpack]>=2.3.1
Requires-Dist: jinja2>=2.7.3
Requires-Dist: sqlitedict==2.1.0
Requires-Dist: boto3>=1.9.92
Requires-Dist: bloom-filter2==2.0.0
Requires-Dist: collection-scanner
Requires-Dist: tenacity
Requires-Dist: typing-extensions
Requires-Dist: scrapy
Requires-Dist: prettytable
Requires-Dist: jmespath
Requires-Dist: timelength
Provides-Extra: with-s3-tools
Requires-Dist: s3fs>=0.4.0; extra == "with-s3-tools"
Provides-Extra: with-gcs-tools
Requires-Dist: google-cloud-storage>=1.38.0; extra == "with-gcs-tools"

A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

# Installation

```
pip install shub-workflow
```

If you want to support s3 tools:

```
pip install shub-workflow[with-s3-tools]
```

For google cloud storage tools support:

```
pip install shub-workflow[with-gcs-tools]
```

# Usage

Check [Project Wiki](https://github.com/scrapinghub/shub-workflow/wiki) for documentation. You can also see code tests for lots of examples of usage.

# Claude Code plugin

shub-workflow ships a [Claude Code](https://claude.com/claude-code) plugin,
**shub-workflow-toolkit**, that gives Claude working knowledge of shub-workflow tooling. It
currently bundles eight skills:

- **scanjobs-programs** — authoring and running the `scanjobs` job-scanning + plotting tool and its
  command-line "programs".
- **shub-workflow-scripts** — writing or fixing scripts built on the `shub_workflow.script` base
  classes (`BaseScript` / `BaseLoopScript` / `BaseLoopScriptAsyncMixin`), i.e. any script that runs
  on or operates on Scrapy Cloud.
- **shub-workflow-crawl-managers** — building, updating or understanding crawl managers
  (`CrawlManager` / `PeriodicCrawlManager` / `GeneratorCrawlManager` / `AsyncSchedulerCrawlManagerMixin`):
  the `set_parameters_gen()` pattern, outcome/retry hooks, and async scheduling.
- **shub-workflow-graph-managers** — building, updating or understanding graph managers
  (`GraphManager` + `Task` / `SpiderTask`): declaring a DAG of tasks in `configure_workflow()`,
  dependency linking, `on_finish`/retry routing, resources, and parallelization.
- **shub-workflow-monitors** — building, updating or understanding monitors (`BaseMonitor`):
  cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold
  alerts via Slack / Sentry.
- **shub-workflow-fshelper** — reading/writing/listing files through the cloud-agnostic filesystem
  layer (`shub_workflow.utils.futils` / `FSHelper`): prefix-based `s3://`/`gs://`/local dispatch,
  role-assumed credentials, the listing variants, and `op_kwargs`/ACLs.
- **shub-workflow-issuers** — building data-pipeline issuers (`IssuerScript` /
  `IssuerScriptWithFileSystemInput` / `IssuerScriptWithSCJobInput`): reading job/file input,
  dedup, batch output slots, and issuer-based delivery (replacing the deprecated `BaseDeliverScript`).
- **shub-workflow-diagrams** — generating a project's architecture diagrams from an editable model:
  banded workflow/pipeline diagrams and UML class-hierarchy diagrams, in the house style shared by
  these projects, with the rendering engine to copy and the Graphviz constraints that keep the
  output stable.

Install it from this repository's plugin marketplace, from inside Claude Code:

```
/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow
```

To enable it automatically for a project, add it to that project's `.claude/settings.json`:

```json
{
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}
```

## Updates

The plugin is **unversioned** (its `plugin.json` has no `version` field), so each commit pushed to
this repository is a new version. When Claude Code installs the plugin it **copies it into a local
cache** (`~/.claude/plugins/cache/`) and uses that copy — it does **not** read your working tree or
re-pull from GitHub on every session. You choose how new commits reach you:

- **Automatic.** Turn on auto-update for this marketplace: run `/plugin`, open the **Marketplaces**
  tab, and enable auto-update for `shub-workflow` (or set it in settings — see below). With this on,
  Claude Code re-pulls the marketplace from GitHub and updates installed plugins **at startup**, so a
  new session always loads the latest pushed commit. This is the low-friction option for staying
  current.

  ```json
  {
    "extraKnownMarketplaces": {
      "shub-workflow": {
        "source": { "source": "github", "repo": "scrapinghub/shub-workflow" },
        "autoUpdate": true
      }
    },
    "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
  }
  ```

- **Manual.** Leave auto-update off (the default for third-party marketplaces). The cached copy stays
  pinned until you explicitly update — nothing changes under you between sessions. To pull the latest
  when you want it:

  ```
  /plugin marketplace update shub-workflow            # refresh the catalog from GitHub
  /plugin update shub-workflow-toolkit@shub-workflow  # update the installed plugin
  ```

The plugin lives in [`plugins/shub-workflow-toolkit/`](plugins/shub-workflow-toolkit); the
marketplace manifest is [`.claude-plugin/marketplace.json`](.claude-plugin/marketplace.json).

# Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used
for setting up a development environment for shub-workflow library development and testing.


# For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

```
> pipenv install --dev
> cp pre-commit .git/hooks/
```

for installing the environment, and:

```
> pipenv shell
```

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do `git commit` (provided
you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

```
> ./lint.sh
```
