Metadata-Version: 2.4
Name: sluice-pipeline
Version: 0.4.0
License: MIT License
        
        Copyright (c) 2025 DengQi
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML<7,>=6.0
Requires-Dist: httpx[socks]<1,>=0.28
Requires-Dist: beautifulsoup4<5,>=4.12
Requires-Dist: defusedxml<1,>=0.7
Requires-Dist: markdown-it-py<5,>=4.2
Requires-Dist: html-to-markdown<4,>=3.10
Provides-Extra: web
Requires-Dist: curl_cffi<1,>=0.16; extra == "web"
Requires-Dist: trafilatura<3,>=2; extra == "web"
Requires-Dist: newspaper4k<1,>=0.9; extra == "web"
Requires-Dist: resiliparse<1,>=0.16; extra == "web"
Provides-Extra: ui
Requires-Dist: Flask<4,>=3.0; extra == "ui"
Provides-Extra: prefect
Requires-Dist: prefect<4,>=3.0; extra == "prefect"
Provides-Extra: test
Requires-Dist: pytest<9,>=8.0; extra == "test"
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.0; extra == "dev"
Requires-Dist: ruff<1,>=0.8; extra == "dev"
Requires-Dist: ty>=0.0.1; extra == "dev"
Dynamic: license-file

English | [简体中文](README.zh_CN.md)

# Sluice v4

[![CI](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/ci.yml)
[![Docker](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/docker.yml/badge.svg)](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/docker.yml)
[![Publish to PyPI](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/publish.yml/badge.svg)](https://github.com/nerdneilsfield/sluice-pipeline/actions/workflows/publish.yml)
[![Python 3.12+](https://img.shields.io/badge/python-%3E%3D3.12-blue)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

Sluice collects RSS and webpage content, runs YAML-defined task graphs, and delivers results through independently retried Sinks. SQLite stores publications, execution state, shared Stage results, and Delivery attempts. Local commands and the optional Prefect flow use the same runtime.

Typical uses are watching a webpage for changes, forwarding RSS entries to Telegram, email, or Feishu, and sending an LLM-written digest of a feed. Each use is a YAML definition that you run with the `sluice` command.

This branch is the v4 clean-room implementation. Use a separate database rather than pointing it at an older Sluice deployment. The runtime uses `fcntl` file locking and targets Unix-like systems; only one App can own a database at a time. See [Operate Sluice](docs/operations.md).

## Install

Sluice needs Python 3.12 or later. Choose one of the three routes below.

### From source

Install [uv](https://docs.astral.sh/uv/), then run this command from the repository root:

```sh
uv sync --locked --extra dev --extra web --extra ui --extra prefect
```

The command installs `sluice` as `.venv/bin/sluice`. The examples below use that path.

### With pip

The distribution is named `sluice-pipeline`; the Python import and command are `sluice`. To install into the current Python environment, run `pip install '.[web,ui,prefect]'` from the repository root, or pass the checkout path instead of `.`. Omit extras you do not use. Email, Telegram, and LLM integrations need their own configuration and credentials; the console example does not exercise them.

The release workflow publishes `sluice-pipeline` to PyPI when a `v*.*.*` tag is pushed. This repository does not record which versions are currently on PyPI, so install from the checkout as shown above rather than assuming a published v4 release.

### With Docker

Build the image from the repository root and check that it starts:

```sh
docker build -f scripts/docker/Dockerfile -t sluice-pipeline:local .
docker run --rm sluice-pipeline:local --help
```

The release workflow pushes `ghcr.io/nerdneilsfield/sluice-pipeline` and `nerdneils/sluice-pipeline` when a `v*.*.*` tag is pushed. This repository does not record which tags are currently published; the local build above is the verified route.

The image runs `sluice` as its entrypoint from `/app` and includes the `web`, `ui`, and `prefect` runtime extras. The directories `/app/configs` and `/app/data` exist for definitions and state. The default `database` path is relative to the YAML file, so for a read-only definition mount, set an absolute `database` such as `/app/data/state.db` in the YAML:

```sh
docker run --rm \
  -v "$PWD/configs:/app/configs:ro" \
  -v "$PWD/data:/app/data" \
  sluice-pipeline:local tick /app/configs/definition.yml \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
```

Replace `configs/definition.yml` with your definition. The image sets no `USER`, so the container writes files in the mounted state directory as root. A definition that uses a custom Sink or other dotted-path callable needs that module importable inside the container, for example by mounting it under `/app/configs` and adding `-e PYTHONPATH=/app/configs`.

## Run the webpage example

From the repository root, after installing from source:

```sh
PYTHONPATH="$PWD" .venv/bin/sluice tick examples/webpage.yml \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
```

The example fetches `https://example.com/`, extracts visible content, converts it to text, and prints a JSON object containing `delivery_key` and `payload`. It does not send email or Telegram messages. Internet access is required. `PYTHONPATH` makes the example's custom Sink, `examples.console.send`, importable; it is not needed for definitions using only installed built-ins.

State is stored in `examples/webpage.db`, relative to the YAML file. To inspect the result without relying on the CLI exit status:

```sh
.venv/bin/python - <<'PY'
import sqlite3
with sqlite3.connect('examples/webpage.db') as db:
    print(db.execute('SELECT kind, state, error FROM executions').fetchall())
    print(db.execute('SELECT state FROM deliveries').fetchall())
PY
```

On a successful first run, Source, Stage, and Delivery states are `succeeded`. The CLI can return normally after recording an execution failure, so check these states when diagnosing a run. The Source has a one-hour schedule; an immediate second tick need not fetch again. Unchanged content does not create another delivery.

## Pipeline concepts

[examples/webpage.yml](examples/webpage.yml) contains an explicit HTTPX provider, fetch chain, extractor chain, Source, Stage, and console Delivery. YAML `version: 1` is the definition format, not the package major version.

- `sources` collect immutable Items and publish changes. Built-ins include `sluice.sources.rss` and `sluice.sources.webpage`.
- `stages` run `each` or `batch` task chains.
- `pipelines` connect Sources and Stages through `from` edges. Every Pipeline needs a Delivery endpoint.
- `web` declares named providers and ordered fallback chains. HTTPX and BeautifulSoup work with core dependencies; the `web` extra adds curl_cffi, trafilatura, newspaper4k, and resiliparse.

Direct web providers reject private and loopback targets to prevent SSRF. Hosted service endpoints have a separate policy. Keep credentials out of YAML values; provider credential fields accept environment references such as `api_key: env:FIRECRAWL_API_KEY`.

For Crawl4AI, use the [0.9.x configuration guide](docs/crawl4ai.md); older API formats are not supported.

Rules for task chains and transient artifacts, the authored binding identity, and the extension contracts, including dotted-path callable requirements, are in [Write a definition](docs/configuration.md).

## Scenario examples

**Webpage.** [examples/webpage.yml](examples/webpage.yml) fetches a page, extracts its text, and prints it through a console Sink. It is the example run above.

**RSS notifications.** RSS 2.0, Atom, and RSS 1.0 feeds can deliver to Telegram, SMTP email, and Feishu custom bots using built-in senders. See [notification setup](docs/notifications.md) and [the three-channel RSS example](examples/rss-notifications.yml). This example sends real messages and needs your credentials; the webpage example above only prints to the console.

**LLM digest.** [examples/rss-llm-digest.yml](examples/rss-llm-digest.yml) runs RSS → fetch → `llm_extract` → `llm_summary` → console, using one OpenAI-compatible `/chat/completions` endpoint configured in a top-level `llm` block. See [LLM digest setup](docs/llm.md).

## Common commands

`sluice` is a one-shot runner, not a background scheduler. Supply an ISO-8601 timestamp with a timezone for execution commands:

```sh
sluice tick definition.yml --at 2026-09-22T00:00:00Z
sluice reconcile definition.yml pipeline_name node_name --at 2026-09-22T00:00:00Z
sluice rebuild definition.yml 42 --at 2026-09-22T00:00:00Z
sluice retry-delivery definition.yml DELIVERY_KEY --at 2026-09-22T00:00:00Z
sluice ui definition.yml --host 127.0.0.1 --port 8000
```

Replace `definition.yml`, Pipeline/Node names, execution ID `42`, and `DELIVERY_KEY` with values from your deployment. Retrying a Delivery can repeat an external side effect if the Sink did not honor the stable Delivery key. The management UI has mutation actions and no authentication; keep it on a trusted local interface.

[Operate Sluice](docs/operations.md) covers scheduling `tick` with cron, rebuild and retry semantics, the UI, Prefect, the database lock, and inspecting state.

## Contributing

Contributions are welcome. [CONTRIBUTING.md](CONTRIBUTING.md) describes the development setup, the verification commands, the CI matrix, and the release process. [v4 verification](docs/v4-verification.md) records the checked scope and remaining limits.
