Metadata-Version: 2.5
Name: sekoia-hwl
Version: 0.11.1
Summary: Sekoia HTTP Workflow Language: write, validate and run integrations locally
Project-URL: Homepage, https://www.sekoia.io
Project-URL: Documentation, https://docs.sekoia.io
Author: SEKOIA.IO
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.13
Requires-Dist: click<9.0.0,>=8.2.1
Requires-Dist: defusedxml<0.8.0,>=0.7.1
Requires-Dist: google-re2<2.0.0,>=1.1.20251105
Requires-Dist: httpx2<3.0.0,>=2.16.0
Requires-Dist: iregexp-check<0.2.0,>=0.1.4
Requires-Dist: jsonpath-rfc9535<3.0.0,>=2.0.0
Requires-Dist: jsonschema<5.0.0,>=4.24.0
Requires-Dist: minijinja<2.22.0,>=2.20.0
Requires-Dist: orjson<4.0.0,>=3.12.0
Requires-Dist: pydantic<3.0.0,>=2.12.0
Requires-Dist: pyjwt[crypto]<3.0.0,>=2.10.0
Requires-Dist: python-dateutil<3.0.0,>=2.9.0
Requires-Dist: pyyaml<7.0.0,>=6.0.2
Requires-Dist: structlog<27.0.0,>=25.4.0
Requires-Dist: tenacity<10.0.0,>=9.1.2
Description-Content-Type: text/markdown

# sekoia-hwl

`sekoia-hwl` is a library and command-line tool for the Sekoia HTTP Workflow Language (HWL). It validates YAML
documents, and runs workflows and connectors, so that an integration can be written and tested locally
before it is deployed on Sekoia.io:

| Kind | Purpose |
|---|---|
| `workflow` | A pipeline of HTTP, GraphQL and output stages (a document without `kind` is a workflow) |
| `connection` | An API: its account fields (`config`), base URL, auth method, HTTP options and query `engine` |
| `table` | A table queryable with SOL, backed by an API (filters, pagination, hydration, aggregations) or by a query in the language of the connection's engine |
| `connector` | A collector of events or assets (checkpoint, window, events pipeline, asset mapping) |
| `dialect` | How SOL commands are written in a query language |

Table and dialect documents are validated here; they are queried on Sekoia.io. The library has no connection to
Sekoia.io: collected events and workflow outputs are printed on stdout, accounts come from a local file and connector
state lives in a local directory.

## Installation

```bash
pip install sekoia-hwl
```

The `hwl` command is installed with the package. The older `shwl-runner` and `shwl-validator` commands are installed
too, as aliases of `hwl run` and `hwl validate`.

## Command line

`run`, `validate` and `connector run` share their exit codes: 0 success, 1 invalid documents, 2 file not found, 3
invalid file (not readable, not YAML, an invalid accounts or state file), 4 runtime error. Usage errors (an unknown
option, a missing argument) exit with 2, as in every click command. `connector run` prints records only on stdout,
one JSON object per line; logs and warnings go to stderr.

### Validate documents

```bash
hwl validate workflow.yml                         # one workflow, human-readable summary
hwl validate --json workflow.yml                  # {"valid": ..., "errors": [...]}
hwl validate documents/ more/connection.yml       # several files and directories, validated as one set
hwl validate --jsonl documents/                   # one diagnostic per line; nothing at all when valid
```

### Run a workflow

```bash
hwl run workflow.yml -i query=sekoia.io -i limit=10
hwl run workflow.yml --inputs inputs.yaml -i limit=10   # --input values win over the file
```

The workflow outputs are printed as one JSON object. Validation diagnostics go to stderr; `--no-validate` skips
validation and `--strict-validation` also reports W101 warnings.

### Run a connector

```bash
hwl connector run example_events --documents documents/ --accounts accounts.yml
hwl connector run example_assets --documents documents/ --accounts accounts.yml --account eu --follow
```

The connector's checkpoint, deduplication keys and rotated secrets are stored in
`<state-dir>/<connector>/<account>.json` (`--state-dir`, default `.hwl-state`). With `--follow`, a run that emitted
something is followed immediately by another one; otherwise the next run waits for `schedule.every`.

## Documents

A connection declares the fields of an account and how to authenticate:

```yaml
hwl: "1.1"
kind: connection
name: example
config:
  base_url:  {type: string, format: uri}
  api_token: {type: string, secret: true}
base_url: "{{ config.base_url }}"
auth:
  bearer: {token: "{{ config.api_token }}"}
```

Tables and connectors reference their connection by name. The JSON Schema of each kind, shipped in
`sekoia_hwl/schemas/`, describes every key; editors that read JSON Schemas complete and check documents with them.

In the new kinds, a string that is exactly one `{{ expression }}` evaluates to the expression's native value
(a datetime, a list, a number...), other templated strings are rendered to text, and request parameters, headers,
cookies and JSON body members whose value is null are omitted. Paths such as `$.data[*].id` are RFC 9535 JSONPath.

## Accounts file

```yaml
connections:
  example:
    - name: eu
      config: {base_url: "https://eu.api.example.com", api_token: "${EXAMPLE_EU_TOKEN}"}
    - name: us
      config: {base_url: "https://us.api.example.com", api_token: "${EXAMPLE_US_TOKEN}"}
```

`${VAR}` is replaced from the environment (a missing variable is an error), and each `config` is validated against
the connection's `config`. Fields declared with `secret: true`, workflow inputs declared with `secret: true` and all
credentials are masked (`***`) in logs, errors, warnings and command output.

## Python API

Run a workflow:

```python
from sekoia_hwl.workflow.configuration import Configuration
from sekoia_hwl.workflow.runner import WorkflowRunner

runner = WorkflowRunner(definition, Configuration())
outputs = runner.run({"query": "sekoia.io"})  # from synchronous code
outputs = await runner.arun({"query": "sekoia.io"})  # from a running event loop
```

Validate documents, then run a connector once:

```python
import sys
from pathlib import Path

from sekoia_hwl.accounts import load_accounts, resolve_config
from sekoia_hwl.connectors import ConnectorRunner, JsonLinesSink, StateStore
from sekoia_hwl.validation.documents import load_document_set

# Raises InvalidDocumentsError (with the diagnostics of every file) when a document is invalid
documents = load_document_set([Path("documents")])
connector = documents.connectors["example_events"]
connection = documents.connection_of(connector)
account = load_accounts(Path("accounts.yml")).account(connection.name, "eu")
runner = ConnectorRunner(
    connector,
    connection,
    account.name,
    resolve_config(connection, account),
    sink=JsonLinesSink(sys.stdout.buffer),
    state=StateStore.for_account(Path(".hwl-state"), connector.name, account.name),
)
result = await runner.run_once()  # RunResult(emitted=..., warnings=[...])
```

`sekoia_hwl.validation.documents.validate_documents` returns per-file diagnostics without raising, and
`python -m sekoia_hwl.schemas` regenerates the JSON Schemas of the new kinds in `sekoia_hwl/schemas/`.

## Extensions

Other distributions add `hwl` commands through the `sekoia_hwl.commands` entry point group (a click command, named
after its entry point) and rules across a document set through `sekoia_hwl.set_rules` (a callable taking a
`sekoia_hwl.validation.documents.SetContext` and yielding `(SourceDocument, ValidationError)` pairs).

## Development

```bash
uv sync --all-extras --dev
uv run pytest --keepalive tests
uv run ruff check && uv run ruff format --check
uv run mypy sekoia_hwl tests
```

## License

Apache-2.0, see [LICENSE](LICENSE).
