# Changelog

## 0.7.0 - 2026-09-28

### Flows can be Python files

- **Point `flowFile` at a `.py` and the flows are kept as code** - one file,
  every tab, no JSON. The editor is unchanged and still where flows are
  drawn; only what reaches the disk is different, and the suffix is the whole
  of the configuration.
- It runs on its own: `python flow.py` starts the engine and keeps running,
  taking its user directory from the file's own directory.
  `pyflowred check flow.py` reads and validates it without starting anything.
- Nodes are declared and wires are separate statements, so **a cycle needs no
  special syntax** - `attempt[1] >> waited` is written like any forward wire.
  A flow may be deliberately cyclic, and a form that nested the calls could
  not express that; it would also be claiming something untrue, since the
  variable carries no message at run time.
- Three outputs unpack into three names; one output into two nodes is the
  handle used twice; two outputs into one node is two statements, because a
  node has at most one input and none of the seventy-four types has more.
- Tabs, subflows and Function bodies are functions, which is what they are. A
  tab's locals are its node identifiers, so two tabs may each have a
  `readings`. A Function node becomes an `async def` that ruff and pytest can
  reach into.
- A node's identifier survives a rename: the editor changes `name`, never the
  id. Measured through the editor in a browser on an 821-node set - renaming
  a node changes two lines, moving one changes two, adding one changes
  thirteen and removes none, and a deploy with nothing changed changes
  nothing.
- Measured lossless against that same set: 821 nodes, 69 types, 290
  configuration keys, everything identical bar one documented normalisation -
  a `def` cannot carry trailing blank lines, so a Function body that had them
  loses them once and idempotently.
- Credentials stay JSON, encrypted with the instance key. They are not
  anybody's to read in a diff.
- See [docs/porting.md](docs/porting.md#flows-as-python) and
  [examples/python-flows](examples/python-flows/README.md).

### Blocks, files and a body you can compute over

- **`buffer`** holds messages until an interval, a count, a quiet gap or a
  control message closes the block, then emits one message carrying all of
  them in arrival order. The messages are not changed - no field added,
  renamed or reordered, nothing aggregated - because a buffer that altered
  what it held would make the block something other than what was measured.
- What it holds lives in RocksDB, so a part-filled buffer survives a restart
  or a redeploy. `maxMessages` closes early and says so, because a boundary
  that never arrives would otherwise grow until the process is killed.
- **`parquet`** turns the block into a file on the message. The schema is
  read from the block rather than declared: a field only some records carry
  becomes a column with nulls, and a field whose type varies becomes text and
  is reported - guessing from the first row would write every other row wrong
  in silence.
- **`duckdb-store`, `duckdb out` and `duckdb`** are a body of messages you
  can run SQL across. Twenty million rows are 171 MB on disk here and a
  grouped aggregate over all of them takes 34 ms; the same rows held in
  memory instead would cost 919 MB and still need the computation written by
  hand.
- Writing is buffered through RocksDB and the buffer is switchable, because
  feeding DuckDB one row at a time reaches about 1,100 rows a second - below
  what this runtime itself carries. Blocks of five hundred reach a few
  hundred thousand, and at two thousand messages a second the writing takes
  3% of the time.
- Table, row property, SQL and parameters are typed inputs, so a flow can
  work out which table it writes to and assemble a statement elsewhere. A
  table name is checked rather than trusted, because it is allowed to come
  from the message.
- A query returns JSON records and runs in a thread: DuckDB releases the GIL
  for about 99% of a query, so unlike the same work in a Function node it
  does not hold the event loop.
- pyarrow and duckdb are dependencies rather than extras.

### Measured, and written down

- **An IoT ingest benchmark against Node-RED 5.0.7** - `tools/iotflows.py`
  and `tools/iotload.py` - driving the same flow set into both runtimes over
  MQTT and Kafka, steady and in bursts. Throughput turns out to be one over
  the CPU a message costs and nothing else, the fork costs what its branches
  cost, and the gap between the runtimes is raw speed rather than scheduling:
  3x with no work in the flow, 1.06x with half a millisecond, the other way
  round with two.
- Under a burst PyFlowRED stays answerable where Node-RED does not - 479 ms
  against 10,517 ms at the 95th percentile while draining five thousand
  messages - because the same `call_soon` scheduling that stops branches
  running in parallel is what lets anything else in between.
- **`tools/hopprofile.py`** takes a node hop apart. It is 10 us for a Change
  node, and about 40% of it is the Prometheus instrumentation. Turning
  metrics off lifts throughput 38-46%. That is written down as a price list
  rather than a task list: the instrumentation is most expensive exactly
  where it is most needed, and a counter that goes cheap by going approximate
  is worthless in the one situation it exists for.

### Settings

- **The settings file a fresh instance writes is now the reference it should
  have been.** It was thirty-five lines and named nine settings; the runtime
  reads fifty-five. It is now grouped and explained, with everything commented
  out at its default except a small active core - the server, the flow file,
  logging, the health endpoint and the editor theme.
- It lists what *this* runtime reads rather than what Node-RED offers, so
  nothing in it does nothing. `tests/test_settings_template.py` extracts the
  names from the source and fails when one is missing, and the other way round
  when the file promises something nothing reads.
- Written down there for the first time: that `PYFLOWRED_UI_PORT` is how
  `uiPort` is spelled in the environment, the order in which the four
  configuration sources win, and that **`httpNodeResponseTimeout` is seconds
  here and milliseconds in Node-RED**.

## 0.6.0 - 2026-09-27

- **The Function node's editor knows Python.** Completion, hover, signature
  help and problems as you type, from a `pylsp` process the runtime starts
  when somebody first opens the dialog. Both halves at once: `node.`, `msg.`,
  `flow.`, `global_.` and `env.` from stubs **generated by introspecting the
  runtime's own classes**, and `json.`, `hashlib.sha2` or anything the Setup
  tab installed from the language server proper.
- The stubs carry the real signatures and the real docstrings, so twenty-one
  methods on the node and context proxies gained one — they are what the
  hover shows.
- **The 2 MB of Node.js TypeScript declarations are gone**, along with the
  editor code that fetched them. They described `node.send(msg, clone?:
  Boolean)` and `import('util')` — a JavaScript API this runtime does not
  have — and were never reachable anyway, because every model in the Function
  node is a Python model.
- Only the plugins an editor needs are enabled, and only the `pyflakes` extra
  is installed. The formatters and style checkers in
  `python-lsp-server[all]` would report on the generated preamble rather than
  on anything a person wrote.
- `GET <httpAdminRoot>lsp` says whether it is there and running. Without
  `pylsp` the editor says so once and falls back to a plain code box; nothing
  about how a body *runs* depends on any of this.
- `tools/uilsp.py` checks it in a browser, which is the only place it exists.

## 0.5.0 - 2026-09-27

- **A health check, in the product rather than in the repository's tools.**
  `pyflowred health` asks a running instance whether it is keeping up and
  exits 0 healthy, 1 degraded, 2 failing, 3 unreachable; `GET
  <httpAdminRoot>health` returns the same report as JSON with 200 or 503, for
  a Kubernetes probe or an uptime monitor. It reports CPU as a share of one
  core with how much more would fit, event loop lag against the budget,
  throughput with the CPU cost of a message, node errors and backpressure.
  Only *the flows are not running* and *the loop is a second behind* fail the
  probe - a busy core or a flow raising errors warn, because restarting would
  not fix either.
  Rates need a window and a probe cannot wait one, so the sampler that already
  ticks once a second for the lag histogram now writes a small record per
  tick; five minutes are kept and asking costs a lookup.
- Both Dockerfiles health-check the verdict rather than just whether HTTP
  answers, so an unhealthy container means something a restart may fix.
  Building the released `Dockerfile` with `PYFLOWRED_VERSION` older than
  0.5.0 would check an endpoint that version lacks; it is documented in the
  file.
- **A worked example flow per added node type**, offered by the editor under
  *Import > Examples*: sixteen tabs, one for each node PyFlowRED adds beyond
  the ported set. Each is a header saying what the node is for, then one row
  per feature - an inject that produces the input, the node configured for
  that feature, a debug showing what came out - with a comment block beside
  each row saying what to look for. `tools/exampleflows.py` writes them and
  `tests/test_example_flows.py` deploys eleven of them and drives them exactly
  as their comments say to, so those sentences are assertions rather than
  claims.
- **`examples/e2e-stack`** — the end-to-end integration test as a compose
  stack: the application, both brokers, and a one-shot that deploys the flows
  and runs the checks. `docker compose -f examples/e2e-stack/compose.yml up -d`
  and the editor is on <http://127.0.0.1:1881/pyflowred/> with all eighteen
  tabs in it. It reaches 149 of 157 assertions and 95.5% of node types on a
  machine with nothing set up; the whole difference from a host run is the four
  checks that need a Google service account.
- **The stack builds its image, it does not pull one.** `pyflowred:<version>`
  is not on any registry, and a Compose that resolves `image:` by pulling
  stopped with `pull access denied for pyflowred` - on which `docker login`
  looks like the remedy and is not. The service now says `pull_policy: build`,
  and the one-shot deployer uses a public `python:3.13-slim` rather than the
  local tag, so nothing on the stack asks a registry for something it does not
  have.
- **`.gitattributes`, and Dockerfiles that tolerate CRLF.** A checkout made on
  Windows turned `docker/entrypoint.sh` into CRLF, the shebang then named an
  interpreter called `/bin/sh\r`, and the container failed with
  `exec /usr/local/bin/pyflowred-entrypoint: no such file or directory` for a
  file that was plainly there. The repository is now LF in the working tree on
  every platform, and both Dockerfiles strip carriage returns from the
  entrypoint so an image built from an older checkout works anyway.
- The stack runs `tools/e2eflows.py` from a read-only bind mount rather than
  carrying a checked-in `flows.json`, so there is one test flow set and not
  two. The generator now takes the broker addresses from `E2E_MQTT_HOST`,
  `E2E_MQTT_PORT` and `E2E_KAFKA_BROKERS`, which is the only thing about the
  test an environment changes; with the defaults it produces byte-identical
  output to before.

## 0.4.0 - 2026-09-27

Documentation only; no change to any node's behaviour.

- **A node reference, one page per node package**, in `docs/nodes/`. The
  nineteen node types that have no Node-RED counterpart — stream processing,
  Cloud Storage, Kafka, secrets, Prometheus — now have every setting, every
  option value and every message property they read or write written down,
  together with the reasoning wherever the behaviour is not the obvious one.
  Up to now they were described only in `docs/operations.md`, from the
  operator's side, and in the editor's info sidebar.
- **The README says what the package contains.** It listed the added nodes as
  three lines in a table of porting layers, so a reader on PyPI — where the
  README *is* the project page — could not tell that the distribution ships a
  stream-processing engine and four other node packages. There is now a
  section naming all nineteen types and linking to their pages.
- **README links are absolute.** Relative links resolve on GitHub and not on
  PyPI, so every link into `docs/` and the editor screenshot were dead on the
  project page. They point at `github.com`/`raw.githubusercontent.com` now.
- `docs/porting.md` lists the added types beside the ported ones and gains a
  section on stream processing; `docs/operations.md`, `docs/writing-nodes.md`
  and `docs/porting.md` link to the node reference from their matching
  sections. A `Node reference` URL is on the PyPI sidebar.
- Removed a duplicated block in the README's layout listing, where the
  end-to-end test tooling appeared twice — once with its superseded figures.

## 0.3.0 - 2026-09-27

- **An end-to-end integration test** over the node set and its settings, run
  against the service rather than in-process: `tools/e2eflows.py` deploys
  eighteen tabs with 157 assertions, `tools/coverage.py` measures what they
  reached. **66 of 66 countable node types** carry a message during a run
  (97.1% counting the two set aside), and **183 of 191 reachable dialog
  choices** are covered by a passing assertion (95.8%). Everything set aside
  is named with its reason and both figures are printed, so an exclusion
  cannot flatter the result. `tools/e2e/compose.yml` brings up the broker and
  Kafka it wants.
- Defects it found:
  - **Window closing wedged for good** on a timestamp containing the byte
    used as the index separator - about one millisecond in 250. The key was
    split on that separator, the unpack raised, and because the index is
    walked oldest-first the poisoned entry blocked every window behind it
    from ever closing again. It is sliced by offset now.
  - **The Assemble node shadowed `Node._complete`**, the runtime's own method
    for finishing a message, so every message it emitted raised.
  - `kafka out` took the topic and key from the message in preference to the
    dialog, the opposite of the MQTT node and of what its own placeholder
    says.
  - The stream operators showed raw catalogue keys as their node status.
  - A Global Config node carrying coordinates is treated as a placed node
    and, belonging to no tab, dropped - so its environment variables were
    never read.

- **Stream processing.** Seven operators - `aggregate`, `derive`,
  `threshold`, `transition`, `assemble`, `compress`, `pattern` - on a
  `stream-state` configuration node backed by RocksDB. Windows tumble, hop,
  slide, count, session, and close on a control message, which is what a
  shift, a batch, a production order and a cleaning cycle all are. Event time
  with watermarks and a second output for late messages; quality-aware
  aggregation; a key limit that refuses rather than evicts; and
  `GET <httpAdminRoot>stream-state` to see what is held.
  RocksDB is the state store rather than a fallback: 100,000 keys each
  holding a 300-sample window cost 1,193 MB as Python objects and 39 MB
  resident through it, and its slowest hot path is eighteen times the rate
  the runtime can feed it.

- Google Cloud Storage nodes: a `gcs-config` bucket configuration plus
  `gcs in` and `gcs out`, on obstore - the layer the flow store already uses.
  Every operational value is a typed property, so bucket, path, format,
  character set and content type can each be a literal, a message property, a
  flow or global value, an environment variable or a JSONata expression; left
  blank, each falls back to a documented message property. A flow that works
  out where it is writing needs one node, not one per destination.
  Credentials stay on the configuration node, where `${secret:NAME}` reaches
  Secret Manager, and a path cannot climb out of the bucket's prefix.
- `gcs list` and `gcs delete` alongside them. Listing is shallow (objects and
  child folders at one level) or deep, with a limit and a `truncated` flag.
  Deleting takes one object or a whole prefix, and is idempotent.
- Folders work the same with and without a bucket's hierarchical namespace,
  as far as the object API allows: shallow listing reports a folder as a
  folder in either case, while obstore cannot address a name ending in `/`,
  so a prefix delete leaves folder markers behind and reports them in
  `msg.gcs.remainingFolders` rather than claiming a clean sweep.

## 0.2.0

**Everything is now called `pyflowred`.** 0.1.0 shipped under that name on
PyPI but kept `pyred` everywhere else, which meant the distribution and the
thing you imported disagreed. This release makes them one name, and there is
no compatibility shim - upgrading means changing the names below.

| was | is |
|---|---|
| `import pyred` | `import pyflowred` |
| the `pyred` command | `pyflowred` |
| `PYRED_*` environment variables | `PYFLOWRED_*` |
| `[tool.pyred]` in `pyproject.toml` | `[tool.pyflowred]` |
| the `pyred.nodes` entry point group | `pyflowred.nodes` |
| `pyred_*` Prometheus metrics | `pyflowred_*` |
| the default user directory `~/.pyred` | `~/.pyflowred` |

Flows, credentials and project repositories are untouched: the flow file
format has not changed, so moving the user directory across is all an
existing installation needs. A node package has to re-declare its entry
point group to be found.

Also in this release:

- The Google Secret Manager provider could never have worked. The reply
  object handed to google-auth assigned `status`/`headers`/`data` in
  `__init__`, but they are abstract properties on its base class, so the
  class could not be instantiated and every token refresh raised
  `TypeError`. The path is now covered by an offline test as well as the
  one that needs a reachable Secret Manager.
- `POST /flows` with a bare array under API v2 returned an internal
  attribute error as its message. It answers 400 and names the v1 header.

## 0.1.0

- Initial port of the Node-RED 5.0.7 runtime and editor to Python.
- Palette catalogue served from PyPI, so the palette manager's Install tab
  can search for PyFlowRED node packages.
- Multiplayer: two open editors see each other's presence and cursors.
- Projects: git-backed flow storage, with the SSH key store that goes with
  it. Off unless `editorTheme.projects.enabled` is set and git is available.
- Benchmark flows and a load-test harness (`tools/benchflows.py`,
  `tools/loadtest.py`), plus the three defects they found: the HTTP Response
  node replaying an upstream response's headers, JSONata refusing to evaluate
  against an HTTP message, and the HTTP Request node ignoring its keep-alive
  setting.
- A native Prometheus endpoint at `<httpAdminRoot>metrics`: process, instance,
  per-node-type throughput, and backpressure as event loop lag, in-flight
  messages and the queue depth of the nodes that hold messages back. Flows can
  publish their own metrics through the ported `prometheus-metric-config` and
  `prometheus out` nodes, into the same registry.
- The wheel now ships the built editor. It was being excluded because the
  build output is git-ignored, so the package installed and served a blank
  page; the editor source, which is useless at runtime, was shipped instead.
  A build hook refuses to build a wheel without an editor.
- Flows can be kept on object storage. `flowStore` names a URL - GCS, S3,
  Azure, a directory - and obstore syncs to it: the store wins at start-up,
  the local file is pushed on every deploy, and the editor gains an "Object
  storage" menu to force either direction. Settings can now also come from a
  `[tool.pyred]` table in the working directory's `pyproject.toml` and from
  `PYRED_*` environment variables (renamed in 0.2.0).
- Secrets can be read at runtime from Google Secret Manager, or from a
  directory of files as Kubernetes mounts them. `${secret:NAME}` works in any
  node property - resolved when the node is built, so a broker or a Kafka
  client gets its credential at connect time and the flow file keeps the
  placeholder - and a `secret` node puts one on a message for the per-call
  case. Values are cached for `secrets.ttl`, so a rotation needs no rebuild.
- Kafka: a `kafka-broker` configuration node with `kafka in` and `kafka out`,
  the same shape as the MQTT nodes. On confluent-kafka, with any librdkafka
  setting reachable through a properties table. Values are JSON in both
  directions; a record that is not valid JSON raises an error rather than
  travelling on. `tools/kafka/compose.yml` brings up a broker to develop
  against.
- A container image, built in three stages with `uv`: Node builds the editor,
  uv resolves the environment, and the runtime image carries neither. Proxy,
  internal package index and internal npm registry are build arguments. Your
  own CA can be baked in or mounted, and reaches httpx, obstore, Python's ssl,
  certifi and librdkafka - the last of which ignores `SSL_CERT_FILE` and now
  takes its default `ssl.ca.location` from the container's bundle.
- Published on PyPI as **pyflowred**, because `pyred` there belongs to an
  unrelated project. At this version the import package and the command were
  still `pyred`; 0.2.0 renamed them to match.
- A second image. `Dockerfile` installs the released package with uv and
  carries no build tooling; `Dockerfile.source` is the previous one, which
  builds this checkout. Build arguments no longer leak into the runtime, and
  both images carry uv with a writable environment so the palette manager
  works.
