Metadata-Version: 2.4
Name: embeint-htf-station
Version: 0.2.0
Summary: Extensible Python runtime for Embeint hardware test stations
Keywords: hardware-testing,manufacturing,mqtt,test-station
Author: Karl Von Richter
Author-email: Karl Von Richter <karl@embeint.com>
License-Expression: LicenseRef-FSL-1.1-ALv2
License-File: LICENSE.md
Requires-Dist: cryptography>=50.0.1
Requires-Dist: filelock>=3.20
Requires-Dist: aiomqtt>=2.5.1
Requires-Dist: click>=8.3.2
Requires-Dist: pydantic>=2.13.2
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: structlog>=25.5.0
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/Embeint/embeint-htf-station
Project-URL: Documentation, https://github.com/Embeint/embeint-htf-station#readme
Project-URL: Repository, https://github.com/Embeint/embeint-htf-station
Project-URL: Issues, https://github.com/Embeint/embeint-htf-station/issues
Description-Content-Type: text/markdown

# Embeint HTF Station

An extensible Python runtime for Embeint Hardware Test Framework (HTF) stations. It subscribes to commands over MQTT, drives programmers and DUTs, and reports logs and results to a compatible HTF server.

The server and web UI are separate private deployments. This repository contains the station runtime, sample stations, and the public MQTT wire contract only.

## Install

```sh
uv add embeint-htf-station
```

For a development checkout:

```sh
uv sync --all-groups
uv run pytest
```

## Compatibility

The MQTT contract is versioned in [`protocol/asyncapi.yaml`](protocol/asyncapi.yaml). Station releases use semantic versioning and are tagged as `vX.Y.Z`.

- Patch releases do not change the wire contract.
- Minor releases may add backwards-compatible fields or messages.
- Major releases may require a compatible server upgrade.

Pin the station version in an HTF workspace with the `west.yml` manifest supplied by the private server repository. Application projects should depend on released package versions through `uv`/PyPI; use an editable dependency only while developing the runtime itself.

### Publishing a release

The package is published to PyPI from `vX.Y.Z` tags on the long-lived `release`
branch. The tag must match `project.version` in `pyproject.toml`, and its commit
must be reachable from `release`. CI runs on each release-branch push; the tag
workflow reruns checks before building a wheel and source distribution. Only
its separate publish job can request a PyPI identity token.

Before the first release, protect the `release` branch and `v*` tags, require CI
for branch changes, and configure the GitHub `pypi` environment with required
reviewers and deployment limited to `v*` tags. Add a PyPI Trusted Publisher
for owner `Embeint`, repository `embeint-htf-station`, workflow `publish.yml`,
and environment `pypi`. No long-lived PyPI API token is needed.

Create `release` from tested `main` once, then advance it with subsequent
changes from `main`. Update `project.version` for each release and apply any
release-only fixes there. Once release-branch CI passes, tag its tip and push
the tag. For the first release:

```sh
git switch -c release origin/main
git push -u origin release
git tag -a v0.2.0 -m v0.2.0
git push origin v0.2.0
```

Repeat with new versions and tags along the same branch. PyPI distributions
cannot be replaced under the same version. Pin station installations to an
exact version and upgrade deliberately. A GitHub Release can be created from
the published tag for release notes; creating it does not trigger PyPI upload.

## Layout

```
src/embeint_htf_station/
├── cli.py                    # Click entry points
├── config.py                 # Pydantic settings
├── messaging/                # MQTT client and batched logs
├── stages/                   # Reusable stage implementations
├── stations/                 # Station runtimes
└── contracts/                # Generated from protocol/asyncapi.yaml
```

## Samples

```sh
cp samples/basic-station/.env.example samples/basic-station/.env
uv run python samples/basic-station/main.py DUT-001
```

Configuration reads identity and credentials from environment variables, so secrets remain outside source control. Library-provided stages live under `src/embeint_htf_station/stages/`; samples can register their own stage factories.

The `htf-station run` command only maintains a broker heartbeat; use a sample `main.py` as the starting point for a station that executes test plans.

Station-specific task secrets can be configured in the HTF operator UI. With
`HTF_API_KEY` set, the station pulls them from the server at startup and after
reconnecting, then exposes them to built-in and custom stages through
`context.require_secret("INFUSE_API_KEY")` or the read-only `context.secrets`
mapping. The runtime keeps the values in memory and does not add them to run
outputs or logs. A failed authenticated pull stops station startup instead of
running with an old value. Restart or reconnect after changing a secret.

## Station credentials

Create a station from the HTF operator UI and save the one-time `.env` output. Each station has its own MQTT username/password and API key:

```sh
HTF_MQTT_USERNAME=station-...
HTF_MQTT_PASSWORD=st_mqtt_...
HTF_API_KEY=st_api_...
```

Never reuse these values between stations. MQTT credentials are restricted by the broker to the station's own heartbeat, stage, log, and result topics, plus its command subscription. Rotate a credential in the operator UI after exposure; revoked credentials cannot reconnect or access station APIs.

### MQTT session takeover detection

The broker requires the provisioned MQTT username as the station's MQTT client ID. MQTT permits only one active connection for a client ID, so another connection using the same provisioned credentials will disconnect the active station session. This can be caused by a network or broker fault as well as credential reuse; MQTT does not expose enough information to distinguish them reliably.

An established session that ends unexpectedly emits a `broker.disconnected` warning with `possible_session_takeover=true`. Treat that event as a credential-exposure signal: investigate the broker and network logs, then rotate the station credential in the operator UI when reuse cannot be ruled out.

## Contract generation

The generated MQTT models are committed to the package. Regenerate them after changing the AsyncAPI contract:

```sh
./scripts/gen-mqtt-contracts.sh
git diff --exit-code -- src/embeint_htf_station/contracts
```

## MQTT TLS and station certificates

New configurations default to verified TLS on port 8883. The production endpoint
is `mqtt.app.embeint-htf.com:8883` and requires a unique station client certificate
**and** the station MQTT username/password. Download the one-time provisioning
bundle from the HTF station setup page. Keep its private key readable only by the
station service account (for example `chmod 600 client.key`). Never commit it.

```yaml
mqtt:
  transport: mtls
  host: mqtt.app.embeint-htf.com
  port: 8883
  client_cert: ./credentials/client.pem
  client_key: ./credentials/client.key
  # Optional custom broker trust; omit to use the system public CA store.
  # ca_cert: ./credentials/broker-ca.pem
station:
  org_id: ${HTF_ORG_ID}
  station_id: ${HTF_STATION_ID}
```

`HTF_MQTT_TRANSPORT`, `HTF_MQTT_HOST`, `HTF_MQTT_PORT`, `HTF_MQTT_USERNAME`,
`HTF_MQTT_PASSWORD`, `HTF_MQTT_CA_CERT`, `HTF_MQTT_CLIENT_CERT`, and
`HTF_MQTT_CLIENT_KEY` override their YAML values. Adjacent `.env` values fill
missing process environment variables. Certificate paths in YAML or its overrides
resolve relative to the YAML file. With the environment-only CLI, paths resolve
relative to the working directory. Legacy `HTF_BROKER_*` names remain supported
by the environment-only CLI; prefer `HTF_MQTT_*` for new installations.

Modes are `tls` (verified server TLS, optional paired client certificate/key),
`mtls` (requires both client files), and `plaintext` (explicit development only).
Local samples select plaintext; set both transport and port when switching an
existing sample to production. For an environment-only local development run,
set `HTF_MQTT_TRANSPORT=plaintext` and `HTF_MQTT_PORT=1883`.
There is no insecure TLS mode. A client trust CA issued for stations is not
necessarily the CA that signed the broker certificate.

Missing/unreadable PEM files and mismatched/encrypted keys stop startup before
MQTT connects. TLS handshake failure prevents the station from accepting runs.
For rejected connections check the broker hostname, system clock, broker CA
chain, client chain/expiry and MQTT credentials. Do not share keys or passwords
in support logs. To rotate, replace the certificate/key pair atomically while
the station is stopped, restart to load them, verify a heartbeat, then revoke the
old certificate. Existing contexts are not changed by overwriting files.

### Automatic renewal

Updated HTF servers support station-owned automatic certificate renewal. It is
enabled by default for `mtls`; set `HTF_MQTT_AUTO_RENEW=false` to opt out. Keep
`HTF_API_KEY` and a verified HTTPS `HTF_API_BASE_URL` configured. The station
does not need an OpenBao token. Initial certificate installation is still manual.

The runtime checks hourly while idle and renews seven days before expiry (or in
the last third of a shorter certificate lifetime). Active and queued jobs defer
renewal, so allow an idle interval before expiry. It generates a fresh private
key locally, proves possession of its current key, and requests a replacement
through the HTF API. Network failures retry with bounded backoff without
discarding the current pair. Revoked/expired certificates require admin recovery.

The certificate's parent directory must be writable by the station account.
The private `.htf-mtls-<station UUID>` directory stores pending requests and
versioned key/certificate pairs, using owner-only permissions on POSIX. Keep
this directory persistent across restarts, private, and out of Git/support logs.
An atomic pointer activates the replacement and MQTT reconnects between jobs.
The server retires the old certificate after the overlap (normally 24 hours).
Original enrollment files are preserved. For manual re-enrollment, stop the
station and securely archive the old renewal directory before installing the
new bundle. Do not run two station processes with the same identity.

Look for `station.certificate.renewed` or `station.certificate.renewal_failed` in
station logs. The GUI shows the new certificate's expiry and the old one's
retirement time. Older server deployments will reject renewal until upgraded;
do not assume installing only the station update enables end-to-end renewal.

Run `uv run pytest` for unit tests. Run
`HTF_TLS_INTEGRATION=1 uv run pytest tests/test_tls_integration.py` with Docker
available for real Mosquitto mTLS tests (also required by CI).

A complete production transport sample is in `samples/production-mtls/config.yaml`.
After installing the credentials and setting station environment variables, run
`uv run python samples/basic-station/main.py --listen --config samples/production-mtls/config.yaml`.

## Project ID pools

Project variables use uploaded CSV pools. Configure `HTF_API_KEY` and
`HTF_API_BASE_URL`, then reserve values, use them, and explicitly commit after
all required work succeeds:

```yaml
stages:
  - name: Reserve IDs
    kind: reserve_variables
    variables: [infuse_id, serial_number]
    record_version: v2

  - name: Show ID
    kind: print
    message: 'ID=${provisioning.infuse_id}'
    wait_seconds: 0

  - name: Register with external service
    kind: http_request
    http:
      url: https://manufacturing.example.com/devices
      method: POST
      headers:
        Authorization: 'Bearer ${MANUFACTURING_TOKEN}'
      json:
        dut_id: '${dut_id}'
        infuse_id: '${provisioning.infuse_id}'
        serial_number: '${provisioning.serial_number}'
      timeout_seconds: 30
      expected_statuses: [200, 201]
      outputs:
        external.receipt: receipt.id

  - name: Program reserved ID
    kind: infuse_provisioning
    provisioning_source: id_pool
    programmer: jlink_1
    uicr:
      - name: infuse_id
        source: context
        value: provisioning.infuse_id
        bytes: 8
        endian: LSB

  - name: Commit IDs
    kind: commit_variables
    variables: [infuse_id, serial_number]
```

Replace the example service URL and response mapping with your service contract.
`jlink_1` must be a local programmer with a target device that defines its UICR
address; alternatively supply an explicit UICR address. Hardware programming is
optional: use any required stages between reservation and commit. The same stage
syntax works inside named lane plans.

`reserve_variables` (also available as `allocate_variables`) **only reserves**.
Reservations are exclusive to the project/DUT/variable and never expire or
release automatically. The server handles concurrent stations atomically.
Re-requesting the same DUT returns its reserved or committed values, even if the
pool is otherwise empty. Additional variables in a later configuration version
are reserved without replacing existing committed values. Values stay strings,
including leading zeros.

Reservation writes `provisioning.<variable>` and `reservation.<variable>` into
this run's context. `${provisioning.infuse_id}`, `${external.receipt}`, `${dut_id}`,
and `${run_id}` can be used in string settings, including nested HTTP JSON and
headers. `${context.provisioning.infuse_id}` is an equivalent explicit context
reference. References resolve once immediately before each stage; a missing
reference fails the stage. Stage names, kinds, programmer routing, variable
lists, locks and dependency declarations remain static. Numeric configuration
fields such as timeout and width must be literal numbers. Uppercase `${ENV_VAR}`
is the existing configuration-time environment expansion; runtime values use
lowercase/dotted names. Python stages can still use
`context.get_output_value("provisioning.infuse_id")`.

`http_request` supports GET/POST/PUT/PATCH/DELETE, explicit headers, optional JSON,
timeout up to 120 seconds, expected status codes, and response outputs mapped
from dotted JSON paths (including array indexes). Outputs must be strings or
integers; response JSON is limited to 1 MiB when extracting outputs. The station
credential is never automatically sent to an external service. Requests do not
follow redirects or retry automatically. The stage supplies an `Idempotency-Key`
derived from DUT, stage name and reservation IDs unless you provide one. It is
stable across retries for the same reservations, and changes after explicit
release/re-reservation. **Your service must implement idempotency** for this to
prevent duplicate side effects; keep the stage name, request and reserved
variable set stable when retrying.

A timeout may mean the external service consumed the ID. The run fails and keeps
the reservation. Reconcile the external outcome before rerunning side effects,
committing, or releasing; never release just because a request timed out. After
confirmed external success, an operator can use an approved recovery plan that
reserves the same DUT, verifies completion and commits without repeating that
side effect. Commit timeouts can be retried with the same reservation IDs.

Put `commit_variables` after every required operation. It commits exactly the
listed variables using their reservation IDs and is retry-safe. A missing,
released or replaced reservation is rejected, and a repeated commit preserves
the original timestamp/station. Plans containing a commit stop after any failed
stage. Re-reserving a variable within one run must return the same token and
value; a changed pair fails the run before it can be committed. Commit also
rejects failed prerequisites and changed context values. Every stage in a plan
that reserves IDs checks its declared dependencies, including standalone runs.
Same-lane dependencies use stages already completed in this run. A cross-lane
dependency without a batch result fails before that stage runs, so registration
cannot proceed without its verification prerequisite. Every dependency in a
reservation plan must declare and reach `passed`; a declared `failed` outcome
cannot authorize an external side effect or commit.
Only declared cross-lane dependencies are checked; independent lanes keep their
own contexts.

Admin release in the project's **DUT records & IDs** history is the only way to
make a reserved or committed value reusable. History retains reservation,
commit and release attribution. No failure, abort, disconnect or reimport clears
an ID.

`infuse_provisioning` with `provisioning_source: id_pool` can also reserve its
requested keys directly, but still needs a subsequent `commit_variables` stage.
Explicit `provisioning.hardware_id` references select the pool even when a chip
hardware ID exists. Context-only/literal UICR writes make no allocation request.
The default `infuse_api` retains the existing Infuse workflow; it is a separate
data source and does not automatically import legacy IDs into project pools.

Deploy the companion server and its reservation migration before this station
change, and update pool plans to include explicit commit. Existing permanent
assignments remain committed during migration. This change does not publish or
deploy automatically.

## License

Licensed under the Functional Source License, Version 1.1, ALv2 Future License
(FSL-1.1-ALv2). See [LICENSE.md](LICENSE.md) for the full terms.
