Metadata-Version: 2.4
Name: pj-bridge
Version: 1.4.0
Summary: Bridge a delimiter-framed binary stream into JSON over WebSocket for PlotJuggler (or stdout if file).
Author: Carlos Asmat, Gene Gelfand
License: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: websockets<14.0,>=12.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.3.0; extra == "dev"
Requires-Dist: ruff>=0.6.8; extra == "dev"
Requires-Dist: mypy>=1.10.0; extra == "dev"
Requires-Dist: bandit>=1.7.9; extra == "dev"
Requires-Dist: pytest>=8.2.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: uv>=0.4; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Requires-Dist: types-setuptools; extra == "dev"

# pj-bridge

Bridge a delimiter-framed TCP binary stream into JSON over WebSocket for [PlotJuggler](https://github.com/facontidavide/PlotJuggler) or stdout (if file).

**Mode:** PlotJuggler runs the **WebSocket Server**. The bridge connects as a **WebSocket client** and pushes JSON messages.

## What’s in this repo

- `derive_struct.py` — Parse a C header (`typedef struct { ... } Name;`) and derive the Python `struct` format and expanded field labels.
- `stream_parser.py` — Connect to the device (or read a `.bin` file), parse **[DE AD BE EF][COUNT][MSG_ID][PAYLOAD] × COUNT** batches into NDJSON (one JSON per line), or into CSV with `--csv`.
- `socket_client.py` — Read NDJSON (stdin or file) and forward to PlotJuggler’s WebSocket Server.
- `bridge.py` — One-process solution: connect to device, parse, and forward to PlotJuggler (no shell pipes needed) or stdout (if file, as NDJSON or CSV with `--csv`).
- `streamer.py` — Library API (`PlotJugglerStreamer`) to run the parse-and-forward pipeline in-process from any byte source (e.g. a serial port), no subprocess or shell pipes.

## Requirements

- Python 3.12+
- Device emits batches framed like:

      [ 0xDE 0xAD 0xBE 0xEF ][ COUNT:1 byte ][MSG_ID:2 bytes][ PAYLOAD ] * COUNT

- The payload is a *packed* C struct defined in a header file.
- PlotJuggler with the WebSocket Server plugin enabled (Protocol: JSON).

## Install

```bash
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e .
```

## Start PlotJuggler’s WebSocket Server

- In PlotJuggler: **Streaming → WebSocket Server**
- Protocol: **JSON**
- Port: for example **9871**
- Click **Start**

## Start Bridge

Connect to device, parse batches, forward to PJ.
Default TCP port is **5000**; default WS URL is **ws://127.0.0.1:9871**.
Timestamps are in **milliseconds** by default (`--ts-scale 1e-3`).

```bash
python3 bridge.py \
  --host 192.168.1.91 \
  --struct-header /path/to/telemetry.h \
  --struct-name MyStruct \
  --name-prefix "device_a."
```

Notes:
- Add `--controller-out-size` if using a struct with `HiddenHighLevelControllerOutputData_s` to specify size (ex: 109)
  (this is `CONTROLLER_OUTPUT_DATA_SIZE` value in `high_level_controller_common.h`)
- Add `--ignore-errors` only applies to file conversions, streaming always ignores errors regardless of this flag
- Add `--csv` with `--file` to write CSV instead of NDJSON (see [Parsing log files](#parsing-log-files)).
  File conversions only — passing it with `--host` is an error, since TCP mode forwards JSON to PlotJuggler.
- Add `--ws-url ws://<pj_host>:9871` if PlotJuggler runs elsewhere.
- If needed, guard against corrupted batches with `--max-frames-per-batch N`.
- To fall back to single `[DELIM][PAYLOAD]` (no COUNT), pass `--no-counted-batch`.

## Two-process pipeline (for debugging)

0) Derive the struct (optional sanity check). Prints JSON describing the derived `struct_fmt`, `fields`, and `record_size`.

  ```bash
  python3 derive_struct.py \
    --header /path/to/telemetry.h \
    --struct-name MyStruct \
  ```

1) Parse from device to NDJSON:

  ```bash
  python3 stream_parser.py \
   --host 192.168.1.91 \
   --struct-header /path/to/telemetry.h \
   --struct-name MyStruct \
   --name-prefix "device_a."
  ```

2) Forward NDJSON to PlotJuggler:

  ```bash
  python3 stream_parser.py --host 192.168.1.91 --struct-header /path/to/telemetry.h --struct-name MyStruct | python3 socket_client.py --ws-url ws://127.0.0.1:9871
  ```

## Use as a library

The same pipeline is importable, so a host application can supply its own byte
source instead of TCP or a file. `PlotJugglerStreamer` takes a blocking
`read_fn() -> bytes` (return `b""` when idle), derives the struct from a header,
parses `0xDEADBEEF` batches, and forwards JSON to PlotJuggler's WebSocket
Server on a background thread.

```python
from pj_bridge import PlotJugglerStreamer

streamer = PlotJugglerStreamer(
    read_fn=lambda: ser.read(max(1, ser.in_waiting)),  # e.g. a pyserial port
    struct_header="/path/to/telemetry.h",
    struct_name="MyStruct",
    ts_field="timestamp",
    ws_url="ws://127.0.0.1:9871",
)
streamer.start()
# ...
streamer.stop()
```

`stop()` ends the reader and the WebSocket sender even if PlotJuggler's server
was never up. Pass `on_connect` / `on_disconnect` callbacks to be notified when
the WebSocket link to PlotJuggler is established or drops (they run on the event
loop, so keep them cheap and marshal any UI work to your own thread):

```python
PlotJugglerStreamer(
    ...,
    on_connect=lambda: print("connected to PlotJuggler"),
    on_disconnect=lambda: print("PlotJuggler link dropped"),
)
```

Other public names: `derive_struct`, `DelimitedRecordParser`, `ws_sender`,
`source_reader_to_queue`.

## Field naming

- All non-timestamp fields are emitted with the optional prefix:

  ```bash
  --name-prefix "device_a."
  ```

  Example JSON:

  ```json
  {"t": 1727370023.415, "device_a.ax": 0.02, "device_a.ay": -0.01, "device_a.az": 9.81}
  ```

- Arrays like `float gyro[3];` become `device_a.gyro[0]`, `device_a.gyro[1]`, `device_a.gyro[2]`.

## Timestamp (`t`)

- If `--ts-field ts_ms` is provided, `t = ts_ms * --ts-scale` (default `1e-3`, ms → seconds).
- If no `--ts-field` is set, arrival time is used (`time.time()` in seconds).
- If your device time is relative (since boot) and you want wall-clock, you can add an epoch offset in code; ask if you want a ready-made flag for that.

## Parsing log files

`stream-parser` reads either source and writes to stdout:

1. Live over TCP, using **`--host`**
2. Offline from stored binary log files, using **`--file`**

By default it emits NDJSON (one JSON object per line). Add **`--csv`** to get CSV
directly, ready for Excel, Pandas, or visualization tools.

### Straight to CSV (recommended)

```bash
# offline log file
stream-parser --file capture.bin --struct-header telemetry.h --struct-name MyStruct --csv > logs.csv

# live TCP stream
stream-parser --host 192.168.1.91 --struct-header telemetry.h --struct-name MyStruct --csv > live.csv
```

`pj-bridge` accepts the same flags for its `--file` mode, so either entry point works:

```bash
pj-bridge --file capture.bin --struct-header telemetry.h --struct-name MyStruct --csv > logs.csv
```

With `--host`, `pj-bridge` forwards JSON to PlotJuggler instead, so `--csv` is
rejected there rather than silently ignored.

Options:

- `--csv-delimiter ';'` to change the column separator (default `,`).
- `--no-header` to omit the header row.

Column order follows the parser's field order, and the header is taken from the
first record — the same layout `json-to-csv` produces, so existing CSVs stay
comparable.

### Via NDJSON and `json-to-csv`

`json-to-csv` converts NDJSON on stdin into CSV on stdout. All JSON lines must
contain the same fields (the parsers ensure this).

```bash
stream-parser --file capture.bin ... | json-to-csv > logs.csv
```

This produces byte-identical output to `--csv`, but every record is serialized
to JSON, written to a pipe, and parsed back again. Prefer `--csv`
unless you actually want the NDJSON, for example to feed `socket-client`
or to inspect individual records.

## Uninstall

- Deactivate the venv and remove the project directory, or run `pip uninstall pj-bridge` inside the venv (if installed as a package).

## License

[MIT](https://dephy-inc.mit-license.org/)
