Metadata-Version: 2.3
Name: zelos-packet
Version: 0.0.1
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Requires-Dist: zelos-sdk >=0.0.12a1
Summary: Rust-first packet capture and decode for the Zelos ecosystem. Live capture through a bundled libpcap helper process, with pcap/pcapng offline decode.
Author-email: Zelos Cloud <info@zeloscloud.io>
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# zelos-packet

Rust-first packet capture and decode for the Zelos ecosystem: live capture
through a bundled helper process (`AF_PACKET` on Linux, `/dev/bpf` on macOS),
plus pcap/pcapng offline decode, emitting packet and capture-statistics events
into a Zelos trace.

The helper links libpcap **statically**, so the wheel has no runtime dependency
on a system `libpcap`; the extension module links none at all. Live capture
needs a one-time grant (below); offline decode needs none.

```bash
pip install zelos-packet
```

## Offline

```python
import zelos_packet, zelos_sdk

with zelos_sdk.TraceWriter("capture.trz"):
    d = zelos_packet.PacketDecoder("dump")
    d.convert_file("dump.pcap")     # or d.decode_stream(raw_bytes)
    # rows land at Packet.dump/packets.<field>
```

## Live

```python
# The helper streams to this agent; rows are read back from there, never from
# a local TraceWriter. Defaults to $ZELOS_AGENT_URL, else localhost:2300.
cap = zelos_packet.PacketCapture("en0", snaplen=128, agent_url="http://localhost:2300")
cap.start()                        # rows land at Packet.en0/packets.<field>
print(cap.stats().kernel_drops, cap.metrics().emit_stall_ms)
cap.stop()
```

```bash
sudo "$(command -v python3)" -m zelos_packet install-helper   # once per machine
python3 -m zelos_packet status                                # would capture work now?
```

Then pick the group up. **Linux** stamps a session's group set at login
(`initgroups`), so log out and back in. **macOS** resolves membership per
process, so a fresh terminal is enough. Either way, anything already running
(this shell, a running Zelos app) keeps the group set it started with.

| Platform | What `install-helper` does | What the group grants |
| --- | --- | --- |
| Linux | Installs the helper to a root-owned path with `cap_net_raw=ep` (never `CAP_NET_ADMIN`) and creates/joins `zelos-packet` | The ability to *run a capture*. The helper opens the socket, drops every capability, verifies the drop from `/proc/self/status`, and only then reads. It never hands out the socket, so **not** frame injection. |
| macOS | Installs a ChmodBPF-style boot daemon putting `/dev/bpf*` in `access_bpf` | Capture **and sending** arbitrary frames — a bpf device is opened read-write and capture needs the write side, exactly as for Wireshark. |

`zelos_packet.permission_remediation()` returns the same text a failed open
raises inside `CapturePermissionError`.

### One capture path: the helper

`zelos-packet-helper` runs every live capture on both platforms. It opens the
handle, drops every capability it holds, and streams decoded rows straight to
an agent; `PacketCapture` supervises that process and nothing else. No packet
enters the calling process, so capture privilege never lands on the interpreter
and no capture-capable descriptor is handed out.

- **`source=` is a hard error.** Rows are produced behind the helper's own SDK
  connection, so nothing would reach a source passed here. Use `agent_url=`.
- **An agent must be reachable** for rows to land anywhere. A local
  `TraceWriter` sees nothing.
- **Constructing is not starting.** `PacketCapture(...)` validates the
  interface name, resolves the agent exclusion, and locates a runnable helper,
  so a typo, an unresolvable agent host, or a missing helper raises *there*.
  The *permission* verdict comes from `start()`, where the helper opens the
  handle. (zelos-can starts from its constructor; this does not.)

## Catalog layout

Every capture writes into ONE trace source, `Packet`, and is told apart by its
own event-name prefix, so a field is addressed `Packet.en0/packets.src_ip`:

```
Packet
└── en0                      # the capture's `name`; defaults to the interface
    ├── packets              # zelos.packet.v1 — one row per frame
    └── stats                # zelos.packet.stats.v1 — one row per interval
```

Pass `name=` to override the prefix; it goes through `sanitize_name()`, which
collapses catalog separators (`.`, `:`, `@`, `/`) to `_`, so a VLAN device
`eth0.100` lands at `Packet.eth0_100/packets`.

Two captures may share one source as long as their names differ. Event
registration is **strict-create**: a second capture registering a name already
on that source raises rather than merging, which keeps two captures from
silently interleaving into one table.

## API

| Symbol | Purpose |
| --- | --- |
| `PacketDecoder(...)` | Bring-your-own-bytes: `decode_frame`, `decode_stream`, `convert_file`, `push_stats`, `register_schemas`, `metrics`, `flush` |
| `PacketCapture(...)` | Live handle: `start`, `stop`, `stats`, `metrics`, `interface`, `link_type`, `is_active`, `error`; also a context manager |
| `list_interfaces()` | `InterfaceInfo` per NIC — name, index, up/running/loopback, addresses, MAC |
| `permission_remediation()` | Copy-pasteable privilege fix for this platform |
| `capture_supported()` | False outside Linux/macOS; offline decode still works |
| `sanitize_name(name)` | The event-prefix rule, shared with the packet extension |
| `CapturePermissionError` | `PermissionError` subclass, message embeds the remediation |
| `InterfaceNotFoundError` | `ValueError` subclass for an unknown interface name |

## The `frame` column

`frame` is the source of truth every decoded column is checked against, so both
entry points populate it by default (`log_frames=True`), clipped to
`stored_frame_bytes=256` bytes. `stored_frame_bytes=None` stores every captured
byte; `log_frames=False` drops the column.

| Knob | Applies to | Effect |
| --- | --- | --- |
| `snaplen` | live capture only | Bytes the **kernel** copies out. Uncaptured bytes are dissected by nobody; the shortfall shows up as `orig_len > cap_len` with `truncated` set. |
| `stored_frame_bytes` | live + offline | Bytes **stored** in `frame`. Dissection always reads the full captured bytes, so this changes storage and nothing else. |

`orig_len` / `cap_len` / `truncated` describe capture truncation only. Storage
truncation is `len(frame) < cap_len`; the frozen schema has no column for it.

## Drop policy

**While the sink is consuming, userspace blocks rather than dropping.** Every
emit is the SDK's normal send path, never `try_send`, so a full router channel
stops the read loop, the kernel ring fills, and the kernel drops where it can be
counted.

| `kernel_drops` | `decode_stall_ms` | Meaning |
| --- | --- | --- |
| 0 | high | Trace store is backpressuring |
| climbing | ~0 | Line-rate overload; the read loop is the bottleneck |

`Metrics.emit_stall_ms` is the sink-only slice of `decode_stall_ms`; the
difference is dissection cost.

**Where that guarantee stops.** Backpressure only exists while something is
pulling. If the helper's gRPC publish stream to the agent breaks, the router's
publisher subscription goes away while decode keeps running: rows are produced
and discarded, and **`kernel_drops` stays flat** because the kernel handed them
over successfully. So `kernel_drops == 0` means "the kernel lost nothing", not
"the trace is complete". What does move is `Metrics.emit_errors` /
`Metrics.flush_errors`, and at stop time `CaptureStats.tail_abandoned`
(non-zero = rows the publisher held never reached the agent) and
`CaptureStats.kernel_stats_errors` (non-zero = `kernel_packets` / `kernel_drops`
are a stale reading, not totals). Check those before treating a capture as
complete. There is no counter for rows dropped into a dead subscription mid-run.

## Self-traffic exclusion

When the agent streams over the interface being captured, every emitted row
makes more captured bytes. At a large snaplen that loop does not converge.

**On by default for `PacketCapture`, derived from `agent_url`.** With neither
`exclude_agent_addrs` nor `exclude_agent_port` passed, both come from the URL
the capture streams to (`agent_url`, else `$ZELOS_AGENT_URL`, else
`$ZELOS_TRACE_FORWARD_URL`, else `http://localhost:2300`), resolving a host name
to **every** address it has — so the `localhost` default excludes `127.0.0.1`
and `::1` both. `PacketDecoder` takes no default at all: it is handed bytes that
already exist, so there is no loop to close and dropping rows out of a
user-supplied file would be silent data loss.

| You want | Pass |
| --- | --- |
| A different endpoint | `exclude_agent_addrs=[...]` **and** `exclude_agent_port=...` — no resolution happens |
| The agent's traffic captured | `exclude_agent_addrs=[]` |
| Nothing else | (nothing) |

Either argument alone is an error; the empty list is the one address list a port
may be absent from. A host that does not resolve is a **construction error**,
not a fallback to capturing unfiltered.

A **live** capture compiles the exclusion with `pcap_compile` and installs it
with `pcap_setfilter` on the helper's handle, so the kernel rejects excluded
traffic before a packet is copied out. If `pcap_compile` cannot express it for
the link type, the capture fails to open rather than running unfiltered. Offline
decode has no kernel in the path; its userspace `AgentFilter` matches on ports
only, but it is still applied — a pcap being converted loses its agent-endpoint
rows unless you pass `exclude_agent_addrs=[]`.

The expression is not recorded in the trace. It is logged at capture start
(`RUST_LOG=zelos_packet=debug`, "compiling the agent filter"), which is the only
place to read back what was installed — worth capturing alongside a trace you
intend to analyze, because of the three gaps below.

**It drops more than the agent port.** Closing the IPv4 fragment hole
statelessly costs precision, on **any port**, for the agent addresses only:

| Traffic to/from an agent address | Kept? |
| --- | --- |
| IPv4 first fragment (offset 0) | kept, unless on the agent port |
| IPv4 continuation fragment (offset > 0) | **dropped**, any port |
| IPv6 with any extension header — Fragment, Hop-by-Hop, Routing, Destination Options, **ESP (50) or AH (51)** | **dropped**, any port |

So fragmented UDP telemetry to an agent address on port 9999 keeps each
datagram's *first* fragment and loses every continuation, and an agent address
reached over IPsec transport mode contributes no packets at all. The idiomatic
`not (host X and port Y)` *accepts* continuation fragments, and those are what
the feedback loop is made of; statelessly, rejecting on the address alone is the
only correct answer. There is no counter for it either — `pcap_stats`'s
`ps_recv` counts packets that already passed the kernel filter, so excluded
traffic appears in no column. To see what was excluded, capture with
`exclude_agent_addrs=[]` on an interface the agent does not stream over.

**Tunnels defeat it.** Both the kernel program and the userspace backstop read
the **outer** IP header, so agent traffic inside WireGuard, any other VPN, VXLAN
or GRE carries its addresses in the inner header and is re-admitted. Capture the
tunnel interface (`wg0`, `utun0`, …) instead, where the agent's addresses *are*
the outer ones.

**Three or more stacked VLAN tags defeat it.** libpcap's `host` / `port`
primitives do not look past a VLAN tag, so the expression is repeated under
`vlan` and `vlan and vlan` — untagged, single-tagged and QinQ. Under a third tag
the address test reads the wrong offset, does not match, and the enclosing
`not (…)` therefore **accepts** the frame. The arms stop there because each one
multiplies the compiled program, and the worst case already sits close to BSD's
512-instruction `BPF_MAXINSNS` ceiling.

## Development

```bash
just develop     # uv sync + maturin develop
just test        # cargo test (all three crates) + pytest + check-static
just lint        # cargo clippy -D warnings
just stub_gen    # regenerate py/zelos_packet/_native.pyi
```

Live-capture tests are opt-in (`ZELOS_PACKET_TEST_LIVE=1`, optionally
`ZELOS_PACKET_TEST_IFACE=<name>`) and need a `zelos-agent` binary, since rows are
read back out of a real agent. Supervision and the permission verdict need no
privilege on any platform: `rs/supervisor.rs` and `test/test_helper_backend.py`
both drive a fake helper.

The Rust toolchain is pinned by the nix dev shell; there is deliberately no
`rust-toolchain` file. Release wheels come from the `api-py-zelos-packet`
workflow, or locally from `just build-linux-wheels`; both build the static
libpcap with `docker/manylinux/build-libpcap.sh` (the one place its version,
hash and configure flags live) and gate the artifact with
`docker/manylinux/check-wheel-static.sh`. A bare `maturin build` elsewhere does
neither. [RELEASE.md](RELEASE.md) has the release order.

