Metadata-Version: 2.4
Name: network_core
Version: 0.8.0
Summary: Core networking utilities and data models
Author: Your Name
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pytz
Requires-Dist: orjson
Requires-Dist: matplotlib
Requires-Dist: brotli<=1.1.0,>=1.0
Requires-Dist: blackboxprotobuf
Requires-Dist: cryptography<44.1,>=42.0
Requires-Dist: polars
Requires-Dist: tqdm

# network_core

A generic Go + Python library that extracts flow-level features from
network captures. Designed to survive high-speed captures with real
packet loss (including capture-side truncation and non-IP tail
noise).

Author: rushi. Contributors: peter, levi.

## What v0.8.0 changes

**Schema bump (SchemaVersion=3).** `mergedFlows.csv` gains five columns.
Existing columns keep their positions; readers of the pre-v0.8.0
5-column schema continue to work if they use `csv.DictReader` or
`row.get()`; strict-positional readers of columns `[0..4]` see the
same values as before.

Full v0.8.0 header:

```
FlowId,FiveTuple,StartTime,EndTime,SNI,ALPN,DNSName,DNSDelta,DNSCandidates,SNIReason,ImplausibleTsSkipped
```

## Column semantics

### `SNI`

The hostname parsed from the flow's ClientHello (TLS on TCP, or
QUIC v1 Initial). When the extractor could not observe a
ClientHello but a prior flow with a live carry-map entry did, the
inherited SNI lands here (see `SNIReason=carry`).

Value is `nan` when the extractor could not attribute a hostname —
see `SNIReason` for the specific reason.

### `ALPN` — CLIENT-OFFERED from THIS flow's ClientHello

The value is the FIRST-OFFERED ALPN entry from the ClientHello THIS
flow parsed, or carried alongside the SNI from an earlier flow via
§2 continuity carry / §3 QUIC CID linkage. `nan` if no CH was
parsed and no carry succeeded.

- TCP: `tls_sni.ALPN[0]` from the parsed CH.
- QUIC: `quic_sni.ALPN[0]` from the parsed QUIC-CRYPTO-carried CH.
- Carry: inherited from the prior flow's `SNIInfo`.

`nan` explicitly means "this flow's CH didn't parse and no carry
supplied a value". It does NOT mean the client didn't offer an ALPN
extension — some clients omit it entirely — but from the wire we
cannot tell those cases apart.

The column is a client-side observation, not a negotiated protocol.
For TLS 1.2 the server's negotiated choice is in the (cleartext)
ServerHello; for TLS 1.3 it moves into `EncryptedExtensions` and is
NOT recoverable. If you need the definitely-negotiated protocol
you must decrypt the handshake — this extractor doesn't.

**Behavior change vs 0.7.0 (rc6):** removed the ServerHello
cross-flow inference, the plaintext HTTP-version fallback, and the
QUIC port-443 → `h3` heuristic. All three could put a value in the
column that the flow's own CH never offered. See CHANGELOG §Behavior
changes for details.

### `DNSName` / `DNSDelta` / `DNSCandidates`

**Populated by the finalize step, not the Go extractor.** The Go
pass writes `mergedFlows.csv` with these columns empty and
concurrently appends valid DNS observations to a temp
`dns_obs.csv`. The Python finalize step (`network_core.finalize`)
runs a polars join_asof at EOF, streams the enriched Flows.csv to
a temp path, atomically renames it onto `mergedFlows.csv`, and
deletes `dns_obs.csv` (`--keep-dns-obs` preserves it for debug).

**Consumer contract:** if you read `mergedFlows.csv` AFTER the
finalize step (the normal case), these columns are populated.
Raw Go-only output (e.g. during a live capture before finalize
has run) has them EMPTY strings.

- `DNSName` — the LATEST canonical A/AAAA name for this flow's
  server IP, answered BEFORE `StartTime`, from the full pcap DNS
  scan. If no answer exists before start, the EARLIEST answer
  after start (with a positive `DNSDelta`).
- `DNSDelta` — signed seconds (`answer_ts − StartTime`). Negative
  when the answer preceded flow start (fresh). Positive when the
  answer arrived after (post-hoc; consumers decide whether to
  trust).
- `DNSCandidates` — all distinct canonicals mapped to the server
  IP across the WHOLE pcap, `|`-joined, chronological.

### `SNIReason` — one of 8 machine-friendly codes

| Code | Semantics |
|---|---|
| `clienthello` | Wire ClientHello parsed (TCP or QUIC). SNI populated. |
| `carry` | Inherited via observed-continuation (TCP idle-split or QUIC CID linkage). SNI populated. |
| `no_clienthello` | TLS/QUIC transport observed, no CH message reached the parser. |
| `ch_parse_failed` | CH bytes present but unparseable (gap, truncated, retransmit mismatch, QUIC CRYPTO gap). Specific pathology in `flow_pathology[].specific_detail`. |
| `unlinkable_migration` | QUIC connection migration to an unseen CID (short-header-only, zero-length CID, ambiguous CID, CID-collision). Specific pathology in `flow_pathology[].specific_detail`. |
| `carry_refused` | TCP continuation candidate refused (FIN/RST in gap, or seq-advance past hard-cap). Specific pathology in `flow_pathology[].specific_detail`. |
| `unsupported_version` | Non-v1 QUIC (draft, v2, unknown). |
| `not_tls` | Flow carries no TLS/QUIC at all (HTTP:80, DHCP, mDNS, STUN…). Empty SNI is the CORRECT value here — NOT loss. |

### `ImplausibleTsSkipped`

Per-flow count of packets dropped by the timestamp-sanity gate
(implausible ts: pre-2020 or > now+24h). Non-zero flows are
flagged in the diagnostics sidecar.

## The diagnostics sidecar

`mergedFlows.diagnostics.json` sits next to `mergedFlows.csv`.
Every key listed below is REQUIRED — consumers may reject a
diagnostics blob with UN-listed keys.

- `session_start_utc`, `session_end_utc`
- `capture_span_from_plausible_packets_s` — capture length as
  measured by plausible-ts packets only. The yt_7 truncated tail
  never widens this.
- `n_pcap_records_read`, `n_skipped_non_ip`, `n_skipped_implausible_ts`,
  `n_dropped_all_implausible_flows`
- `n_dns_answers_accepted`, `n_dns_answers_rejected_qr`,
  `n_dns_answers_rejected_source`, `n_dns_answers_rejected_txnid`
- `n_dns_stale_seconds_hist` — `[0-60, 60-300, 300-3600, 3600-inf]`
  age of the DNS answer that produced each flow's `DNSName`.
  Written by the Python finalize.
- `n_dns_post_hoc` — count of flows whose ONLY DNS answer arrived
  after `StartTime` (positive `DNSDelta`).
- `schema_version` — matches `MergedFlowsSchemaVersion`.
- `producer_version` — the network_core binary's version string
  (`network_core --version`). Brad's oracle reads this to identify
  which extractor produced the artifact.
- `flow_pathology` — array of per-flow finer-grained failure
  details. Each entry: `{flow_id, reason, specific_detail, extra}`.
  `reason` matches the coarse SNIReason column; `specific_detail`
  names WHICH concrete pathology hit (`gap_in_clienthello`,
  `short_header_no_length_context`, etc.).

## CLI

```
network_core <pcap> <flows.csv> <packets.csv> [dns_obs.csv] [diagnostics.json]
network_core --version
```

Default output paths (when the last two args are omitted):
`<flows>.dns_obs.csv` and `<flows>.diagnostics.json`.

Pass `-` as `<pcap>` to read a pcap or pcapng stream from stdin. Useful
for piping a decompressor or a remote fetch without staging the file to
disk:

```
zstdcat capture.pcap.zst | network_core - flows.csv packets.csv
```

Stdin mode is bit-for-bit equivalent to file input; the reader detects
pcap vs pcapng from the first four magic bytes.

## Python API

```python
import network_core
print(network_core.__version__)  # "0.8.0"

# read a finalized mergedFlows.csv
from network_core import get_connections_from_csv
conns = get_connections_from_csv("mergedPackets.csv", "mergedFlows.csv")
```

## CHANGELOG

See `CHANGELOG.md`.
