Metadata-Version: 2.4
Name: sluice-intel
Version: 0.1.0
Summary: Sluice intelligence sidecar: drift detection, semantic typing, parser proposals. Propose-only, off the hot path.
License-Expression: MIT
Project-URL: Homepage, https://github.com/dark-14100/sluice
Project-URL: Source, https://github.com/dark-14100/sluice
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.13
Requires-Dist: PyYAML<7,>=6.0
Requires-Dist: jsonschema<5,>=4.26
Requires-Dist: drain3<1,>=0.9.1
Requires-Dist: cachetools<7,>=5
Requires-Dist: httpx<1,>=0.28
Provides-Extra: test
Requires-Dist: pytest<10,>=9; extra == "test"
Dynamic: license-file

# intel: the intelligence sidecar

Propose-only, off the hot path, opens no listener. It reads samples and quarantine through the admin API
(`contracts/admin.openapi.yaml`), scores structural drift, types unknown fields from value shapes, generates a parser
proposal, asks the data plane to dry-run it, and posts it for a human. It never touches the vault, the parser store
or the pipeline, and no code path can call approve, reject, rollback or replay (a test asserts the request log).

```sh
cd intel
python3.13 -m venv .venv && . .venv/bin/activate          # 3.11+ required
pip install -e '.[test]'
pytest                                                     # 100+ tests, warnings are errors

python -m ulpf_intel --admin http://127.0.0.1:9000 --state-dir ./intel-state   # the loop
python -m ulpf_intel --once                                # one pass and exit
python -m ulpf_intel --healthcheck                         # GET /healthz; exit 0 healthy, 1 not (no shell in the image)
python -m ulpf_intel --egress-check                        # exit non-zero if ANY outbound attempt succeeds
```

`--egress-check` is the air-gap probe: a public DNS lookup and TCP connects to 1.1.1.1:443 and 8.8.8.8:53, 2 s each.
All three are expected to fail; on a machine with internet access it correctly reports `FAIL`.

## What each module does

| Module | |
|---|---|
| `fingerprint.py` | Structural fingerprint of raw lines (delimiters, key set, value shapes, Drain3 templates) and the drift score with human-readable signals (`keys renamed: srcip->src`). |
| `typer.py` | Semantic typing of a column from its values: IPs, ports, timestamps (with the Go layout), protocol, action, user names. Headerless columns supported. Source vs destination disambiguation. Abstains below 0.6. |
| `propose.py` | Proposal generators: `propose_csv` (headerless CSV), `propose_kv` (new kv source), `propose_kv_patch` (drifted kv parser -> patch), `propose_json` (routes on `event_type`), `propose_text` (Drain3 templates -> anchored RE2 with typed named groups), `propose_cef` (CEF and LEEF, from the standard extension keys, confirmed against the values). |
| `dsl_eval.py` | A small reference evaluator of the parser DSL. NOT the engine: it derives the `tests:` vectors of a proposal and gives a local pre-check. |
| `validate.py` | Acceptance thresholds, RE2-safety check (Python's `re` accepts lookaround the Go engine refuses), local dry-run. |
| `llm.py` | OPTIONAL (`--llm-endpoint`, off by default): a local model names anonymous csv columns. Loopback only, refused otherwise; at most 5 sample values per column in the prompt, never a whole record; can only fill `_` entries of a `columns:` list, so it cannot touch a mapping, pattern or type; failures never break a proposal; the result still goes through the same dry-run and human approval. |
| `client.py`, `state.py`, `loop.py`, `egress.py`, `__main__.py` | The admin client (bounded retries), JSON baselines, the loop, the air-gap probe, the CLI. |
| `models.py` | Pydantic mirrors of the contract types (`extra="forbid"`). |
| `llm.py` | OPTIONAL (`--llm-endpoint`, off by default): a local model names anonymous csv columns. Loopback only, refused otherwise; at most 5 sample values per column in the prompt, never a whole record; can only fill `_` entries of a `columns:` list, so it cannot touch a mapping, a pattern or a type; a failure never breaks a proposal; the result still goes through the same dry-run and human approval. |

## The loop, per source

1. Fetch recent samples (parsed and quarantined). Fewer than `min_samples` (50): do nothing.
2. Compare with the stored baseline. No baseline and a high quarantine rate: "never parsed".
3. Score >= 0.35, or quarantine rate > 0.2: post **one** `DriftAlert` (reused while open).
4. At least `min_cluster` (50) quarantined records of one shape: generate a proposal, dry-run it via
   `POST /admin/parsers/dryrun`, post it. One proposal per alert.
5. A proposal that misses its thresholds (`match_rate >= 0.95`, `mean_coverage >= 0.9`) is still posted, with
   `acceptance:` warnings, so the UI shows it as needing edits. It is never dropped.
6. The baseline is refreshed only from healthy passes, never while the source is drifting.

## Measured (fixtures, reference evaluator; the data plane's dry-run is authoritative and not built yet)

| Check | Result |
|---|---|
| Drift, `fortinet.log` vs a fresh half of itself | 0.00 (bar < 0.1) |
| Drift, `fortinet.log` vs `fortinet_drift.log` | 0.54 structure only, 0.63 with the drifted file's quarantine rate 0.94 (bar >= 0.6 holds only with the quarantine term) |
| Typing, `palo_alto_unknown.log` | 7 of 7 manifest fields typed to the right column, all values matching (bar 90%) |
| Proposal, Palo Alto CSV | 500/500 records parsed, manifest reproduced |
| Proposal, Fortinet drift patch | old parser 0/500, patch 500/500 |
| Proposal, Suricata JSON | 2000/2000 |
| Proposal, Cisco ASA text | 2000/2000 parsed; source/destination agrees with the manifest for 1725/2000 (see below) |

## Known limits

- **ASA direction.** On ASA the first address is the foreign host: the source of an inbound connection, the destination of
  an outbound one. The proposal reads the words `inbound`/`outbound`, then cue words (`src`, `dst`, `from`), `->`, port
  behaviour, and last position, flags every weak decision with the other path as an alternative, and the reviewer confirms.
  It agrees with the manifest on 1725/2000 records; the 275 it gets wrong are `Teardown TCP` (302014), which carries no
  direction word at all (only the matching `Built` message with the same connection id would say). A test pins the split.
- **CEF/LEEF** are tested on SYNTHETIC samples only (no CEF/LEEF file exists in the fixture corpus), and the severity banding
  (0-3 Low ... 9-10 Very-High to OCSF 2..5) follows the CEF specification and is stated to the reviewer in every proposal.
  The optional local-LLM step names only anonymous csv columns; it does not improve mappings or free-text slot names.
- The reference evaluator has no render-back, so `render_back_ok_rate` comes only from the data plane's dry-run.
- Timestamps without a zone use `--timezone` (default `+00:00`) and every such proposal says so in its warnings.
- "Stale" marking of proposals whose base version moved is done by the data plane on approve (`409`); the sidecar only avoids re-proposing.
