Metadata-Version: 2.4
Name: log-ai-compressor
Version: 3.0.4
Summary: Local log forensics: every finding carries line numbers and the sha256 of the raw text, re-checkable without this tool; no causal chain means INSUFFICIENT, with an explicit list of what is still missing. Analysis stays on-device; the optional AI layer sends only redacted excerpts to your configured provider (off by default). Reproducible Loghub 2.0 benchmarks vs Drain3 / tail / grep.
License-Expression: MIT
Project-URL: Homepage, https://gitee.com/hu-chenyu-dev/log-ai-compressor
Project-URL: Repository, https://gitee.com/hu-chenyu-dev/log-ai-compressor
Project-URL: Changelog, https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/CHANGELOG.md
Project-URL: Releases, https://gitee.com/hu-chenyu-dev/log-ai-compressor/releases
Keywords: log,analyzer,llm,compressor,troubleshooting,benchmark,reproducible,root-cause,local-first,forensics,mcp
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Logging
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Provides-Extra: web
Requires-Dist: fastapi>=0.110; extra == "web"
Requires-Dist: pydantic>=2; extra == "web"
Requires-Dist: uvicorn[standard]>=0.27; extra == "web"
Requires-Dist: sse-starlette>=1.6; extra == "web"
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == "mcp"
Provides-Extra: ai
Requires-Dist: httpx>=0.27; extra == "ai"
Provides-Extra: all
Requires-Dist: log-ai-compressor[ai,mcp,web]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.1; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: drain3>=0.9.11; extra == "dev"
Requires-Dist: logslop>=0.1.2; extra == "dev"
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Dynamic: license-file

# log-ai-compressor

**Every finding carries line numbers + sha256. Change one byte → `verify` says TAMPERED.**

For ops / backend / security engineers reading logs. Analysis runs locally — 0 API calls, logs stay on your machine. The optional AI interpretation layer is **off by default**; if you enable it, only redacted excerpts are sent to the provider you configure, with preview and explicit consent before sending.

`verify` re-reads the raw log and checks every quote byte-for-byte. If the log was edited, it exits 1 — you never have to trust the tool.

```console
$ log-ai-compressor verify --report examples/app_report.json
模式：复核已导出报告
对象：报告 app_report.json
证据：共 4 条引用，匹配 3 条，失配 1 条，源不可读 0 条，缺行号 0 条，缺摘要 0 条

自校验结论：TAMPERED        # 退出码 1

  ✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29
```

Key lines from the run (Chinese CLI; stdout + stderr, so the `MISMATCH` line
may appear before or after the verdict): 4 quotes checked, 3 matched,
1 mismatched → verdict TAMPERED, exit 1. The full run also ends with a one-line
explanation. Hashes and exit codes are byte-identical to what you will get;
reproduce it with the one-byte edit below.

▶ Install & self-proof below · [中文 README](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/README.md)

```console
$ python -m pip install -i https://pypi.org/simple "log-ai-compressor==3.0.4"
$ python -c "import log_ai_compressor, pathlib, shutil, os; os.makedirs('examples', exist_ok=True); shutil.copy(pathlib.Path(log_ai_compressor.__file__).parent/'data/examples/app.log', 'examples/app.log')"
$ log-ai-compressor run examples/app.log -f json -o examples/app_report.json
$ log-ai-compressor verify --report examples/app_report.json
自校验结论：VERIFIED
```

The last line is verbatim output (Chinese CLI): verdict VERIFIED, all 4 quotes
matched, exit 0. No clone needed: line 2 copies the sample log out of the
installed wheel. The key lines at the top are from a real run too: change `db-primary`
to `db-primarY` in `examples/app.log`, re-run `verify`, and you get exactly
that. Change it back → `VERIFIED` again. There is no "mostly passed" tier.

[![Release](https://img.shields.io/badge/release-v3.0.4-c71d23)](https://gitee.com/hu-chenyu-dev/log-ai-compressor/releases/tag/v3.0.4)
![Python](https://img.shields.io/badge/python-3.11%2B-3776AB)
![License](https://img.shields.io/badge/license-MIT-green)

> Repository is hosted on Gitee (reachable worldwide, but slow from some
> regions). The package itself installs from PyPI with no dependency on Gitee.
> Screenshots live in the [Chinese README](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/README.md) —
> this page stays text-only so every line renders anywhere.

## Open the UI (the recommended entry point)

| Your machine | One command |
| --- | --- |
| Windows | double-click `start.bat` in the repo root — finds Python, installs what is missing, opens the browser |
| anything else | `pip install "log-ai-compressor[web]" && log-ai-compressor web` |

Three steps to a conclusion: **① drag a log into the window (or pick a local file
with Browse) → ② levels default to ERROR/FAIL/WARN only → ③ press Start**. The
first thing you get is one card — located root cause, or "insufficient evidence"
with the exact gap — not a wall of template clusters. Press **Re-check evidence**
and the tool re-reads the raw log line by line and returns `VERIFIED` / `TAMPERED`.

> The UI is zero-build and zero-CDN, so it works offline; the file browser picks
> local paths and never uploads. The one exception is FastAPI's own `/api/docs`
> (Swagger), which pulls its stylesheet from jsdelivr — the main screen is unaffected.

## CLI (no UI: pipes, CI, agents)

The install and sample commands are in the first screen above. After them:

```bash
log-ai-compressor run examples/app.log -f summary
```

Pin the version: mirrors lag by days, and 2.13.1 (an older release) has no
`-f summary`. Samples ship inside the wheel (`log_ai_compressor/data/examples/`);
if `examples/` is absent, `run` / `compare` fall back by basename and print a
notice. Verbatim output of the command above (the CLI prints Chinese —
numbers, fingerprint and hashes are byte-identical to what you will get):

```
【日志分析摘要】examples\app.log
原始日志 ≈379 tokens → 摘要 ≈234 tokens（压缩 38%）
确定性指纹 `48e5781aafc4b27fe0f68c5847afe6d891936ac9f2e0af59aefaa9860f9c2886`（同一份日志跑两遍必须完全一致）
总行数 24，错误 8 行，去重后 4 种，时间范围 2024-03-15 08:00:00 ~ 2024-03-15 08:00:25。
已定位根因：java.net.ConnectException: Connection refused（另有 1 个根因候选）

（证据出处末尾 16 位十六进制 = 该段原文 sha256 前 16 位，可自行复核）
1. [P0 | ERROR | ×3 | 根因 | 行 4~13] authorization failed: connection refused to db-primary:5432
   证据出处：L4 bc5b93cec7cde01c
   -> 因果链源头（1 个错误由其衍生）
2. [P0 | ERROR | ×3 | 行 6~11] request 8812 failed: upstream unavailable
   证据出处：L6 21d12907c3115d34
3. [P0 | ERROR | ×1 | 根因 | 行 18~22] java.net.ConnectException: Connection refused
   证据出处：L18-L22 68af24f230baff5c
   -> Caused-by 因果链直连，根因：java.net.ConnectException: Connection refused
4. [P0 | ERROR | ×1 | 行 23~23] stock check timeout after 3000ms item=SKU-7781
   证据出处：L23 a72b9f38845a14ed
```

### Prove it to yourself: break one byte

```bash
log-ai-compressor run examples/app.log -f json -o examples/app_report.json
log-ai-compressor verify --report examples/app_report.json   # VERIFIED, exit 0
```

Now change `db-primary` to `db-primarY` in `examples/app.log` and re-run the
second command. Key lines (Chinese CLI; the `MISMATCH` line is stderr, so its
order relative to the verdict can vary; the full run ends with a one-line
explanation):

```
模式：复核已导出报告
对象：报告 app_report.json
证据：共 4 条引用，匹配 3 条，失配 1 条，源不可读 0 条，缺行号 0 条，缺摘要 0 条

自校验结论：TAMPERED

  ✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29
```

Translation: 4 quotes checked, 3 matched, 1 mismatched → verdict TAMPERED,
exit 1. Change it back → `VERIFIED` again. No "mostly passed" tier.
Machine-readable: add `--json` and read `checked: 4 / matched: 3 / mismatched: 1`.

## What it is (and what it is not)

| | grep / tail | Drain3 | 3-shot LLM | this tool |
|---|---|---|---|---|
| What you get | matching lines | templates | templates | conclusion + line range + sha256 |
| Model needed | no | no | yes | **no (core analysis: 0 API calls)** |
| Full 10-set GA | — | 71.58% | not run | **88.68%** |
| Full 10-set PA | — | 58.06% | not run | **85.05%** |
| Output you can re-check | yes | no | no | yes (`verify` re-reads the log) |

## Do the numbers hold up?

Full Loghub 2.0, ten datasets: **1,940,760 lines / 1,621 clusters**.

- GA (grouping accuracy): **88.68%** vs Drain3 **71.58%** — same tokenization, same scoring
- PA (parsing accuracy): **85.05%** vs Drain3 **58.06%**
- Reproducible: `docs/repro_manifest.json` pins 64-bit hashes, byte sizes, per-dataset line counts,
  and 7 snapshot `sha256`s; `scripts/verify_repro_manifest.py` checks all 11 entries.

## What I will not claim

- Reproducible ≠ state of the art. There is no leaderboard rank and no third-party run yet.
- The LLM comparison is a 2,000-line-per-dataset sample, not full data.
- The HDFS block-level result cannot be re-run here (the 1.5 GB raw log is not on this machine);
  merged recall is 37.17% — that is the number most worth doubting.

## Want to reproduce it? (please do)

Post back three things: OS + Python version, exit code, last 20 lines on failure.
**A failure is worth more than a pass** — it means the docs or environment assumptions are wrong.

- Repo: <https://gitee.com/hu-chenyu-dev/log-ai-compressor>
- Release v3.0.4: <https://gitee.com/hu-chenyu-dev/log-ai-compressor/releases/tag/v3.0.4>
- Reproduction guide: [`docs/REPRODUCIBILITY.md`](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/docs/REPRODUCIBILITY.md)
- Real-reproduction invitation: <https://gitee.com/hu-chenyu-dev/log-ai-compressor/issues/IKWWD3>
- PyPI: <https://pypi.org/project/log-ai-compressor/>

## Full reproduction (optional, ≈445 MB / 11 entries)

Clone the repo, drop the Loghub datasets under `data/` as described in
[`docs/REPRODUCIBILITY.md`](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/docs/REPRODUCIBILITY.md) §5.2, then:

```bash
python scripts/verify_repro_manifest.py   # all 11 PASS before continuing
python scripts/refresh_benchmarks.py      # ~60 min
python scripts/prove.py                   # every public claim re-checked
```

Without data, `verify_repro_manifest.py` reports 11 `MISS` and exits 1 — that is expected.

## Chinese

Full documentation, design notes and stage-by-stage evidence are in the Chinese
[`README.md`](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/README.md).

## 🙏 External reproducers

Every number in this repository is currently **self-reported by the author**. This table only lists people who actually ran it (rules: [docs/REPRODUCIBILITY.md](https://gitee.com/hu-chenyu-dev/log-ai-compressor/blob/main/docs/REPRODUCIBILITY.md), section 8):

| # | Environment | Result | Date |
|---|---|---|---|
| — | none yet | be the first | — |

To get listed: run the two commands in Issue [#IKWWD3](https://gitee.com/hu-chenyu-dev/log-ai-compressor/issues/IKWWD3), then reply with `OS + Python version + exit codes`. Named or anonymous, pass or fail — both count.

## License

MIT.
