Metadata-Version: 2.4
Name: loglynx
Version: 1.1.1
Summary: Log analysis and anomaly detection for application logs
Author: Cem
License: MIT
Project-URL: Homepage, https://github.com/LocalinTheEngineer/loglens
Project-URL: Issues, https://github.com/LocalinTheEngineer/loglens/issues
Keywords: logs,log-analysis,anomaly-detection,observability,sre
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Provides-Extra: demo
Requires-Dist: pillow>=10; extra == "demo"
Dynamic: license-file

# LogLens

Log analysis and anomaly detection for application logs. LogLens parses plain
text and JSON logs, groups errors by signature, finds repeated failures and
bursts, flags time windows that deviate from the file's own baseline, reduces
everything to a single health score, and keeps a history so runs can be
compared over time.

It has no dependencies. Terminal output is in Turkish.

[![ci](https://github.com/LocalinTheEngineer/loglens/actions/workflows/tests.yml/badge.svg)](https://github.com/LocalinTheEngineer/loglens/actions/workflows/tests.yml)
[![python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![coverage](https://img.shields.io/badge/coverage-87%25-brightgreen)](#development)
[![tests](https://img.shields.io/badge/tests-303-blue)](tests/)

<img src="docs/demo.gif" alt="LogLens analysing a log file" width="540">

## Install

Python 3.10 or newer. No third-party dependencies.

```
git clone https://github.com/LocalinTheEngineer/loglens
cd loglens
python main.py sample_logs/server.log
```

Or install it from PyPI:

```
pip install loglynx
loglynx app.log          # loglens works too
```

The distribution is published as `loglynx` because `loglens` is too close to an
existing PyPI name. Both commands are installed and do the same thing; the
module and the repository stay `loglens`.

From a checkout, to put the `loglens` command on your PATH:

```
pip install .
loglens sample_logs/server.log
```

Or run it in a container, with the dashboard as the entry point:

```
docker compose up -d                 # http://localhost:8000
docker run --rm -v "$PWD/logs:/logs" loglens:1.1.1 /logs/app.log
```

The examples below use `python main.py`; `loglens` works identically.

## Usage

```
python main.py app.log                          analyse one file
python main.py logs/                            a directory, .gz included
python main.py "logs/*.log.gz"                  a glob
cat app.log | python main.py -                  stdin

python main.py app.log --since 18:00 --until 20:00
python main.py app.log --grep timeout --level ERROR --logger db

python main.py app.log --html report.html       standalone HTML report
python main.py app.log --json                   machine-readable output
python main.py app.log --quiet                  score and findings only

python main.py app.log --save --label "after deploy"
python main.py --history                        past runs
python main.py --compare                        last two runs
python main.py --compare 3:7                    two runs by id

python main.py app.log --watch                  follow the file live
python main.py --serve                          dashboard on localhost:8000
```

Rotated files are merged and sorted chronologically, so `app.log`,
`app.log.1` and `app.log.2.gz` can be analysed as one timeline. Encoding is
detected automatically. Multi-line tracebacks are attached to the entry they
belong to instead of being discarded.

Sample output:

```
╔══════════════════════════════════════════════════════╗
║ SYSTEM HEALTH         44.8 / 100   [D]               ║
║ █████████████░░░░░░░░░░░░░░░░░  Sorunlu              ║
╠══════════════════════════════════════════════════════╣
║ -35.0  Hata oranı            ▪▪▪▪▪▪▪▪▪▪              ║
║ -10.0  Anomaliler            ▪▪▪▪▪·····              ║
║  -5.2  Kritik kayıtlar       ▪▪▪·······              ║
╚══════════════════════════════════════════════════════╝

── SAATLİK HATA DAĞILIMI ───────────────────────────────
17 ──                                        4
18 ── ██████████████████████████████████    229 ← zirve
19 ── ███████████████                       105

┌─ Error cluster #1 ────────────────────────────────────
│ DatabaseConnectionError
│ Occurrences : 275  (4.2/dk)
│ First seen  : 18:15:00
│ Last seen   : 19:19:57  (1s 4dk sürdü)
│ Module      : db
└──────────────────────────────────────────────────────
```

## What it looks like in practice

**A database outage.** You are handed yesterday's log and told "something
happened around six".

```
python main.py app.log --quiet
```

The score comes back 44.8/100 (D), and the findings name the incident without
you reading a single line: 229 errors in the 18:00 hour against a baseline of
4, one signature accounting for 72% of them, the `db` module at a 59% error
rate, and — the part that is easy to miss by eye — a twelve minute stretch at
22:47 with no log output at all. Running it again without `--quiet` prints the
cluster: `DatabaseConnectionError`, 275 occurrences, 18:15:00 to 19:19:57,
with a rule-based guess at the cause.

**A deploy that made things worse.** Save a run before the deploy and another
after, and the second one prints the difference:

```
python main.py app.log --save --label "before"
# deploy
python main.py app.log --save --label "after"
```

```
Sağlık skoru     87.9 → 44.8  ▼ (-43.1)   KÖTÜLEŞME
Yeni ortaya çıkan hatalar:
  + CircuitBreakerOpenError (20)
```

`CircuitBreakerOpenError` did not exist in the earlier run. That is not a
guess: error signatures are stored per run, so "new" means literally absent
from the previous set. As a CI gate:

```
python main.py app.log --fail-on health --min-health 80
```

**A spike of 5xx on the front end.** Point it at the access log and narrow to
the window you care about:

```
python main.py /var/log/nginx/access.log --since 18:00 --until 20:00 --level ERROR
```

Access logs have no level field, so the HTTP status supplies one — 5xx becomes
ERROR, 4xx WARNING. The top-errors table then groups by request shape rather
than by URL, because ids and query strings are masked out of the signature.

## Formats

| Name | Example |
|------|---------|
| `standard` | `2026-08-22 10:31:02 ERROR [db] Database connection failed` |
| `bracket` | `[2026-08-22 10:31:02] [ERROR] [db] Database down` |
| `python_logging` | `2026-08-22 10:31:02,123 - auth - ERROR - Bad token` |
| `json` | `{"timestamp":"...","level":"ERROR","message":"..."}` |
| `syslog_rfc5424` | `<11>1 2026-08-22T18:15:00Z web-01 nginx 1234 ID - refused` |
| `syslog` | `Aug 22 18:15:00 web-01 nginx[1234]: connection refused` |
| `access_log` | `1.2.3.4 - - [22/Aug/2026:18:15:00 +0300] "GET / HTTP/1.1" 500 12` |

The format is detected per file, and lines that do not match the dominant
format fall back to the other parsers, so mixed files still work.

JSON field names are not standardised, so the known spellings are all tried:
OpenTelemetry (`severity_text`, `body`), Elastic ECS (`log.level`,
`@timestamp`, `service.name`) and Python's own (`levelname`, `asctime`,
`name`). Unrecognised keys are kept on the entry. Timestamps may be ISO
strings or epoch numbers; offset-aware values are normalised to UTC so a file
mixing both does not break comparisons.

Two of these carry no level field, so one is derived. Syslog encodes severity
in the priority number (`<11>` is facility 1, severity 3, an error), and it is
decoded from there; a BSD line without a priority falls back to a level word at
the start of the message, and INFO otherwise. Access logs have a status code
instead: 5xx becomes ERROR, 4xx WARNING, the rest INFO. RFC 3164 also omits the
year, so the current one is assumed.

Adding a format means adding one class with an `extract(line)` method.

## Anomaly detection

Anomalies are scored against the file's own baseline rather than a fixed
threshold, because "too many errors" only means something relative to what
that system normally produces.

The default is a modified z-score, `0.6745 * (x - median) / MAD`. The obvious
alternative, `(x - mean) / stdev`, has a flaw that matters here: the outlier
you are looking for inflates the standard deviation and therefore hides
itself. In the sample log there are two error spikes, at 18:00 and 19:00. The
classic score only reports the first; the larger spike raises stdev to 49.6
and the second one falls inside the noise. The median and MAD are unaffected
by outliers, so both are reported.
`tests/test_anomalies.py::test_robust_beats_classic_zscore` pins this down.
The classic method is still available with `--method zscore`.

Long silences are treated as anomalies too. A stopped service or a dead log
agent produces nothing at all, which error counting never catches.

## Health score

A single 0-100 number, computed by subtracting penalties from 100. Each
factor has a ceiling, so one bad metric cannot flatten the score and make
every bad log look identical.

| Factor | Ceiling | Full penalty at |
|--------|--------:|-----------------|
| Error rate | 35 | 25% of entries |
| Critical entries | 20 | 2% of entries |
| Anomalies | 20 | 4 windows |
| Bursts | 10 | 3 bursts |
| Log stream gaps | 10 | 10% of the time span |
| Unparsed lines | 5 | nothing parsed |

Grades: A ≥ 90, B ≥ 75, C ≥ 60, D ≥ 40, F below. The weights are a judgement
call, not a derived truth; `WEIGHTS` in `src/analyzer/health.py` is the one
place to retune them.

## Live mode

`--watch` follows a file the way `tail -f` does and redraws a panel as lines
arrive, raising an alert on CRITICAL entries, on error bursts, and the first
time an error signature appears.

"First time seen" only means something once there is a baseline, so signature
alerts stay off during a warm-up period (50 entries and 30 seconds of log
time). Without it every signature is new at startup and the panel fills with
alerts that carry no information. Bursts and CRITICAL entries alert
immediately, since neither depends on history.

```
╔══════════════════════════════════════════════════════╗
║ LOGLENS LIVE  ·  15 dakikalık pencere                ║
╠══════════════════════════════════════════════════════╣
║ Pencere 18:15:10 → 18:15:30                          ║
║ Toplam  13 kayıt  (başlangıçtan beri 13)             ║
║ Hata    %100.0   son 60sn: 13                        ║
║ Sağlık  42 / 100  [D]                                ║
╠══════════════════════════════════════════════════════╣
║ DatabaseConnectionError                 12           ║
╚══════════════════════════════════════════════════════╝

Uyarılar:
  18:15:19 [BURST] 10 hata / 60 saniye
  18:15:30 [CRITICAL] Service unavailable: all database replicas down
```

Analysis runs over a rolling window (`--window`, 15 minutes by default), so
the numbers describe what is happening now rather than the whole file. File
rotation and truncation are detected by watching the inode and the file size,
so the follower survives `logrotate` and `> app.log`.

To try it without a live service, replay an existing file into another one
from a second terminal:

```
python tools/replay_log.py sample_logs/server.log sample_logs/live_test.log --speed 5
python main.py sample_logs/live_test.log --watch
```

## History

`--save` writes the run to SQLite (`loglens.db` by default, `--db` to change
it) and prints the difference against the previous run:

```
── ÖNCEKİ ANALİZLE KARŞILAŞTIRMA ───────────────────────
#1 2026-08-23 16:17  →  #2 2026-08-23 16:17

  Sağlık skoru     87.9 → 44.8  ▼ (-43.1)   KÖTÜLEŞME
  Hata oranı        8.7% → 30.0%  ▲ (+21.4 puan)
  Anomali             0  → 2  ▲ (+2)

  Yeni ortaya çıkan hatalar:
    + CircuitBreakerOpenError (20)
    + Service unavailable: all database replicas down (3)

  Sayısı değişenler:
    ▲ DatabaseConnectionError: 1 → 293 (+29200%)
```

Error signatures are stored per run, which is what makes "this error type is
new" and "this one is gone" answerable rather than guesswork. `--prune N`
keeps the newest N runs.

Note that SQLite needs real file locking, so a database on a network share or
a syncing folder may fail to open; pass `--db` with a local path in that case.

## AI commentary

Everything above is computed in Python. `--explain` adds an optional layer on
top: the findings are handed to a language model, which writes the incident up
in prose — probable root cause, alternative explanations, what to check next.

```
export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY, or LOGLENS_API_KEY
python main.py app.log --explain
python main.py app.log --explain --ai-provider openai --ai-model gpt-4o-mini
```

Without a key the flag prints why it did nothing and the analysis is unchanged.
A network failure, a bad key or an unexpected response are all reported the
same way: commentary is a bonus, and losing it must never cost the analysis.

**Raw log lines are never sent.** What goes out is the aggregated brief —
counts, signatures, timestamps, health factors — roughly 3 KB, with per-event
values masked first (`host=db-01` leaves as `host=<val>`). The masking is the
same pass used to build error signatures, so ids, paths, IPs, emails and
quoted values are replaced with placeholders.

`--ai-base-url` is not needed for OpenAI-compatible servers running locally;
set `LOGLENS_AI_BASE_URL` instead (for example `http://localhost:11434/v1`).

## Notifications

`--notify` posts the result to a webhook. Slack and Discord are recognised from
the URL and get a formatted message; anything else receives plain JSON.

```
python main.py app.log --notify "$SLACK_WEBHOOK"
python main.py app.log --notify "$WEBHOOK" --notify-on anomaly
python main.py logs/ --notify "$WEBHOOK" --notify-on health --min-health 80
```

`--notify-on` decides when a run is worth a message: `health` (default, below
`--min-health`), `anomaly`, `critical`, `error` or `always`. A run that does not
meet the condition says so and sends nothing. A webhook that cannot be reached
is reported as a warning — a failed notification never fails the analysis.

## Dashboard

`--serve` starts a local read-only dashboard over the saved history: the
latest score, the trend, a table of runs, and a detail panel per run.

```
python main.py --serve            http://127.0.0.1:8000
python main.py --serve 9000 --open
```

| Endpoint | Returns |
|----------|---------|
| `/` | the page |
| `/api/runs?limit=N` | saved runs, newest first |
| `/api/runs/{id}` | one run with its factors and error signatures |
| `/api/series?limit=N` | health score over time |

It is built on `http.server` rather than a web framework, which keeps the
whole tool installable with nothing but Python. The endpoints are plain JSON,
so swapping in FastAPI later would not change the contract. The server binds
to localhost and only reads.

## Docker

The image installs the package and runs the dashboard by default:

```
docker compose up -d
```

`./logs` is mounted read-only at `/logs` and `./data` holds the history
database, so saved runs survive a rebuild. Because the entry point is the CLI
itself, any command works against the same volumes:

```
docker compose run --rm dashboard /logs/app.log --save --db /data/loglens.db
docker compose run --rm dashboard /logs/app.log --json --fail-on anomaly
```

The container runs as an unprivileged user and has a healthcheck against
`/api/health`.

## CI

`--fail-on` controls the exit code: `error` (default), `critical`, `anomaly`,
`health`, or `none`. Exit 2 means the input could not be read.

```
python main.py logs/ --json --fail-on anomaly
python main.py logs/ --fail-on health --min-health 80
```

## Layout

```
loglens/cli.py                  argument parsing and wiring
loglens/parser/formats.py       format rules, timestamp parsing
loglens/parser/log_parser.py    reading, merging, filtering
loglens/analyzer/statistics.py  counting and aggregation
loglens/analyzer/patterns.py    repeats, bursts, clusters
loglens/analyzer/anomalies.py   baseline and deviation
loglens/analyzer/health.py      health score
loglens/analyzer/insights.py    findings in plain language
loglens/analyzer/explain.py     optional LLM commentary
loglens/database/database.py    run history and comparison
loglens/notify.py               webhook notifications
loglens/watcher.py              live following and rolling window
loglens/server.py               dashboard HTTP server
loglens/reports/                console, HTML, JSON, dashboard
tools/make_sample_logs.py       sample data generator
tools/replay_log.py             replays a log to test --watch
tools/make_screenshot.py        renders terminal output as SVG
```

Everything under `loglens/analyzer` is pure: it returns data and never prints,
so the same code backs the terminal, the HTML report, the JSON output and the
dashboard.

## Development

```
pip install -e ".[dev]"
ruff check .
pytest
python tools/make_sample_logs.py
```

303 tests, no network access from any of them. The sample generator produces a
day of traffic containing a deliberate database outage between 18:15 and 19:20,
which is what the pattern and anomaly tests are written against.

See `CONTRIBUTING.md` for the module boundaries and how to add a log format,
and `docs/KARARLAR.md` (Turkish) for why the design decisions were made the
way they were.

`docs/YOL_HARITASI.md` has the detailed roadmap (Turkish).

## License

MIT
