Metadata-Version: 2.4
Name: viclix-sdk
Version: 0.0.2
Summary: Traffic logging (IPs, users, sessions, endpoints) for FastAPI apps, with a built-in viewer, ready for Viclix
License: MIT
Keywords: fastapi,asgi,middleware,logging,observability,viclix
Classifier: Framework :: FastAPI
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Logging
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.100; extra == "fastapi"
Provides-Extra: dev
Requires-Dist: fastapi>=0.100; extra == "dev"
Requires-Dist: httpx>=0.24; extra == "dev"
Requires-Dist: pytest>=7; extra == "dev"

# viclix-sdk

Traffic logging for **FastAPI** apps, ready to visualize in **Viclix**.

Installs an ASGI middleware that records **every request** (IP, user, endpoint,
time, duration, status…) into a **dedicated JSONL file**. Writing happens on a
**background thread** with the file kept open, so the request path never does
disk I/O. Those logs then feed the animated Viclix visualization: who came in,
from which IP, to which pages and in what order.

## Installation

```bash
pip install viclix-sdk
```

## Minimal usage

```python
from fastapi import FastAPI
from viclix_sdk import MiddlewareLogger

app = FastAPI()
app.add_middleware(MiddlewareLogger)
```

That already creates the `.viclix/middleware/` folder and writes
`traffic-YYYY-MM-DD.jsonl` there.

## Configuration

```python
app.add_middleware(
    MiddlewareLogger,
    log_path=".viclix/middleware/traffic.jsonl",  # base path (the date is appended)
    trust_proxy=True,                       # real IP from X-Forwarded-For (PythonAnywhere)
    rotation="daily",                       # "daily" or None
    include_query=False,                    # include the query string (careful with sensitive data)
    ignore_paths=["/static", "/health"],    # prefixes that are not logged
)
```

| Parameter       | Default                  | Description                                             |
|-----------------|--------------------------|---------------------------------------------------------|
| `log_path`      | `.viclix/middleware/traffic.jsonl` | Base path of the log.                         |
| `trust_proxy`   | `True`                   | Reads the IP from `X-Forwarded-For` / `X-Real-IP`.     |
| `user_resolver` | reads `request.state.user` | `Callable[[scope], id]` to extract the user.          |
| `rotation`      | `"daily"`                | Daily file rotation, or `None`.                        |
| `include_query` | `False`                  | Includes the query string in the event.                |
| `ignore_paths`  | `[]`                     | Path prefixes to ignore.                               |
| `track_sessions`| `True`                   | Anonymous session cookie (`viclix_sid`) per visitor.   |
| `session_cookie`| `"viclix_sid"`           | Name of the session cookie.                            |
| `session_ttl_days` | `365`                 | Lifetime of the session cookie.                        |

### Sessions (trace a visitor's flow)

With `track_sessions=True` (the default), each visitor gets an anonymous
`viclix_sid` cookie and every event carries a `session` field. This lets you
follow a single visitor's full sequence of pages **even without a login** — for
example, to see the flow that led a user to a 4xx/5xx error:

```
GET /            -> 200   sess=44ef64b0
GET /checkout    -> 200   sess=44ef64b0
GET /boom        -> 500   sess=44ef64b0   ← the flow that ended in an error
```

### Identifying the user

By default `request.state.user` is read (whatever your auth layer leaves there).
If you store an object, `username`, `email`, `id`, `pk` are tried in order. For
custom logic:

```python
def resolver(scope):
    state = scope.get("state") or {}
    user = state.get("user")
    return getattr(user, "email", None)

app.add_middleware(MiddlewareLogger, user_resolver=resolver)
```

## Rotation and retention (log size)

By default logs rotate **hourly**, closed files are **compressed to `.gz`** and
**deleted after 24 h**:

```python
app.add_middleware(
    MiddlewareLogger,
    rotation="hourly",       # "hourly" (default) | "daily" | None
    retention_hours=24,      # delete files older than this (None = no limit)
    max_total_mb=200,        # hard size cap: deletes the oldest (None = no cap)
    compress=True,           # gzip already-rotated files
)
```

- `retention_hours` bounds by **time**; `max_total_mb` is the safety net by
  **size** (a traffic spike or a bot can fill the disk within the retention
  window).
- Cleanup and compression run on the **background thread** at rotation time —
  never on the request path — and only touch the log's own files.
- The viewer reads `.jsonl` and `.jsonl.gz` transparently.

**The biggest saving is usually not logging static assets:**

```python
app.add_middleware(MiddlewareLogger, ignore_paths=["/static", "/assets", "/favicon.ico"])
```

Background tasks accept the same parameters:

```python
track_background_tasks(name="checkout", rotation="hourly", retention_hours=24)
```

## SQL queries (SQLAlchemy)

From the outside you see *that* a request took 2 s; never *why*. With **one line**
every event starts carrying the detail of what it did against the database:

```python
from viclix_sdk import track_db_queries

track_db_queries()              # every engine in the process
track_db_queries(engine)        # or just one
```

No query is touched: it hooks SQLAlchemy's `before/after_cursor_execute` events.
It does **not** write a line per query — that would multiply the volume. It
aggregates per request (and per background task) and attaches it as the `db`
field of the event that was already being emitted:

```json
"db": {"n": 21, "ms": 312.4, "slow": 2,
       "slowest_ms": 40.1, "slowest": "SELECT id FROM projects",
       "repeat": {"sql": "SELECT name FROM projects WHERE id = ?", "n": 20, "ms": 180.2}}
```

`repeat` is the **N+1** detector: the same query —normalized, without values—
repeated many times within the same request. In the viewer, endpoints with N+1
are flagged with `⚠` and the average number of queries.

Only the **text** of the statement is stored, never the parameters: in
SQLAlchemy the values travel separately (bind params), so no data leaks.

Options: `slow_ms` (slow-query threshold, 100 by default), `repeat_threshold`
(repeats needed to flag N+1, 5) and `max_distinct` (cap on distinct queries
tracked per request, 200).

## Exceptions

A 500 seen from the outside is an identical black box every time. The middleware
captures **unhandled** exceptions —the only place where the traceback still
exists— and adds them to the event, attributed to the endpoint, the IP and the
user:

```json
"error": {"type": "ValueError", "msg": "something broke badly",
          "where": "routes.py:142 in checkout", "traceback": "Traceback ..."}
```

Enabled by default (`capture_errors=True`) and it **does not alter the flow**:
the exception is re-raised as-is. With `error_traceback=False` only the type,
message and `file:line` are stored. Background tasks that fail log the same in
their own log (and are not duplicated in the request that created them).

No local variables are captured, only the formatted traceback.

## External services (httpx / requests)

From the outside you can't attribute "checkout was slow because Stripe took 3 s":
the proxy only sees a slow request. With **one line**:

```python
from viclix_sdk import track_outbound_calls

track_outbound_calls(slow_ms=500)
```

Wraps `httpx.Client.send`, `httpx.AsyncClient.send` and `requests.Session.send`.
Like the SQL one, it aggregates per request/task and travels as the `out` field:

```json
"out": {"n": 2, "ms": 3200.5, "slow": 1,
        "hosts": [{"host": "api.stripe.com", "n": 1, "ms": 3000.2}],
        "slowest": {"host": "api.stripe.com", "method": "POST",
                    "path": "/v1/charges", "ms": 3000.2, "status": 200}}
```

From the URL, **host and path** are kept; the **query string is dropped** because
it usually carries tokens and API keys. In the viewer there is a port (`🛰`) where
the little person waits as long as each third party took.

With *streaming* responses the timing reaches the headers, not the last byte of
the body.

## Process health

This one does go to **its own log** (`.viclix/health/<name>.jsonl`), because it
doesn't hang off any request: it's periodic sampling.

```python
from viclix_sdk import track_health

track_health(name="www", interval=15)
```

```json
{"kind": "health", "app": "www", "loop_lag_ms": 3.4,
 "rss_mb": 182.5, "threads": 12, "in_flight": 2,
 "pool": {"in_use": 3, "size": 5, "overflow": 0}}
```

The key figure is **`loop_lag_ms`**: how long the event loop takes to attend to
something that was already ready. From the outside you only see "high latency";
the lag tells you there's a **synchronous call blocking** the async loop. It's
measured from the sampling thread with `loop.call_soon_threadsafe`, so it takes
no room in the loop itself.

`rss_mb` uses `psutil` if installed, otherwise `/proc/self/statm` (Linux). `pool`
comes from the first SQLAlchemy engine seen in use, so it needs
`track_db_queries()` active. In the viewer this is the city's **weather**: it
clouds over and rains when the process is suffering.

## Hung requests and who is blocking the loop

Two parameters of `track_health`, no new lines:

```python
track_health(name="www", stuck_after=30, profile_on_lag=200)
```

**`stuck_after`** (seconds) reports requests that are **still in flight**. It's the
only way to see a request that never finishes: the normal event is written when
it ends, so a deadlock or a third party with no timeout **left no trace at all**.
They show up in the health sample while they're still hung:

```json
"stuck": [{"method": "GET", "path": "/pay/charge", "ip": "1.2.3.4", "age_s": 47.2}]
```

**`profile_on_lag`** (ms) takes the snapshot of the stacks when the loop stalls.
The trick is in the ordering: the probe waits only that long and, if the ping
hasn't come back, the loop is blocked **right now** — that's the moment to
photograph. Waiting for it to return would give the stack of the loop already at
rest, which says nothing.

```json
"blocked_by": "billing.py:61 in summarize",
"stacks": [{"thread": "MainThread", "stack": ["...", "billing.py:61 in summarize"]}]
```

## Cache (Redis)

```python
from viclix_sdk import track_cache
track_cache()               # redis.Redis and redis.asyncio.Redis
track_cache(my_client)      # or just that client
```

`cache` field on the event: `{"n": 10, "hits": 8, "misses": 2, "ms": 4.1}`.
From the outside you can't tell a cached response from a recomputed one. Only the
**command** is counted, never the key or the value.

## Authentication and security

```python
from viclix_sdk import track_security
track_security(name="www", login_paths=["/login", "/signup"])
```

Its own log (`.viclix/security/<name>.jsonl`), **30-day** retention. Detects
without touching your code: `unauthorized` (401), `denied` (403),
`login.failed` and `login.ok` on the login paths.

The important part is that it can be grouped **by account**: an attacker rotating
IPs against a single user is invisible grouped by IP and obvious by account.

**Honest limit:** from the middleware only the status code is visible. The
*reason* (wrong password vs. locked account vs. expired token) is only known by
your auth layer. For that there's an optional call wherever it matters:

```python
from viclix_sdk import record_auth
record_auth("login.failed", user=email, reason="bad_password")
```

When used, the automatic detection **stays quiet** for that request, so the same
attempt isn't logged twice.

## Startup, shutdown and deploy

```python
from viclix_sdk import track_lifecycle
track_lifecycle(name="www", version=settings.release, config=settings)
```

Its own tiny log: one or two lines per deploy. From the outside a redeploy and a
crash-loop look **identical**; here they're told apart because a `start` without
its preceding `stop` means the previous process died dirty (SIGKILL, OOM). No
signal handlers are installed, so as not to step on uvicorn's.

From the config, the **names** of non-sensitive keys and a **fingerprint** of all
values are stored — never the values. If the fingerprint changes between two
starts of the same version, someone touched the configuration.

## Response sizes

Runs on its own, with no configuration: a `bytes` field on every event. Detects
the endpoint returning 4 MB of unpaginated JSON. Disable with
`MiddlewareLogger(track_bytes=False)`.

*Unlike the rest of this README, this one isn't exclusive to the inside: your
proxy sees the bytes too. It's here because it's free.*

## Viewer inside your app (no CLI)

Two lines and you have the viewer on your own domain, without starting anything:

```python
from viclix_sdk import mount_viewer
mount_viewer(app, "/_viclix", token=settings.viclix_token)
```

You get `yourdomain.com/_viclix` and `yourdomain.com/_viclix/experimental`.

### Only your admins, no token

```python
mount_viewer(app, "/_viclix", authorize=my_admin_check)
```

`authorize` receives the ASGI `scope` and returns `True`/`False`; from there you
read the session cookie just like in `user_resolver`. If you pass both, whoever
satisfies **either** gets in.

### What it does for security

- **The token disappears from the URL.** When you arrive with a valid `?token=…`
  it's stored in an `HttpOnly; SameSite=Strict; Path=<viewer>` cookie and you're
  **redirected** to the same path without the query. The token appears once and
  is not left in history, nor in the `Referer` of later requests, nor in the
  proxy logs.
- **It responds 404, not 401.** Whoever lacks the key doesn't know the viewer
  exists.
- **It fails closed.** With no `token` and no `authorize`, or with a token shorter
  than 16 characters, `mount_viewer` raises `ValueError` at startup instead of
  serving the logs to anyone. Generate the token with `secrets.token_urlsafe(32)`.
- **Constant-time comparison** (`hmac.compare_digest`).
- **Failed attempts** are logged as `viewer.denied` in the security log, if
  `track_security()` is active.
- **The viewer does not log itself:** browsing it does not generate the events
  you're looking at.

### What it does NOT do

It doesn't encrypt anything or rate-limit attempts. And above all: **these logs
carry emails, tracebacks, SQL statements, IPs and config key names.** Mount it
only behind HTTPS and treat it like an admin panel, because that's exactly what
it is.

## Multiple apps (same venv / server)

If you have **several FastAPI apps** in the same directory/venv, give each one a
distinct `name` and each writes to **its own file**
(`.viclix/middleware/<name>.jsonl`), instead of sharing a single one:

```python
app.add_middleware(MiddlewareLogger, name="checkout")   # → checkout.jsonl
app.add_middleware(MiddlewareLogger, name="auth")       # → auth.jsonl
track_background_tasks(name="checkout")                 # tasks separated too
```

In addition, every event carries an `app` field. In the viewer:

```bash
viclix-monitor                  # aggregates ALL apps (tells you which ones)
viclix-monitor --app checkout   # only that app
```

Without `name`, all share `traffic.jsonl` (previous behavior).

## Viewer (CLI)

The package installs the `viclix-monitor` command. Running it in your terminal:

```bash
viclix-monitor
```

1. Reads the logs from `.viclix/middleware/` (all days aggregated). If that folder
   doesn't exist, it searches recursively (up to 3 levels) as a fallback.
2. Starts a local server and opens the browser with an interactive visualization.
3. Shows stats, a timeline per session/IP (with traffic **animation**), the most
   active endpoints and IPs, and a table filterable by IP, path, user or status
   class.

### Living world (experimental)

At `/experimental` (linked from the dashboard) there's an isometric
representation where **each IP is a little person** who leaves the "entrance" and
walks toward the endpoint they requested. Endpoints are grouped by class (first
path segment) into **districts** — `/products`, `/products/42` → *products*
district — and the map **starts at 0 and grows** as IPs and endpoints are
discovered. Root-level files probed by scanners (`/config.json`, `/.env`,
`/.github`, `/values.yaml`…) all fall into a single root-files district
(labeled `/ archivos`) instead of creating one district per probe. The pulse
color indicates the status class of the last request
(green/yellow/orange/red). While walking, each person carries a **thread hooked
to their shadow**: dotted for what they've already walked, solid with an arrowhead
for what's left. The color says the errand — neutral toward an endpoint, green to
the warehouse, amber if that trip is an N+1, blue to the third-party port. Toggle
it with the `➶` button next to the flow arrows.

Map controls: play, speed, drag to move, wheel to zoom, *Fit* to reframe. The
character engine is POKER's (`sprites.js`).

The **speed** control has two halves with `1×` right in the middle. `1×` is
**real time** (one second of log = one second of wall clock). The left half goes
from `0.1×` to `1×` in 0.1 steps (slow-motion, to inspect a burst with a
magnifying glass); the right half from `1×` up to `1000×` in 1-2-5 steps, to
fast-forward through long logs (a full day plays in ~86 s at `1000×`).

**Live mode** (`🔴`): follow the log in real time. The SDK polls for new events
and appends them to the world without resetting it, keeping the playhead pinned
to *now*. Taking manual control (seek, apply a range, reset) turns it off.

**Time panel.** Drag it by the `⠿` handle and it stays where you leave it
(remembered in `localStorage`); double-click the handle to re-center it.

Over the bar:

| gesture | effect |
|---|---|
| click / drag | jump to that instant |
| **wheel** | zoom **anchored to the cursor**: the instant you point at doesn't move and the bar stretches around it |
| **Shift + drag** | pan the visible window |
| double click | back to 1× zoom |
| `⟚` + drag | select a range (loops within it and proposes that range below) |

The zoom **only stretches the bar**: useful to see bursts of seconds within a
whole day. When zoomed in, a rail appears on top showing which slice of the total
you're viewing, and the window follows the playhead while playing.

**Load range** (bottom row): day + from-hour to-hour, **24 h maximum**. This is
not a zoom: it actually trims the simulated window — it recomputes the world, the
tasks, the security events and the clock. If `to` is less than `from`, it crosses
midnight (`22:00 → 06:00` = 8 h). The counter next to it previews how many
requests fall in the range before applying it, and an empty range is not applied:
it warns you. `Todo` returns to the full log.

**Counters with lists.** The counters at the bottom left (IPs, districts,
endpoints, requests, jobs and the weather) open, on **hover**, the list of what
they count; a click on the counter **pins** it until the next click.

| menu | what it lists | on clicking an item |
|---|---|---|
| IPs | each IP with **the accounts it has used** (or *anonymous*), its origin, whether it's a bot and whether it has auth rejections | opens its card and the camera **follows it** |
| Districts | the neighborhoods with their endpoints, hits and errors | district card (aggregate of the whole neighborhood) |
| Endpoints | by hits, with status, errors and SQL load (`⚠` if N+1) | endpoint card and the camera goes to it |
| Requests | the last 30 fired, with IP, user, duration, SQL and exception | jumps to whoever made it |
| Jobs | the latest runs and, below, the **task types** | the run's card or the bench's |
| Weather | loop lag, memory, in-flight, threads, pool, who blocked and the hung requests | process card (the hung ones jump to their IP) |

The same in the **ranking** on the right (endpoints, IPs, errors and origins —
an origin filters the world) and inside the cards: the endpoints listed in an IP,
a district or the warehouse are links.

This view is self-contained and public: it's the basis of what will later be
shown in Viclix.

Options:

```bash
viclix-monitor [LOG]         # path to a specific .jsonl (optional)
  --dir .                    # root search directory
  --depth 3                  # maximum search depth
  --port 8787                # local server port
  --all-days                 # aggregate all daily files of the same base
  --no-browser               # don't open the browser automatically
```

## Log format (JSONL)

One JSON line per request:

```json
{"v":1,"ts":"2026-07-24T16:50:55.349029+00:00","ip":"1.2.3.4","method":"GET","path":"/dashboard","status":200,"duration_ms":3.06,"user":"jp@sizth.com","ua":"Mozilla/5.0","referer":null}
```

| Field         | Description                                    |
|---------------|------------------------------------------------|
| `v`           | Schema version.                                |
| `ts`          | ISO-8601 timestamp in UTC.                     |
| `ip`          | Client IP.                                     |
| `method`      | HTTP method.                                   |
| `path`        | Requested path.                                |
| `status`      | Response code.                                 |
| `duration_ms` | Request duration in ms.                        |
| `user`        | User identifier (or `null`).                   |
| `session`     | Anonymous session ID (`viclix_sid` cookie).    |
| `ua`          | User-Agent.                                    |
| `referer`     | Referer.                                       |
| `query`       | Query string (only if `include_query=True`).   |

## Performance notes

- Events are queued in memory and written by a daemon thread; the request does
  not wait on disk.
- If the queue saturates (`queue_maxsize`, 10k by default), new events are
  **dropped** instead of blocking the app. The writer keeps `written` and
  `dropped` counters.
- The file is flushed every second and closed cleanly when the process ends
  (`atexit`).
