Metadata-Version: 2.5
Name: visionary-sdk
Version: 0.5.2
Summary: Visionary collector: logs API requests, database and HTTP spans, errors and system metrics into your database.
Project-URL: Homepage, https://github.com/styropyr0/Visionary
Project-URL: Documentation, https://github.com/styropyr0/Visionary/tree/main/sdks/python#readme
Project-URL: Source, https://github.com/styropyr0/Visionary/tree/main/sdks/python
Project-URL: Issues, https://github.com/styropyr0/Visionary/issues
Author: styropyr0
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: analytics,api,asgi,django,fastapi,flask,metrics,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Django
Classifier: Framework :: FastAPI
Classifier: Framework :: Flask
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: psutil>=5.9; extra == 'all'
Requires-Dist: psycopg>=3.1; extra == 'all'
Requires-Dist: pymysql>=1.0; extra == 'all'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.0; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg>=3.1; extra == 'postgres'
Provides-Extra: system
Requires-Dist: psutil>=5.9; extra == 'system'
Description-Content-Type: text/markdown

# visionary-sdk (Python)

The Python collector for [Visionary](../../Readme.md). It runs inside your web
service and writes:

- one row per HTTP request into `api_logs` (method, route pattern, status,
  timing, user, redacted headers and bodies, error codes, and how many
  database and outbound HTTP calls it made),
- the database and HTTP calls of slow or failed requests into `spans`,
- unhandled and reported exceptions into `errors`,
- one `releases` row per deploy (when the release is known), and
- a periodic host / process / runtime sample into `system_metrics`.

The Visionary server reads these tables for its dashboards. All slow work
happens on background threads, so requests are never blocked or slowed. If
the database is down, the SDK only writes log lines.

Every Visionary SDK behaves the same way. The contract is
[`sdks/SPEC.md`](../SPEC.md).

## Install

```sh
pip install visionary-sdk                 # core: stdlib only (D1, SQLite)
pip install "visionary-sdk[system]"       # + psutil: host CPU/memory/network, process CPU/RSS/threads/fds
pip install "visionary-sdk[postgres]"     # + psycopg 3
pip install "visionary-sdk[mysql]"        # + PyMySQL
```

Requires Python 3.9+. Without the `system` extra, Linux hosts read every host
and process column from `/proc` instead. On macOS and Windows most of them are
NULL; disk usage, thread count, GC and HTTP throughput are still recorded.

## Quickstart

First configure a database (see [Choosing the database](#choosing-the-database)). For example:

```sh
export VISIONARY_DATABASE_URL=postgres://user:pass@localhost:5432/app
```

### Django

```python
# settings.py
MIDDLEWARE = [
    "visionary.integrations.django.VisionaryMiddleware",  # first, so it sees the final response
    # ...
]
```

The middleware calls `visionary.init()` the first time it loads, so nothing
else is needed. It works under WSGI and ASGI, with sync and async views.

- **Route:** `request.resolver_match.route` with a leading `/`, e.g. `/users/<int:id>/`.
- **User:** if you don't call `set_user`, an authenticated `request.user` is
  used (`pk` as the id, `get_username()` as the handle).

### FastAPI / Starlette (any ASGI app)

```python
from fastapi import FastAPI
from visionary.integrations.asgi import VisionaryASGIMiddleware

app = FastAPI()
app.add_middleware(VisionaryASGIMiddleware)
# or, for any ASGI app: app = VisionaryASGIMiddleware(app)
```

The route is the matched route's path (`/users/{user_id}`). Streaming
responses keep streaming. Only the first `VISIONARY_LOG_MAX_BODY_BYTES` of a
body are copied.

### Flask

```python
from flask import Flask
from visionary.integrations.flask import Visionary

app = Flask(__name__)
Visionary(app)            # or: ext = Visionary(); ext.init_app(app)
```

The route is `request.url_rule.rule` (`/users/<int:user_id>`).

### Anything else

```python
import visionary

agent = visionary.init()   # reads the environment, starts, flushes at exit
```

`init()` never raises. If no database is configured, it logs one warning and
returns a no-op agent, and the integrations pass requests straight through.

For explicit control (tests, several agents, custom config):

```python
cfg = visionary.from_env()          # or visionary.Config(...)
cfg.path_prefix = "/api"
agent = visionary.Agent(cfg).start()
app.add_middleware(VisionaryASGIMiddleware, agent=agent)
...
agent.close(timeout=10)             # drains the queue and flushes samples
```

## Choosing the database

The SDK checks these in order:

1. **Your own connection.** Use this when the app already has a driver:

   ```python
   import psycopg, visionary

   cfg = visionary.from_env()
   cfg.connection_factory = lambda: psycopg.connect(DSN)
   cfg.dialect = "postgres"            # "postgres", "mysql" or "sqlite"
   visionary.init(cfg)
   ```

   The factory must return a new DB-API connection. Visionary keeps that
   connection for its own use:
   - PostgreSQL and MySQL: one shared connection.
   - SQLite: one connection per worker thread.

   After an error it reconnects through the factory, and it closes the
   connection at shutdown. Each batch runs as one transaction. If you hand it
   an autocommit connection, batches are not atomic.

2. **`VISIONARY_DATABASE_URL`**, chosen by scheme:

   | URL | Backend |
   |---|---|
   | `postgres://user:pass@host:5432/db` (or `postgresql://`) | PostgreSQL via psycopg 3 |
   | `mysql://user:pass@host:3306/db` | MySQL 8+ / MariaDB via PyMySQL |
   | `sqlite:///absolute/path.db`, `sqlite:///C:/data/x.db`, `sqlite://relative.db` | SQLite via `sqlite3` |
   | `d1://` | Cloudflare D1, credentials from `D1_*` |

3. **Cloudflare D1**, when `D1_ACCOUNT_ID`, `D1_DATABASE_ID` and
   `D1_API_TOKEN` are set. `D1_API_BASE` overrides the API URL.

Tables and indexes are created at startup from the schema files shipped in
the package (`visionary/schema/<dialect>/`). This happens on the background
threads, so a slow database never delays boot. If the database is down at
startup, creation is retried before each write.

## Configuration

Every setting has an environment variable, and `visionary.Config` has the
same settings as attributes. The full list and semantics are in
[SPEC.md, "Environment"](../SPEC.md#environment).

| Variable | Default | Notes |
|---|---|---|
| `VISIONARY_DATABASE_URL` | | see above |
| `D1_ACCOUNT_ID`, `D1_DATABASE_ID`, `D1_API_TOKEN`, `D1_API_BASE` | | D1 only |
| `VISIONARY_INSTANCE` | hostname | labels `system_metrics` rows |
| `VISIONARY_LOG_REQUESTS` | `true` | request logging on/off |
| `VISIONARY_LOG_PATH_PREFIX` | empty | only log paths starting with this |
| `VISIONARY_LOG_BODIES` | `true` | capture request/response bodies |
| `VISIONARY_LOG_MAX_BODY_BYTES` | `8192` | per body |
| `VISIONARY_LOG_RETENTION_DAYS` | `30` | `0` keeps forever |
| `VISIONARY_LOG_REDACT_HEADERS`, `_REDACT_FIELDS`, `_REDACT_REQUEST_FIELDS` | empty | comma lists added to the defaults |
| `VISIONARY_LOG_ERROR_CODE_FIELD`, `_ERROR_KEY_FIELD` | `error.code`, `error.key` | dotted JSON paths |
| `VISIONARY_SYSTEM_METRICS` | `true` | sampler on/off |
| `VISIONARY_SYSTEM_INTERVAL` | `15s` | `15`, `500ms`, `1m` |
| `VISIONARY_SYSTEM_RETENTION_DAYS` | `30` | |
| `VISIONARY_SERVICE` | empty | service name on every row |
| `VISIONARY_RELEASE` | platform commit variable | release on every row; see [Telemetry](#telemetry-spans-errors-releases) |
| `VISIONARY_SPANS` | `true` | spans and call totals on/off |
| `VISIONARY_SPAN_THRESHOLD_MS` | `100` | store spans of requests at least this slow (or 5xx) |
| `VISIONARY_SPAN_MAX` | `100` | span rows per request; later calls still count in the totals |
| `VISIONARY_ERRORS` | `true` | error capture on/off |
| `VISIONARY_INSTRUMENT_HTTP` | `false` | Python only: call `instrument_http()` at start |

Booleans accept `true/false/1/0/yes/no`.

## Telemetry (spans, errors, releases)

### Service and release

`VISIONARY_SERVICE` names the service, so several services can share one
database. The release (`version`) is `VISIONARY_RELEASE`, else the first of
`RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `VERCEL_GIT_COMMIT_SHA`,
`HEROKU_SLUG_COMMIT`, `SOURCE_VERSION`, `GIT_COMMIT`, `GITHUB_SHA`,
`K_REVISION`; commit hashes are cut to 12 characters. Both go on every
`api_logs`, `system_metrics` and `errors` row (`spans`: service only). When
the release is known, one `releases` row (`sdk = python`, `runtime` such as
`CPython 3.13.1`) is written at start; the dashboards mark deploys with it.

### Spans

Every call the SDK can see is counted into `db_count`, `db_ms`, `http_count`
and `http_ms` on the request's `api_logs` row (NULL for a kind of call that is
not instrumented). When the request took at least `VISIONARY_SPAN_THRESHOLD_MS`
or answered 5xx, its first `VISIONARY_SPAN_MAX` calls are also stored as
`spans` rows, in start order. Spans outside a request are stored when they
take at least the threshold themselves, with `request_id` NULL.

The request is found through a contextvar, so spans from sync views, thread
pools that copy the context (FastAPI sync endpoints, `asgiref`,
`contextvars.copy_context().run`) and asyncio tasks all land on the right
request.

| Source | How |
|---|---|
| Django ORM / `connection.cursor()` | automatic: the middleware adds a wrapper to `execute_wrappers` of every database alias |
| SQLAlchemy (sync or async) | `visionary.instrument_sqlalchemy(engine)` |
| psycopg 3 (sync/async), psycopg2, PyMySQL, sqlite3 | `conn = visionary.instrument_connection(conn)` (or `instrument_cursor(cur)`) |
| `requests`, `httpx` (sync and async), `urllib3` | `visionary.instrument_http()` once at startup, or `VISIONARY_INSTRUMENT_HTTP=true` |
| anything else | `with visionary.span("cache", "GET session"):` (also `async with`) |

```python
engine = visionary.instrument_sqlalchemy(create_engine(DSN))

with visionary.span("queue", "publish order.created", target="rabbitmq") as s:
    channel.basic_publish(...)
    s.set_row_count(1)       # optional; s.set_error(...) marks it failed
```

Database spans are named after the first keyword and main table
(`SELECT users`, `INSERT bookings`, `BEGIN`), with the target
`system/database` (`postgresql/app`, `sqlite/app.db`). The statement is stored
sanitized: string and number literals become `?`, `IN (?, ?, ...)` becomes
`IN (?)`, whitespace is collapsed. Parameter values are never stored. HTTP
spans are named `METHOD host`; the detail is the URL without credentials,
query or fragment. An HTTP span fails on a transport error or a 5xx answer.
A call through `requests` makes one span, not a second one for urllib3.

The SDK's own writes are never recorded as spans.

### Errors

The Django, ASGI and Flask integrations record every exception the
framework turns into a 500 (`handled = 0`) and leave the exception and the
framework's response unchanged. 4xx exceptions (`Http404`, `HTTPException`,
`abort(404)`) are not errors. To record one you handled:

```python
try:
    charge(card)
except PaymentError:
    visionary.capture_exception()      # or capture_exception(exc)
```

Inside a request the row carries its `request_id`, method, route, path and
user. `type` is the class name, qualified with its module unless it is a
builtin (`ValueError`, `psycopg.errors.UndefinedTable`); `stack` is
`traceback.format_exception` (the last 16 KB).

`fingerprint` groups the same error: the first 16 hex characters of
`sha1(type + "\n" + frames)`, where `frames` are the 5 innermost frames of
your own code (not the standard library, `site-packages` or this SDK), each
written `function@path` with the path relative to its `sys.path` entry
(`shop/views.py`), joined with `\n`. If no frame is your code, the 5
innermost frames are used; with no frames at all, the message with digit
runs replaced by `0`. At most 20 rows per fingerprint per minute are kept.

Retention: `spans` and `errors` follow `VISIONARY_LOG_RETENTION_DAYS`;
`releases` are kept.

## Users

```python
import visionary

visionary.set_user(user.id, handle=user.email, role="admin")
```

Call `set_user` from auth middleware or a view once the user is known. Only
`id` is required. It is contextvar-based, so it works from sync views, thread
pools (FastAPI sync endpoints) and asyncio tasks. Outside a request handled
by an integration, it does nothing.

`visionary.set_route("/custom/{pattern}")` works the same way for the route,
for routers the integration can't see.

## Hooks

```python
cfg = visionary.from_env()

def lookup_user(user_id):
    """Fill in handle/role for ids logged without them. Runs on the writer
    thread, once per distinct user per batch; keep it cheap (cache)."""
    u = USER_CACHE.get(user_id)
    return (u.email, u.role) if u else None

def db_stats():
    """Your pool's stats for the db_* columns. Counters are cumulative;
    Visionary stores per-sample deltas."""
    s = pool.get_stats()                     # e.g. psycopg_pool
    return {
        "total": s["pool_size"], "idle": s["pool_available"],
        "in_use": s["pool_size"] - s["pool_available"], "max": s["pool_max"],
        "acquires": s.get("requests_num", 0), "empty_acquires": s.get("requests_queued", 0),
        "acquire_wait_ms": s.get("requests_wait_ms", 0),
    }

cfg.lookup_user = lookup_user
cfg.db_stats = db_stats
visionary.init(cfg)
```

If `db_stats()` leaves out a key, that column is stored as NULL.

With Django 5.1+ PostgreSQL pooling (`"OPTIONS": {"pool": ...}`) you don't need
the hook: when `db_stats` is unset, the Django middleware sums the stats of
Django's pools the same way. Without a pool the `db_*` columns stay NULL.

## What is captured

Each request (`api_logs`) records:

- **Request id:** the incoming `X-Request-ID` if it is at most 128 characters,
  otherwise a random 32-hex id. It is echoed on the response.
- **Timing:** start time (`2026-10-17T17:34:00.123Z`) and duration in ms.
- **Request line:** method, path, route pattern and query string.
- **Status:** the final status (500 when the handler raised).
- **Client:** IP (first `X-Forwarded-For` hop, otherwise the peer address) and user agent.
- **User:** id, handle and role.
- **Headers:** all request headers as JSON.
- **Bodies:** content types, bodies up to the cap with `*_truncated` flags,
  and full body sizes.
- **Errors:** for `status >= 400`, the error code and key from the JSON response body.

Bodies are not captured for `multipart/*`, `image/*`, `video/*`, `audio/*`,
`application/octet-stream`, `application/zip` and `application/pdf`.

Streaming responses are counted, and their first bytes are copied without
buffering. File responses (`FileResponse`, Flask `send_file`) keep their
sendfile path, so only their `Content-Length` is recorded.

Each sample (`system_metrics`) records:

- **Host:** CPU, load average (Unix only), memory, disk usage of the working
  directory, and network rates.
- **Process:** CPU, RSS, threads and open files.
- **Runtime:** live Python threads (in the `goroutines` column), the
  `tracemalloc` heap when tracing is on, and GC collections and pause time
  (via `gc.callbacks`).
- **Database pool:** whatever your `db_stats` hook returns, or Django's
  PostgreSQL pools when it isn't set.
- **HTTP:** requests, 5xx responses, requests per second and in-flight
  requests seen by the integrations. These are counted even when request
  logging is off.
- **Node (node-exporter style):**
  - CPU: count, user / system / iowait / steal share.
  - Memory: available and page-cache memory, swap.
  - Disk: total size, inode use, read/write KB/s and IOPS summed over whole
    disks (partitions skipped), average I/O wait (await), and busy % of the
    busiest disk.
  - Network: packets/s, errors and drops.
  - TCP: connections established and retransmits.
  - Processes on the host, running and blocked.
  - Context switches/s.
  - This process: read/write KB/s and its open-file limit.

Sources that don't exist on the platform are NULL:

- **Linux only:** iowait, steal, disk busy, the TCP columns,
  `procs_running` / `procs_blocked`.
- **Not on Windows:** page cache, inodes and the open-file limit.
- **Windows resolution:** disk await comes from whole-second counters there,
  so it is coarse.

On Linux the host, process and node columns are read from `/proc` when
psutil isn't installed, so all of them are filled either way.

Older `api_logs` and `system_metrics` tables are upgraded in place at
startup: missing columns are added with `ALTER TABLE ... ADD COLUMN`, and
existing rows keep NULL in them.

## Redaction

Redaction runs on the writer thread before anything is stored, and all
matching is case-insensitive.

- **Headers:** `Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`,
  `X-Api-Key` and `X-Auth-Token` (plus `VISIONARY_LOG_REDACT_HEADERS`) become
  `[REDACTED]`.
- **JSON keys:** `password`, `passwd`, `secret`, `client_secret`, `token`,
  `access_token`, `refresh_token`, `id_token`, `api_key`, `apikey`, `otp`,
  `pin`, `card_number`, `cvv` and `private_key` (plus
  `VISIONARY_LOG_REDACT_FIELDS`) are redacted in request and response bodies.
  `VISIONARY_LOG_REDACT_REQUEST_FIELDS` adds keys redacted in request bodies
  only.
- **Truncated bodies:** matching uses a regex, so bodies cut at the cap are
  still scrubbed.
- **Form bodies:** `application/x-www-form-urlencoded` request bodies are
  scrubbed the same way (`password=[REDACTED]`).

## Operational notes

- **Request queue:** holds 5000 entries. When it is full, new entries are
  dropped and the count is logged.
- **Request writes:** batches of up to 50 are written every 3 s. A failed
  batch is retried up to 3 times (1 s, then 2 s backoff), then dropped with an
  error log.
- **System writes:** samples are written 4 at a time. On failure they are kept
  (at most an hour's worth) and retried with the next write.
- **Retention:** runs 60 s after start, then daily, deleting in chunks of 5000.
- **Telemetry writes:** spans, errors and the release row share the request
  queue and batches (one transaction, or one D1 `batch` call, per flush).
- **Logging:** all messages go to the `visionary` logger.
- **Forking servers:** with servers like `gunicorn --preload`, the background
  threads are restarted in each worker after fork.
