Metadata-Version: 2.4
Name: filter-json-log
Version: 0.7.1
Summary: Safe JSON serialization for logs
Author-email: Alex Semenyaka <alex.semenyaka@gmail.com>
License-Expression: MIT
Keywords: logging,json,redaction,sanitization,security,secrets,credentials,privacy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security
Classifier: Topic :: System :: Logging
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: filter-url>=1.2.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-cov>=5.0; extra == "test"
Requires-Dist: hypothesis>=6.0; extra == "test"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: hypothesis>=6.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: build>=1.2.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# filter-json-log

A configurable Python utility to serialize JSON-like data for logs without leaking credentials.

`filter-json-log` redacts secret-like fields, sanitizes URLs through `filter-url`, scans emitted strings through a shared sanitizer pipeline, and caps rendered output size so logs stay safe and readable.

## Key Features

- **JSON-safe logging output**: serializes `dict`, `list`, `tuple`, `set`, scalars and raw JSON strings into valid JSON text suitable for logs.
- **Systematic text sanitization**: JSON keys, string values, URL components, headers, query strings, form bodies, environment-like data, command strings and rendered log messages all pass through the same sanitizer pipeline.
- **URL-aware filtering**: URL-like fields are preserved as URLs while userinfo, query values, fragments, path segments and nested URL/query values are redacted where needed.
- **Policy modes**: choose `light`, `normal`, `paranoid` or `lockdown` depending on how much readability you are willing to trade for safety.
- **Configurable rules**: replace or extend default secret markers, URL markers, provider-token regexes and language/domain marker packs.
- **Language marker packs**: built-in packs include technical English, French, German, Ukrainian and Russian markers.
- **Logging integration**: includes a `logging.Filter` subclass that sanitizes `extra`, positional/mapping arguments and already-rendered messages.
- **Output budget control**: defaults to `max_len=4096` and `max_nls=8`; when the output cannot fit safely, visible excess is replaced with `"[...]"`.
- **Strict raw JSON mode**: with `raw_str=True`, strings must parse as JSON; parse failures render as `"[JSON parsing error]"` instead of being silently treated as normal strings.

## Installation

```bash
pip install filter-json-log
```

`filter-json-log` is designed to use `filter-url` for URL sanitization. Package metadata should install it as a dependency; if you are using the module directly from source, install it explicitly:

```bash
pip install filter-url
```

## Quick Start

The quickest way to use the library is the standalone `filter_json_log()` function.

```python
from filter_json_log import filter_json_log

payload = {
    "user": "alice",
    "password": "correct horse battery staple",
    "url": "https://user:secret@example.com/login?token=abc-123",
}

safe = filter_json_log(payload)
print(safe)
```

Output:

```json
{
  "user": "alice",
  "password": "[REDACTED]",
  "url": "https://[REDACTED]@example.com/login?token=[REDACTED]"
}
```

Use `one_line=True` for compact log messages:

```python
safe = filter_json_log(payload, one_line=True)
print(safe)
```

Output:

```json
{"user":"alice","password":"[REDACTED]","url":"https://[REDACTED]@example.com/login?token=[REDACTED]"}
```

## Usage & Examples

### Basic Filtering: Standalone Function

`filter_json_log()` is convenient for one-off calls. It creates a configured `FilterJSONLog` instance internally and immediately renders the supplied data.

```python
from filter_json_log import filter_json_log

payload = {
    "request_id": "req-123",
    "headers": {
        "Authorization": "Bearer very-secret-token",
    },
    "redirect_url": "https://client.example/cb#access_token=abc&id_token=jwt",
}

print(filter_json_log(payload, one_line=True))
```

Output:

```json
{"request_id":"req-123","headers":{"Authorization":"[REDACTED]"},"redirect_url":"https://client.example/cb#access_token=[REDACTED]&id_token=[REDACTED]"}
```

### Raw JSON Strings

By default, a Python string is treated as a JSON string value, not as a JSON document. If you want to parse it as JSON, pass `raw_str=True`.

```python
from filter_json_log import filter_json_log

raw = '{"password":"secret","url":"https://x.test/?token=abc"}'
print(filter_json_log(raw, raw_str=True, one_line=True))
```

Output:

```json
{"password":"[REDACTED]","url":"https://x.test/?token=[REDACTED]"}
```

If `raw_str=True` is used and the string is not valid JSON, the result is explicit:

```python
print(filter_json_log("not json", raw_str=True))
```

Output:

```json
"[JSON parsing error]"
```

This is intentional: when callers promise JSON, malformed input should be visible in the log without leaking the original string.

### Advanced: Reuse `FilterJSONLog` for Performance

When processing many payloads with the same rules, instantiate `FilterJSONLog` once. This compiles marker rules, regexes and URL filters only once.

```python
from filter_json_log import FilterJSONLog

json_filter = FilterJSONLog(
    mode="normal",
    one_line=True,
    max_len=4096,
    max_nls=8,
)

payloads = [
    {"url": "https://api.example/data?api_key=k1"},
    {"headers": "Authorization: Bearer k2"},
    {"message": "password k3"},
]

safe_payloads = [json_filter.filter(payload) for payload in payloads]
```

### Policy Modes

`mode` controls how aggressively text is sanitized.

| Mode | Intended use | Behavior |
|---|---|---|
| `light` | Maximum readability, legacy systems | Redacts secret-like keys, URL-like fields and strong provider tokens. Free text is touched less aggressively. |
| `normal` | Recommended default | Every emitted text atom is sanitized. Marker + payload patterns, URLs, headers, query/form/env/cmd contexts and encoded views are checked. |
| `paranoid` | Security-first logs | Uses deeper decoding and stronger marker matching; trades more false positives for fewer leaks. |
| `lockdown` | Allowlist-style safety | Unknown string values are redacted unless allowed. Use when low-entropy secrets may appear without markers. |

Examples:

```python
from filter_json_log import filter_json_log

payload = {"message": "password LEAK"}

print(filter_json_log(payload, mode="light", one_line=True))
# {"message":"password LEAK"}

print(filter_json_log(payload, mode="normal", one_line=True))
# {"message":"password [REDACTED]"}
```

Use `mode="lockdown"` when unknown strings must not leave the process unless explicitly allowed:

```python
payload = {"status": "ok", "note": "blue"}

print(filter_json_log(
    payload,
    mode="lockdown",
    allow_keys={"status"},
    one_line=True,
))
# {"status":"ok","note":"[REDACTED]"}
```

### Output Length and Newline Limits

By default, output is capped at:

```python
max_len = 4096
max_nls = 8
```

If a value or container cannot be rendered safely within the budget, the visible excess is replaced with `"[...]"`.

```python
from filter_json_log import filter_json_log

payload = {"items": list(range(1000))}
print(filter_json_log(payload, max_len=80, one_line=True))
```

Output shape:

```json
{"items":[0,1,2,3,4,5,6,7,8,9,"[...]"]}
```

The exact cut point is budget-dependent. The output remains valid JSON whenever possible.

If `max_len` is too small, it is normalized to the minimum length needed to render the truncation marker JSON string, currently `"[...]"`.

Disable output caps completely with:

```python
filter_json_log(payload, forced_infinite=True)
```

Only use `forced_infinite=True` when you know the payload size is safe for your log system.

### URL Fields

URL-like keys have priority over secret-like keys. The value is preserved as a URL but sensitive parts are redacted.

```python
payload = {
    "signed_url": "https://user:pass@example.com/file?X-Amz-Signature=abc",
}

print(filter_json_log(payload, one_line=True))
```

Output:

```json
{"signed_url":"https://[REDACTED]@example.com/file?X-Amz-Signature=[REDACTED]"}
```

URL sanitization also checks fragments, nested URL/query values, path segments, path params, protocol-relative URLs and schemeless userinfo-like forms in URL contexts.

### Structured Strings: Headers, Query, Form, Env, Cookies and Commands

When a key implies a structured text context, the value is parsed with that context and then routed through the shared sanitizer.

```python
payload = {
    "query": "access_token=abc&next=https%3A%2F%2Fx.test%2F%3Ftoken%3Ddef",
    "headers": "Authorization: Bearer secret\r\n X-Trace: visible",
    "env": "PASSWORD=secret\nDEBUG=true",
    "cmd": "curl -u user:pass https://example.test",
}

print(filter_json_log(payload, one_line=True))
```

Output shape:

```json
{"query":"access_token=[REDACTED]&next=https%3A%2F%2Fx.test%2F%3Ftoken%3D%5BREDACTED%5D","headers":"Authorization: [REDACTED]\r\n X-Trace: visible","env":"PASSWORD=[REDACTED]\nDEBUG=true","cmd":"curl -u [REDACTED] https://example.test"}
```

### Pair Objects and Pair Arrays

The sanitizer recognizes common key/value shapes used by headers, params, env vars and config dumps.

```python
payload = {
    "headers": [
        {"name": "Authorization", "value": "Bearer abc"},
        ["X-Api-Key", "def"],
    ],
    "env": [
        ["PASSWORD", "secret", "comment"],
    ],
}

print(filter_json_log(payload, one_line=True))
```

Output shape:

```json
{"headers":[{"name":"Authorization","value":"[REDACTED]"},["X-Api-Key","[REDACTED]"]],"env":[["PASSWORD","[REDACTED]","[REDACTED]"]]}
```

### Language and Domain Marker Packs

Built-in marker packs include:

```python
TECHNICAL_EN_MARKER_PACK
FRENCH_MARKER_PACK
GERMAN_MARKER_PACK
UKRAINIAN_MARKER_PACK
RUSSIAN_MARKER_PACK
```

You can provide your own `MarkerPack` for another language or an internal domain vocabulary.

```python
from filter_json_log import MarkerPack, filter_json_log

COMPANY_MARKER_PACK = MarkerPack(
    name="company",
    secret_markers=frozenset({"tenant secret", "internal token"}),
    url_markers=frozenset({"callback endpoint"}),
)

payload = {"message": "tenant secret abc"}

print(filter_json_log(
    payload,
    extra_marker_packs=[COMPANY_MARKER_PACK],
    one_line=True,
))
```

Output:

```json
{"message":"tenant secret [REDACTED]"}
```

### Custom Secret and URL Rules

You can replace or extend the default markers and regexes.

```python
from filter_json_log import FilterJSONLog

json_filter = FilterJSONLog(
    extra_bad_keys={"tenant_secret"},
    extra_url_keys={"callback_endpoint"},
    extra_strong_embedded_secret_re=[
        r"(?<![A-Za-z0-9])corp_[A-Za-z0-9]{24,}(?![A-Za-z0-9])",
    ],
)
```

The `extra_*` parameters extend defaults. The non-extra parameters, such as `bad_keys=...`, replace the corresponding defaults.

### Integration with Python's `logging` Module

`JSONLogFilter` is a `logging.Filter` subclass. It can sanitize:

- selected `LogRecord` attributes, usually supplied through `extra={...}`;
- mapping-style logging args, for example `logger.info("password=%(password)s", {...})`;
- positional args with simple key inference, for example `logger.info("password=%s", secret)`;
- already-rendered message text, including f-string and `.format()` messages.

```python
import logging
import sys

from filter_json_log import JSONLogFilter

logger = logging.getLogger("my_app")
logger.setLevel(logging.INFO)
logger.handlers.clear()

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
handler.addFilter(JSONLogFilter(
    mode="normal",
    one_line=True,
    attrs=("payload", "url", "headers"),
))
logger.addHandler(handler)

logger.info(
    "request failed",
    extra={
        "payload": {
            "password": "secret",
            "url": "https://x.test/?token=abc",
        }
    },
)

logger.info("password=%s", "secret")
logger.info("Authorization: Bearer secret")
```

Output:

```text
INFO: request failed | JSON={"password":"[REDACTED]","url":"https://x.test/?token=[REDACTED]"}
INFO: password=[REDACTED]
INFO: Authorization: [REDACTED]
```

#### Logging Filter Options

```python
JSONLogFilter(
    attrs=("json", "payload", "data", "body", "request", "response", "url"),
    fmt=" | JSON={filtered_json}",
    raw_str=False,
    fallback=True,
    append_from_fallback=False,
    replace_attrs=True,
    sanitize_msg=True,
    infer_arg_keys_from_msg=True,
    mode="normal",
)
```

Notes:

- `attrs` controls which `LogRecord` attributes are inspected.
- `fmt` controls the suffix appended when a selected JSON payload is found.
- `sanitize_msg=True` sanitizes already-rendered messages. Keep it enabled unless another logging layer already guarantees safety.
- `infer_arg_keys_from_msg=True` handles common forms like `password=%s` and `Authorization: %s`.

## Corner Cases & Considerations

### This Library Produces Log Text, Not Reusable Payloads

The output is intended for logs, monitoring and debugging. It is valid JSON text in normal operation, but it is not intended to be deserialized and used as a business payload. Redaction markers such as `"[REDACTED]"` and `"[...]"` are deliberately human-readable.

### Redaction vs Truncation

`"[REDACTED]"` means the library intentionally hid sensitive data.

`"[...]"` means the output was truncated because of `max_len` or `max_nls`.

These markers have different meanings and should not be collapsed into one value.

### False Positives Are Mode-Dependent

`light` prioritizes readability. `normal` is the recommended default. `paranoid` intentionally prefers some false positives over leaks. `lockdown` redacts unknown strings unless explicitly allowed.

### No Sanitizer Can Guess a Secret with No Signal

If a password is the low-entropy string `"blue"`, appears under a neutral key such as `"note"`, and has no marker, URL/query/header/form/cmd context, token shape, encoding clue or entropy signal, no sanitizer can reliably identify it. Use `mode="lockdown"` with `allow_keys` for that policy.

### Performance

For high-volume logging, instantiate `FilterJSONLog` once and reuse it. The standalone `filter_json_log()` function is convenient, but it rebuilds configuration on every call.

### Logging Filter Precedence

Passing structured data through `extra={"payload": ...}` is preferred. Fallback scanning of args and rendered messages is useful, but it costs more CPU and can produce more false positives in aggressive modes.

## API Reference

### `filter_json_log(data, raw_str=False, **kwargs)`

Standalone function for one-off JSON log sanitization.

Common parameters passed through `**kwargs`:

- `mode`: `"light"`, `"normal"`, `"paranoid"`, or `"lockdown"`. Default: `"normal"`.
- `paranoid`: backward-compatible alias. `paranoid=True` means `mode="paranoid"`.
- `max_len`: maximum output length. Default: `4096`.
- `max_nls`: maximum number of newline characters. Default: `8`.
- `one_line`: render compact JSON without indentation. Default: `False`.
- `forced_infinite`: disable length and newline truncation. Default: `False`.
- `redacted`: marker for hidden secrets. Default: `"[REDACTED]"`.
- `truncated`: marker for truncated output. Default: `"[...]"`.
- `ensure_ascii`: passed to JSON string rendering. Default: `False`.
- `sort_keys`: render mapping keys sorted by JSON key text. Default: `False`.
- `raw_str`: when `True`, `data` must be a JSON string and will be parsed with `json.loads()`.

### `FilterJSONLog(...)`

Reusable class that stores compiled sanitizer configuration.

Important constructor parameters:

- `mode`, `paranoid`, `max_len`, `max_nls`, `one_line`, `forced_infinite`, `redacted`, `truncated`, `indent`, `ensure_ascii`, `sort_keys`.
- `marker_packs`: replace default marker packs.
- `extra_marker_packs`: add language/domain marker packs to defaults.
- `bad_keys`, `extra_bad_keys`: replace or extend secret markers.
- `url_keys`, `extra_url_keys`: replace or extend URL markers.
- `bad_keys_re`, `extra_bad_keys_re`: replace or extend secret-key regex rules.
- `url_keys_re`, `extra_url_keys_re`: replace or extend URL-key regex rules.
- `strong_whole_secret_re`, `extra_strong_whole_secret_re`: replace or extend whole-value provider-token regexes.
- `strong_embedded_secret_re`, `extra_strong_embedded_secret_re`: replace or extend provider-token regexes found inside strings.
- `filter_url_bad_keys`, `extra_filter_url_bad_keys`, `filter_url_bad_keys_re`, `extra_filter_url_bad_keys_re`, `filter_url_bad_path_re`: URL sanitizer configuration.
- `paranoid_entropy`, `entropy_min_length`, `entropy_threshold`: optional entropy-based redaction controls.
- `allow_keys`: keys that may keep unknown strings in `lockdown` mode.

Useful methods:

```python
json_filter.filter(data, raw_str=False) -> str
json_filter.filter_text(text, kind="text", key=None) -> str
json_filter.filter_value_for_key(value, key) -> str
```

### `MarkerPack(...)`

Language/domain pack for semantic markers.

```python
MarkerPack(
    name="custom",
    secret_markers=frozenset({"password", "token"}),
    url_markers=frozenset({"url", "callback"}),
    query_markers=frozenset({"query", "params"}),
    header_markers=frozenset({"headers"}),
    form_markers=frozenset({"form", "body"}),
    env_markers=frozenset({"env", "config"}),
    cookie_markers=frozenset({"cookies"}),
    command_markers=frozenset({"cmd", "argv"}),
)
```

### `JSONLogFilter(...)`

`logging.Filter` subclass.

Constructor parameters:

- `attrs`: `LogRecord` attributes to inspect. Default includes `json`, `payload`, `data`, `body`, `request`, `response`, `params`, `query`, `headers`, `env`, `cmd`, `argv`, `url`.
- `fmt`: suffix appended when selected JSON is found. Default: `" | JSON={filtered_json}"`.
- `json_filter_instance`: optional preconfigured `FilterJSONLog` instance.
- `raw_str`: treat selected string values as raw JSON documents.
- `fallback`: inspect logging args. Default: `True`.
- `append_from_fallback`: append fallback-filtered JSON using `fmt`. Default: `False`.
- `replace_attrs`: replace selected attributes with safe JSON strings. Default: `True`.
- `sanitize_msg`: sanitize rendered log message. Default: `True`.
- `infer_arg_keys_from_msg`: infer simple key contexts for positional args. Default: `True`.
- `**filter_kwargs`: passed to `FilterJSONLog` if `json_filter_instance` is not supplied.

## License

This project is licensed under the MIT License.
