Metadata-Version: 2.4
Name: sether
Version: 0.4.0
Summary: Streaming PII redaction for AI applications. The hiding place for sensitive data flowing into LLMs. Secrets pack, SSE-aware streaming, audit events, drop-in middlewares for httpx / ASGI / WSGI / OpenAI / Anthropic.
Author-email: Godfrey Lebo <emorylebo@gmail.com>
License: MIT
Project-URL: Homepage, https://sether.raevenmarket.com.ng
Project-URL: Documentation, https://sether.raevenmarket.com.ng/docs/python
Project-URL: Sandbox, https://sether.raevenmarket.com.ng/sandbox
Project-URL: Repository, https://github.com/emorilebo/sether-py
Project-URL: Issues, https://github.com/emorilebo/sether-py/issues
Project-URL: Changelog, https://github.com/emorilebo/sether-py/blob/main/CHANGELOG.md
Keywords: ai,security,pii,redaction,stream,sse,llm,privacy,gdpr,hipaa,compliance,openai,anthropic,secrets,guardrails,agentic,mcp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: phonenumbers>=8.13
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.39; extra == "anthropic"
Provides-Extra: httpx
Requires-Dist: httpx>=0.27; extra == "httpx"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: anthropic>=0.39; extra == "all"
Requires-Dist: httpx>=0.27; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: phonenumbers>=8.13; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Dynamic: license-file

# Sether (Python)

> **Hide personal data from your AI before it reaches any LLM provider.**
>
> Named for the Hebrew *sether* (סֵתֶר), *the hiding place*. Psalm 32:7.

![Sether redacts PII locally before it reaches the LLM: the input "Hi, this is Amara Okafor — my account email is amara.okafor@acme.com … card 4242 4242 4242 4242 … call me at +1 (415) 555-2671 … SSN 123-45-6789" becomes "Hi, this is &lt;NAME&gt; — my account email is &lt;EMAIL&gt; … card &lt;CC&gt; … &lt;PHONE&gt; … &lt;SSN&gt;". The token vault stays in your infrastructure and restore() swaps the originals back into the reply.](https://raw.githubusercontent.com/raeven-co/sether/main/assets/sether-redaction.png)

*Your app → **Sether** redacts locally → LLM. The token vault stays in your infrastructure; `restore_sync()` swaps the originals back into the reply.*

Sether is a streaming PII-redaction layer that sits between your application
and any LLM API. It detects sensitive data (email, phone, SSN, credit card,
IBAN, IP addresses, secrets, and labelled identity fields), swaps each match
for a stable token before the request leaves your boundary, then restores the
original values transparently in the response.

This is the Python port of [`@raeven-co/sether`](https://www.npmjs.com/package/@raeven-co/sether).
Same detection engine, same token format, same chunk-boundary streaming safety,
ported faithfully to Python with both synchronous and asynchronous streaming and
drop-in integrations for **httpx, ASGI (FastAPI / Starlette), WSGI (Flask),
the OpenAI SDK, and the Anthropic SDK**.

A product of **Raeven Company LTD**.

---

## Why this exists

If your application sends a customer's email, phone number, or any other PII to
an LLM provider, that is a sub-processor disclosure under GDPR Article 28.
Credit-card data pulls you into PCI DSS scope. Health identifiers trigger HIPAA.
Sether stops the leak at the boundary: sensitive substrings become stable tokens
before the bytes leave your process, and `restore()` swaps them back so your
application code does not need to branch on redacted text.

**This package does not phone home.** Streams are not sent to Raeven. The vault
stays in your process (or your own backing store if you implement `Vault`).

---

## Install

```bash
pip install sether
```

Requires Python 3.9+. The phone detector uses [`phonenumbers`](https://pypi.org/project/phonenumbers/)
(installed automatically). Integration extras are optional:

```bash
pip install "sether[openai]"      # wrap_openai
pip install "sether[anthropic]"   # wrap_anthropic
pip install "sether[httpx]"       # wrap_httpx
pip install "sether[all]"         # all of the above
```

The ASGI and WSGI middlewares have no extra dependency.

---

## 60-second quickstart

```python
from sether import Sether

sether = Sether()

# Outgoing: redact before sending to the LLM.
safe = sether.redact_sync("my email is alice@example.com")
# -> "my email is <EMAIL_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx>"

# Incoming: restore before showing the user.
back = sether.restore_sync(safe)
# -> "my email is alice@example.com"
```

The same `Sether` instance shares its vault between redaction and restoration,
which is how the round-trip identity is preserved.

> **Security: never call `restore_sync()` on serialized JSON** (tool-call
> arguments, JSON-mode replies, JSON response bodies). Restore is plain text
> substitution: an original value containing `"` or `\` can add keys to the
> document or break it. Use `restore_json` / `restore_json_value`, see
> [Never restore serialized JSON as text](#never-restore-serialized-json-as-text).

### Streaming (sync and async)

```python
sether = Sether()

# Synchronous: any iterable of text chunks.
def chunks():
    yield "Contact alice@"
    yield "example.com or call "
    yield "+1 415 555 2671."

redacted = "".join(sether.redact_stream(chunks()))
restored = "".join(sether.restore_stream([redacted]))

# Asynchronous: any async iterable (e.g. an LLM token stream).
async def run(llm_stream):
    async for piece in sether.aredact_stream(llm_stream):
        await forward_to_model(piece)
```

The redact stream holds back `safe_distance_bytes` (default 256) at the tail of
each chunk so a PII pattern crossing a chunk boundary is still detected when the
next chunk arrives. A long whitespace-free value (a JWT, an API key) is held
back up to `max(safe_distance_bytes * 4, 8192)` bytes so it is never emitted
partially across a boundary. For values larger than that bound, raise
`safe_distance_bytes` or use `redact_sync` on complete payloads. This round-trip
identity is verified by property-based tests over 60 random chunk partitions.

---

## Never restore serialized JSON as text

`restore_sync`, the restore streams and `sether.restore_sync()` are **plain
text substitution**: each original value is inserted exactly as it was,
unescaped. If the text is serialized JSON, such as LLM tool-call arguments, a
JSON-mode reply or a JSON response body, an original containing `"` or `\`
stops being data inside a string and becomes part of the document:

```python
# Tool-call arguments from the model reference a token...
args = '{"to": "<EMAIL_...>", "subject": "Hello <NAME_...>"}'
# ...whose original is:  Alice", "bcc": "attacker@example.com
json.loads(restore_sync(args, vault))
# -> {'to': 'bob@example.com', 'subject': 'Hello Alice', 'bcc': 'attacker@example.com'}
```

Built-in detectors can capture a trailing backslash (`password: hunter2\`),
which breaks the document the same way; custom detectors and vaults filled from
other systems can carry any character. Restore JSON value by value instead:

```python
from sether import restore_json, restore_json_value

# Serialized JSON in, serialized JSON out: each string value is restored and re-escaped.
safe_args = restore_json(tool_call.function.arguments, vault, strict=True)
# or, on the facade: sether.restore_json(tool_call.function.arguments, strict=True)

# Already parsed, e.g. an Anthropic tool_use block's `input`.
tool_input = restore_json_value(block.input, vault, strict=True)
```

- Object keys are never restored. Invalid JSON raises `json.JSONDecodeError`;
  there is no fallback to text replacement.
- One level at a time: a string value that itself holds serialized JSON is
  restored as plain text, so parse and restore it separately.
- `restore_json` re-serializes compactly; input formatting is not kept.
- `strict=True` (also accepted by `restore_sync`) raises `UnresolvedTokenError`
  when anything token-shaped would be left in place: an invented or expired
  token, a case-altered copy such as `<email_...>`, or a token in a key. Its
  `tokens` attribute lists them; restored originals are never included.
  Without `strict`, those pass through silently.
- Strict mode is not authorization. A model can still put a real token in the
  wrong field (the customer's address in `bcc`), so validate tool arguments
  before acting on them.
- `TOKEN_RE` is exported if you need to scan for tokens yourself.

**Sether's own restore paths restore JSON value by value.** They change only
the string values that hold a token and re-encode just those, so every other
byte is kept (formatting, key order, escapes and numbers). Object keys are
never restored.

- `sse_restore_iter` / `create_sse_restore_stream`: a `data:` payload that
  parses as JSON (OpenAI and Anthropic chunks) is restored value by value, so
  an original containing `"`, `\` or a line break can't change the JSON or
  break the SSE framing. Other payloads (`[DONE]`, plain text) are restored as
  text, and a line break in an original continues it on a new `data: ` line
  (the client receives it as `\n`). Each `data:` line is handled on its own, so
  a JSON payload split across several `data:` lines is restored as text.
- `wrap_httpx`, `SetherASGIMiddleware` and `SetherWSGIMiddleware`: an
  `event-stream` body is restored per `data:` payload as above. A body that
  parses as JSON is restored value by value, a JSON-typed body that doesn't
  (NDJSON, JSON Lines) line by line, and any other text body as text.
- `wrap_openai` / `wrap_anthropic`: when the request asks for JSON output
  (`response_format` `json_object` / `json_schema`, or Anthropic
  `output_config.format` / `output_format` with `json_schema`), the reply's
  content is restored value by value, and left tokenized if it isn't valid
  JSON. Streamed JSON-mode deltas are fragments of a document, so they stay
  tokenized: join them and call `restore_json`. Tool-call arguments stay
  tokenized by default. Pass `restore_tool_arguments=True` to restore
  non-streaming `tool_calls[].function.arguments` and `tool_use.input` value by
  value, and validate them before acting on them.

What is still restored as text:

- **JSON-mode output streamed over SSE.** `sse_restore_iter` (and the
  middlewares on an event-stream body) keeps each chunk valid, but the
  `delta.content` fragments join into a second JSON document the stream can't
  see, so an original lands in that document unescaped. Don't SSE-restore
  JSON-mode streams: collect the tokenized fragments and call `restore_json` on
  the assembled text.
- `restore_sync`, the restore streams and `sether.restore_sync()`, by design.
- `AliasVault.restore()` swaps decoys back by plain substitution too. On JSON,
  apply it to each parsed string value, never to the serialized document.

---

## Detectors

By default Sether runs the **basic pack**. Pass an explicit list to narrow scope,
or add the opt-in packs.

```python
from sether import Sether, basic_detectors, secrets_detectors, identity_detectors

sether = Sether(detectors=[*basic_detectors, *secrets_detectors, *identity_detectors])
```

### Basic pack (`basic_detectors`)

| Detector | Type | Method |
| --- | --- | --- |
| `email_detector` | `EMAIL` | RFC-5321-style regex. ASCII-only. |
| `phone_detector` | `PHONE` | `phonenumbers` (international format). For national-format numbers use `create_phone_detector(default_country="US")`. |
| `credit_card_detector` | `CC` | Bounded regex + Luhn check. |
| `ssn_detector` | `SSN` | Regex + SSA invalid-prefix rules. |
| `ipv4_detector` | `IPV4` | Strict octet-bounded regex. |
| `ipv6_detector` | `IPV6` | Candidate regex + in-tree validator. |
| `iban_detector` | `IBAN` | Regex + mod-97 checksum. |

### Secrets pack (`secrets_detectors`)

`aws_access_key_detector`, `openai_key_detector`, `anthropic_key_detector`,
`github_pat_detector` (classic + fine-grained), `slack_token_detector`,
`stripe_key_detector`, `jwt_detector`, `high_entropy_detector`
(Shannon entropy >= 3.5 bits/char).

### Identity pack (`identity_detectors`, opt-in)

Label-anchored detection for names, dates of birth, passport numbers, and
addresses. A value is redacted only when it appears with the label that
introduces it (`Name:`, `DOB:`, `Passport No:`, `Address:`) or, for a few
distinctive standalone shapes (a street line with a house number, a UK
postcode), a structure strong enough to keep false positives low. Labels are
recognised across many languages (Latin-script plus CJK, Cyrillic, Arabic), and
value capture is Unicode-aware.

Free-text NER (unlabelled names, organisations, locations in running prose) is
not covered here; that is the separate `sether-ner` roadmap item.

### Custom detectors

Anything with a `type` string and a `detect(text)` method works:

```python
import re
from sether import DetectorMatch

class OrderIdDetector:
    type = "ORDER_ID"
    _re = re.compile(r"\bORD-\d{8}\b")

    def detect(self, text):
        return [DetectorMatch(m.start(), m.end(), m.group(0)) for m in self._re.finditer(text)]
```

---

## Token vault

Tokens map back to originals through a vault. Sether ships an in-memory LRU
vault (10,000 entries, 1-hour TTL by default). Implement the `Vault` protocol to
change eviction, encrypt at rest, or namespace tokens per tenant.

```python
from sether import Vault   # a runtime-checkable Protocol: set/get/has/delete/size/clear

class NamespacedVault:
    def __init__(self, prefix):
        self._store = {}
        self._prefix = prefix
    def set(self, token, value): self._store[self._prefix + token] = value
    def get(self, token): return self._store.get(self._prefix + token)
    def has(self, token): return (self._prefix + token) in self._store
    def delete(self, token): return self._store.pop(self._prefix + token, None) is not None
    def size(self): return len(self._store)
    def clear(self): self._store.clear()

sether = Sether(vault=NamespacedVault("tenant-42:"))
```

The `Vault` interface is **synchronous**: restore substitutes tokens as bytes
flow through and cannot `await` a lookup per token. Front an async store (Redis,
Postgres) with a synchronous in-process cache you hydrate before the restore
pass, or keep the vault in-process.

---

## SSE / JSON-stream mode

OpenAI and Anthropic streaming responses come back as Server-Sent Events. The
SSE-aware stream redacts payloads inside `data:` lines while preserving the
`data:` / `event:` / `id:` / `retry:` framing and blank-line separators
verbatim.

```python
from sether import create_sse_redact_stream, basic_detectors, MemoryVault

vault = MemoryVault()
stream = create_sse_redact_stream(basic_detectors, vault)
out = stream.feed(sse_chunk) + stream.finish()
# or the iterator helpers: sse_redact_iter(chunks, detectors, vault)
```

---

## Drop-in integrations

```python
from sether import Sether
sether = Sether()
```

### httpx

```python
import httpx
from sether import wrap_httpx

client = wrap_httpx(httpx.Client(), detectors=sether.detectors, vault=sether.vault)
r = client.post("https://api.example.com/v1/chat",
                json={"q": "email alice@example.com"})
# The request carried <EMAIL_...>; r.text has any tokens the server echoed restored.
```

Works on `httpx.Client` and `httpx.AsyncClient`. Binary bodies pass through
untouched.

### ASGI (FastAPI / Starlette)

```python
from fastapi import FastAPI
from sether import SetherASGIMiddleware

app = FastAPI()
app.add_middleware(SetherASGIMiddleware, detectors=sether.detectors, vault=sether.vault)
```

### WSGI (Flask)

```python
from flask import Flask
from sether import SetherWSGIMiddleware

app = Flask(__name__)
app.wsgi_app = SetherWSGIMiddleware(app.wsgi_app, detectors=sether.detectors, vault=sether.vault)
```

### OpenAI SDK

```python
from openai import OpenAI
from sether import wrap_openai

client = wrap_openai(OpenAI(), detectors=sether.detectors, vault=sether.vault)
# Redacts messages out, restores choices back. Sync, async, and streaming clients.
```

### Anthropic SDK

```python
import anthropic
from sether import wrap_anthropic

client = wrap_anthropic(anthropic.Anthropic(), detectors=sether.detectors, vault=sether.vault)
# Redacts messages/system out, restores content blocks back.
```

The SDK wrappers are **structurally typed**. Sether never imports `openai` or
`anthropic`; any object matching the `chat.completions.create` /
`messages.create` shape works.

---

## Audit events

Each redaction can be described by a structured `AuditEvent` that maps to the
regulation it satisfies (GDPR Art. 28, SOC 2 CC6.7, HIPAA, PCI DSS, and more,
see `DEFAULT_REGULATION_MAPPINGS`). **The original value is never carried in an
event, only its length.** The JSON wire shape matches the TypeScript package
(camelCase keys) so events are interchangeable across both.

```python
from sether import AuditEvent, ConsoleAuditSink, MemoryAuditSink

sink = ConsoleAuditSink()   # JSONL to stderr; MemoryAuditSink accumulates for tests
sink.write(AuditEvent(timestamp="...", detector="EMAIL", value_length=17, token="<EMAIL_x>"))
```

---

## Honest limitations

These match the TypeScript package's documented limits:

- **Email detection is ASCII-only.** IDN/Unicode local parts do not match.
- **IPv6 `::1` (loopback) is not detected.** The candidate regex requires 4+
  chars. Loopback is not customer PII.
- **Credit-card regex is permissive**, then validated by Luhn. False positives
  in dense numeric content are possible.
- **Names / DOB / passport / address are label-anchored, not free-text NER.**
- **Very large whitespace-free values split across chunk boundaries** are held
  back only up to `max(safe_distance_bytes * 4, 8192)` bytes. Raise
  `safe_distance_bytes` or use `redact_sync` on complete payloads.
- **Restore is text substitution.** `restore_sync` and the restore streams
  insert originals unescaped. The SSE, httpx, ASGI, WSGI and SDK-wrapper paths
  restore JSON value by value, except JSON-mode output streamed over SSE. Use
  `restore_json` for your own JSON; see
  [Never restore serialized JSON as text](#never-restore-serialized-json-as-text).

---

## Parity with the TypeScript package

This port reproduces the audited TypeScript engine 1:1: the same detector
regexes (compiled with `re.ASCII` so `\b` / `\d` stay ASCII-only as in JS), the
same Luhn / mod-97 / SSA validation, the same overlap resolution (longest match
wins), the same `<TYPE_uuid>` token format, and the same safe-distance and
long-value streaming guards. 183 tests cover detectors, vault, streaming
(including a property-based chunk-partition round-trip), SSE, audit, and all
five integrations.

---

## License

MIT (c) Godfrey Lebo / Raeven Company LTD

## Reporting security issues

Email `emorylebo@gmail.com`. Do not file public issues for security findings.
