mu|LLM — Security

Secrets & API Keys

API keys live in .env with file-level advisory locking and restricted permissions. The server acquires fcntl.LOCK_SH on every read and LOCK_EX on every write — preventing torn reads during live key rotation. Permissions are hardened to chmod 440 (owner + group read-only) on first write and on each server startup.

File Locking

Uses fcntl.flock() with LOCK_SH for reads and LOCK_EX for writes. Concurrent processes cannot read a partially-written key file.

Atomic Write

Updates are staged to .env.tmp first, then os.rename()'d atomically. A failed write can never corrupt the live key store.

chmod 440

On startup and after every key write, permissions are enforced to 440. Owner and group may read; world access is unconditionally removed.

Never Logged

Key values never appear in stdout, structlog, Redis, the scoring log, or any HTTP response. Only key names appear in logs — e.g. ANTHROPIC_API_KEY: set.

shell Harden .env permissions manually

# Set owner+group read-only, remove world access
chmod 440 .env

# Verify — expect: -r--r----- 1 peter peter
ls -la .env

python Atomic write pattern used internally

import fcntl, os, pathlib

def write_env_atomic(path, content):
    tmp = path.with_suffix('.tmp')
    with open(tmp, 'w') as f:
        fcntl.flock(f, fcntl.LOCK_EX)
        f.write(content)
        f.flush(); os.fsync(f.fileno())
    os.chmod(tmp, 0o440)
    os.rename(tmp, path)  # atomic on POSIX

Sensitive Data Mode

Set MULLM_SENSITIVE=1 to activate an additional data-protection layer. The router strips, refuses to cache, or replaces with a redaction token any content that could contain PII, credentials, or user-defined sensitive patterns.

Vector cache writes blocked No query text is written to the semantic cache. Semantic similarity matching is disabled for the session. Cache reads still work for non-sensitive queries stored in prior sessions.
Scoring log content stripped The content_preview field (first 80 chars stored for routing diagnostics) is replaced with [SENSITIVE REDACTED]. Tier, model, cost, and latency are still logged.
Redis trace suppressed Query content is omitted from the in-memory routing trace. Only intent ID and tier are stored — no content preview of any length in Redis.
Dashboard obfuscation The /dashboard query trace panel shows only block characters in place of content previews. No text fragments are visible to other sessions sharing the LAN.

shell Enable sensitive data mode

# Enable for a single server run
MULLM_SENSITIVE=1 python -m router.main --https

# Or persist in .env
echo 'MULLM_SENSITIVE=1' >> .env

shell Verify the mode is active

curl -sk https://127.0.0.1:6856/api/system \
  | python3 -m json.tool \
  | grep sensitive
# "sensitive_mode": true

Mutual TLS (mTLS)

Standard HTTPS authenticates the server to the client. With MULLM_MTLS=1 the client must also present a certificate signed by a CA you control — set via MULLM_CLIENT_CA. Connections without a valid client cert are rejected at the TLS handshake, before any HTTP is parsed.

How it works

uvicorn's TLS layer is configured with ssl_ca_certs and ssl_cert_reqs=CERT_REQUIRED. The OS TLS stack rejects uncertified clients before the first HTTP byte is read.

Client CA file

Point MULLM_CLIENT_CA at a PEM-encoded CA certificate or chain. Issue per-device leaf certs from this CA using openssl req or step ca certificate.

Cert identity logging

The CN field of the client cert is extracted and attached to every structlog event as client_cn — giving per-device audit trails without passwords.

Hot reload

Sending SIGHUP to the server reloads the CA bundle without dropping live connections. New certs issued from the CA are accepted within one reload cycle, no restart required.

shell Generate a CA + client cert with step-ca

# 1. Create a local CA
step ca init --name=mullm-ca --dns=localhost \
  --address=:8443 --provisioner=peter

# 2. Issue a client cert for this machine
step ca certificate peter@mullm client.crt client.key

# 3. Start with mTLS enforced
MULLM_MTLS=1 MULLM_CLIENT_CA=/path/to/ca.crt \
  python -m router.main --https

shell Test with and without client cert

# With valid client cert — expect 200
curl -sk --cert client.crt --key client.key \
  https://127.0.0.1:6856/api/system

# Without cert — rejected at TLS layer (curl exits 35)
curl -sk https://127.0.0.1:6856/api/system

env Environment variables

MULLM_MTLS=1
MULLM_CLIENT_CA=/home/peter/mullm/certs/ca.crt

Immutable Audit Log

Set AUDIT_LOG_ENABLED=1 to activate append-only routing audit logging. The file descriptor is opened with O_APPEND and never truncated. Monthly rotation gzips the previous month's log and chmod 440s it — old entries are never deleted, only archived.

Append-only writes

Opened with open(path, 'a') — no seek, no truncate. Each JSON line is flushed immediately. Even a kill -9 mid-write produces a valid partial line; prior entries are never corrupted.

Monthly gzip rotation

On the first request after month rollover, the previous month's file is renamed to audit_YYYY_MM.jsonl.gz and compressed in a background thread. The current file starts fresh.

What is recorded

Timestamp (ISO-8601 UTC), intent ID, category, complexity score, tier, model, token counts, cost USD, latency ms, and client_cn when mTLS is active. Query content is never recorded.

Archive permissions

Rotated .gz archives are immediately chmod 440'd — read-only for owner and group. The active log is 640 to permit appends.

env Enable audit logging

AUDIT_LOG_ENABLED=1
AUDIT_LOG_DIR=/home/peter/mullm/logs/audit  # optional; defaults to logs/

jsonl Example audit log entry

{
  "ts": "2026-04-19T14:23:01.042Z",
  "intent_id": "q_8f3a",
  "category": "code",
  "complexity": 0.72,
  "tier": "local",
  "model": "qwen3:30b",
  "input_tokens": 312,
  "output_tokens": 489,
  "cost_usd": 0.00,
  "latency_ms": 1842,
  "client_cn": null  // populated when mTLS is active
}

shell Inspect the audit log

# Count entries by tier
jq -r '.tier' logs/audit/audit_current.jsonl | sort | uniq -c

# Total cost this month (USD)
jq '[.cost_usd] | add' logs/audit/audit_current.jsonl

# List rotated archives
ls -lh logs/audit/*.jsonl.gz

Data Retention

muLLM stores two kinds of operational data: the scoring log (routing metadata) and the semantic cache (SQLite by default, optional ChromaDB). Both obey configurable retention or purge controls. Export and purge controls are on the Privacy page.

Env Var	Default	Effect
DATA_RETENTION_DAYS	90	Scoring log entries older than this are purged on next server start. Set to `0` to disable auto-purge.
CHROMA_RETENTION_DAYS	90	Optional ChromaDB documents older than this are deleted from the local vector store. The default SQLite semantic cache is controlled through the cache purge/export APIs.
AUDIT_RETENTION_MONTHS	24	Gzipped audit archives older than this many months are deleted. The active log is never auto-deleted.
MULLM_SENSITIVE	0	When `1`, no query content is written anywhere. Retention settings still apply to metadata.

env Tight retention example (30 days)

DATA_RETENTION_DAYS=30
CHROMA_RETENTION_DAYS=30
AUDIT_RETENTION_MONTHS=12

For GDPR right-to-deletion and data export (right-to-portability), see /privacy

Dependency Scanning

The dependency tree is scanned weekly with Grype — Anchore's open-source vulnerability scanner backed by NVD + GitHub Advisory Database. scripts/grype_scan.sh fails the build on any high or critical CVE and produces a CycloneDX SBOM.

Grype scanner

Scans pip-frozen requirements against Grype's daily-updated NVD + GHSA database. No external API calls after the local DB is synced — works offline if the DB is under 24 hours old.

Weekly scheduled

GitHub Actions gate

The weekly CI job runs scripts/grype_scan.sh and uploads a SARIF report to the GitHub Security tab. PRs that introduce a new high or critical CVE are blocked from merging.

CI gated

SBOM generation

The scan also generates a CycloneDX JSON SBOM at reports/sbom.json. This can be attached to release artifacts for supply-chain auditability and vendor attestations.

CycloneDX

Local scan

Run scripts/grype_scan.sh at any time for an immediate report. Uses the cached local Grype DB — no outbound requests unless the DB is stale (older than 24 hours).

Offline-capable

shell scripts/grype_scan.sh

#!/usr/bin/env bash
set -euo pipefail

# Update Grype vulnerability database
grype db update

# Scan project — exit non-zero on high/critical CVE
grype dir:. --fail-on high --output sarif > reports/grype.sarif

# Generate CycloneDX SBOM
syft dir:. -o cyclonedx-json > reports/sbom.json

echo 'Scan complete. See reports/grype.sarif'

yaml .github/workflows/security.yml (weekly scan)

name: Security Scan
on:
  schedule:
    - cron: '0 6 * * 1'  # Monday 06:00 UTC
  push:
    branches: [main]
jobs:
  grype:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: anchore/scan-action@v3
        with:
          path: '.'
          fail-build: 'true'
          severity-cutoff: high

Network Security

HTTPS is optional and off by default; HTTP is the default for local use. Enable with ssl_certfile + ssl_keyfile, or set auto_generate_cert=true for a self-signed certificate. A /.well-known/security.txt endpoint follows RFC 9116 for responsible disclosure. Security contact: security@mullm.com

HTTPS only

Started with --https, uvicorn binds TLS-only. If cert files are absent, a self-signed cert is auto-generated for localhost and the detected LAN IP. No plaintext fallback is created.

TLS 1.3 (ECDHE-384)

security.txt

The /.well-known/security.txt endpoint serves a standard RFC 9116 disclosure file. Researchers can report vulnerabilities to security@mullm.com using PGP if preferred.

RFC 9116

LAN binding

Binds 0.0.0.0:6856 by default for LAN access. Set MULLM_BIND=127.0.0.1 for single-machine loopback-only mode. The auto-generated cert SANs cover both localhost and the detected LAN IP.

LAN-scope by default

CORS policy

CORS origins are restricted to https://localhost:* and https://10.0.0.* (or your configured LAN subnet). Wildcard CORS is never set on authenticated endpoints.

No wildcard CORS

shell Start muLLM (HTTPS required)

# LAN-accessible (default, auto-generates cert)
python -m router.main --https

# Loopback-only (single machine)
MULLM_BIND=127.0.0.1 python -m router.main --https

text /.well-known/security.txt

Contact: mailto:security@mullm.com
Expires: 2027-01-01T00:00:00.000Z
Preferred-Languages: en
Canonical: https://github.com/mybrainrunslinux/mullm/security/policy
Policy: https://github.com/mybrainrunslinux/mullm/security/policy

shell Verify the security.txt endpoint

curl -sk https://127.0.0.1:6856/.well-known/security.txt