LockWatch Technical Deep-Dive
-----------------------------


Architecture Overview
---------------------

LockWatch consists of four independent components wired together by two middleware classes. Every
inbound request flows through LockWatchMiddleware (Starlette/FastAPI) or LockWatchFlaskMiddleware
(WSGI) in a fixed call order: rate limit check, then anomaly detection, then application dispatch,
then an async audit log write. The rate check is the only blocking gate; a 429 short-circuits the
remaining steps. All other components run fire-and-forget or in the background to keep them off
the response critical path.

RateLimiter lives in rate_limiter.py and holds a reference to an async Redis client plus a
RateLimitConfig dataclass (requests_per_window, window_seconds, key_func, key_prefix). The key_func
is a callable that receives the raw request object and returns a string identifier, allowing
callers to key by IP, authenticated user_id, or API key header. Three built-in key functions handle
both Starlette and Flask request objects through duck-typing checks on hasattr.

JWTRotationManager in jwt_rotation.py issues RS256 access tokens (15-minute TTL) and opaque-looking
refresh tokens (7-day TTL) that are actually signed JWTs. The raw refresh token is never stored;
only SHA-256(refresh_token) is persisted in Redis so a Redis dump yields no usable credentials.
Access token blacklisting stores the JTI in Redis with a TTL equal to the token's remaining
lifetime, so the blacklist never grows unbounded.

AnomalyDetector in anomaly.py maintains a separate Redis sorted set per IP using the same
sliding-window pattern as the rate limiter but over a shorter burst window (default 10 seconds).
When the count crosses burst_threshold and no dedup key is present, it first sets the dedup key in
Redis synchronously (with await and a 60-second TTL), then dispatches the webhook via
asyncio.create_task using a shared httpx.AsyncClient. Setting the dedup key before dispatching the
task ensures that even if the webhook delivery fails the key is already in Redis, preventing a
second alert from firing on the next request from the same IP.

AuditLogger in audit.py writes one row per request to a Postgres table called lockwatch_audit_log
using SQLAlchemy 2 async sessions backed by asyncpg. The table carries timestamp, ip, user_id,
endpoint, method, status_code, rate_limit_hit, anomaly_detected, and latency_ms. SQLAlchemy and
asyncpg are optional dependencies; if the audit extra is not installed, AuditLogEntry becomes a
stub class and any call to AuditLogger raises a clear ImportError. Two composite indices support
the most frequent query patterns: (timestamp, user_id) and (timestamp, ip).


Key Technical Decisions
-----------------------

Decision 1: Redis sorted sets for rate limiting, not counters.

Fixed-window rate limiters using Redis INCR have a thundering-herd problem: every client that made
N requests just before the window boundary can make N more requests immediately after it resets,
producing a burst of 2N requests in a short span. The sorted set approach eliminates this by
recording each request's millisecond timestamp as both the score and part of the member. ZADD adds
the new entry, ZREMRANGEBYSCORE prunes anything older than window_start_ms, ZCARD counts what
remains, ZRANGE with WITHSCORES retrieves the oldest timestamp in the window to compute when the
window will reset, and EXPIRE keeps cold keys from accumulating indefinitely. Because these run in
a single Redis pipeline, the check is atomic from the perspective of the application server. Member names include a UUID4 suffix to guarantee uniqueness within the same
millisecond, which would otherwise cause ZADD to overwrite a prior entry and undercount the window.
The tradeoff is storage: each request occupies one sorted set member rather than a single counter,
but in practice the window holds at most a few hundred entries per key.

Decision 2: RS256 instead of HMAC (HS256) for JWT signing.

RS256 allows a future JWKS endpoint where resource servers can verify tokens without sharing the
private key. With HS256, every service that needs to verify a token must hold the secret, creating
multiple leak surfaces. The cost is slightly higher CPU on signing and verification and the need to
generate and manage an RSA key pair. The private key is loaded lazily and cached on the manager
instance so the file I/O happens once, not per request.

Decision 3: Fail-open as default, configurable to fail-closed.

When Redis is unreachable, the default behavior (on_redis_failure="allow") is to log the error
and pass the request through rather than return 503. The reasoning is that most APIs prioritize
availability over perfect rate limit enforcement during an infrastructure incident. Operators who
need strict enforcement can set on_redis_failure="deny". This tradeoff is documented in both the
README and PLAN.md and tested explicitly in test_flask.py.


Hardest Part to Get Right
-------------------------

The Flask middleware is the most technically awkward component. WSGI is inherently synchronous, but
all of LockWatch's Redis operations are async. The current solution creates a new event loop per
request via asyncio.run() inside the WSGI __call__ method, instantiates a fresh async Redis client
inside that loop, runs the check, and closes the client in a finally block. This adds roughly 0.5ms
of overhead compared to the async FastAPI path (the PLAN.md acknowledges this). The approach is
correct but suboptimal: a dedicated background thread running a persistent event loop would be
faster, but would require thread-safe connection management. The current design favors simplicity
and correctness over micro-optimization.

Another subtle correctness issue: the rotate() method in JWTRotationManager is not fully atomic.
It reads the refresh token hash from Redis, checks the blacklist, deletes the hash, and then issues
a new pair in separate Redis commands. A race condition exists where two concurrent rotate() calls
with the same refresh token could both pass the hash check before either deletes it, resulting in
two valid new pairs being issued. The PLAN.md explicitly notes this and flags adding MULTI/EXEC as
a Phase 2 improvement. In the current implementation the delete happens before issue_token_pair so
the window is small, but it is not zero.


Data Structures and Algorithms
-------------------------------

Sliding window (rate limiter and anomaly detector): Redis sorted set keyed as
lockwatch:rl:{key_prefix}:{key} for rate limiting and lockwatch:anomaly:burst:{ip} for anomaly
detection. Members are "{now_ms}:{uuid4_hex}" scored by now_ms. Pipeline executes ZADD, ZREMRANGE-
BYSCORE (prune expired), ZCARD (count), ZRANGE with WITHSCORES (retrieves the oldest timestamp in
the window to compute when the window will reset, used to populate reset_at in response headers),
and EXPIRE.
Complexity is O(log N) per operation.

JWT blacklist: Redis string keyed as lockwatch:jwt:blacklist:{jti} with TTL set to remaining token
lifetime. Check is a single EXISTS call. Refresh token store is a Redis string keyed by SHA-256 of
the raw refresh token, value is the JTI, TTL is refresh_token_ttl_seconds.

Audit log indices: composite (timestamp, user_id) and (timestamp, ip) created in the Alembic
migration, matching the AuditQuery filter patterns in audit.py. The AuditQuery supports
endpoint_prefix filtering via a SQL LIKE "{prefix}%" query, which does not use an index and will
scan for wide result sets.


Known Limitations at Scale
---------------------------

The rate limiter is per-app-instance: two application servers sharing one Redis will correctly
enforce the limit because all state is in Redis, but Redlock is not used. If Redis is unavailable
and fail-open is set, both servers independently pass all requests during the outage.

The sorted set approach stores one Redis entry per request within the window. At very high request
rates (tens of thousands per second per key), the sorted set grows large and ZREMRANGEBYSCORE
becomes proportionally slower. Token bucket or leaky bucket algorithms using a single counter and
timestamp would be more memory-efficient at that scale.

The Flask middleware spawns a new asyncio event loop per request via asyncio.run(), which is
explicitly not safe to call from within an already-running async context. Deploying the Flask
middleware behind an async WSGI server that already manages a loop would produce a RuntimeError.

The audit log has no batching or buffering: every request fires one SQLAlchemy session and one
INSERT. Under sustained high traffic this will create significant Postgres write pressure. A write
buffer with periodic flush would be the correct production design.
