Metadata-Version: 2.4
Name: tigzig-concurrency
Version: 0.2.0
Summary: Bounded wait queue + client-IP forwarding for FastAPI backends
Project-URL: Homepage, https://github.com/amararun/tigzig-concurrency
Project-URL: Repository, https://github.com/amararun/tigzig-concurrency
Project-URL: Issues, https://github.com/amararun/tigzig-concurrency/issues
Author-email: Amar Harolikar <amar@harolikar.com>
License: MIT
Keywords: backpressure,concurrency,fastapi,queue,rate-limit
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Requires-Python: >=3.8
Requires-Dist: starlette>=0.27.0
Description-Content-Type: text/markdown

# tigzig-concurrency

Bounded wait queue + client-IP forwarding for FastAPI / Starlette backends.

A standing-rule pattern for the tigzig fleet. Replaces ad-hoc per-app
concurrency-check and IP-extraction code so all backends behave the same
way under load.

## Why this exists

Two recurring problems across the fleet:

1. **Fast-fail concurrency caps reject legitimate users.** A user
   double-clicks submit, two of their parallel sub-requests collide, and
   one gets a 429 even though the server is mostly idle. This package
   replaces the fast-fail check with a small bounded queue: the second
   request waits up to N seconds for a slot, then either runs or 429s.
   The server is still protected from sustained overload (queue is
   bounded), but micro-bursts succeed.

2. **Backend-to-backend calls trip downstream per-IP limits.** When
   backend A (e.g. TA) calls backend B (e.g. yfin) on behalf of an end
   user, B sees the request from A's container IP - so two unrelated
   end users routed via A look like ONE user at B and trip B's per-IP
   cap. This package provides ``forward_client_headers()`` so A passes
   the real user IP, and ``get_real_client_ip()`` (which B already uses)
   reads it.

## Installation

```bash
pip install tigzig-concurrency
```

Add to your ``requirements.txt``:

```
tigzig-concurrency>=0.1.0
```

## Quick start

### 1. Wrap your endpoint with the queue

```python
from fastapi import FastAPI, Request
from tigzig_concurrency import BoundedQueue, get_real_client_ip

app = FastAPI()
queue = BoundedQueue.from_env()  # reads MAX_INFLIGHT_*, MAX_QUEUE_* from env

@app.post("/api/work")
async def do_work(request: Request):
    client_ip = get_real_client_ip(request)
    await queue.acquire(client_ip)
    try:
        # ... your endpoint work ...
        return {"ok": True}
    finally:
        queue.release(client_ip)
```

### 2. Forward the real user IP on backend-to-backend calls

```python
import httpx
from tigzig_concurrency import forward_client_headers, get_real_client_ip

@app.post("/api/work")
async def do_work(request: Request):
    client_ip = get_real_client_ip(request)
    await queue.acquire(client_ip)
    try:
        async with httpx.AsyncClient() as http:
            response = await http.get(
                "https://other-backend.tigzig.com/data",
                headers=forward_client_headers(client_ip),
            )
        return response.json()
    finally:
        queue.release(client_ip)
```

The downstream backend's ``get_real_client_ip()`` will read the
``X-Original-Client-IP`` header and rate-limit per real end user.

## Configuration cheatsheet

Five environment variables. **You will forget what these mean in three
weeks - this is the cheatsheet.** Future-you, hello.

### The five knobs

| Variable | What it does | When to raise / lower |
|---|---|---|
| ``MAX_INFLIGHT_PER_IP`` | Active requests per IP at one time. | One real user can have this many running at once. Higher = more permissive for power-users; lower = stricter fairness. |
| ``MAX_INFLIGHT_GLOBAL`` | Active requests across all IPs combined. | Cap on total CPU/memory the server commits at once. Higher = more parallelism if you have headroom; lower if memory is tight. |
| ``MAX_QUEUE_PER_IP`` | Extra requests per IP that may wait in line. | Bigger queue = more forgiveness for "user clicked submit 4 times in 5 seconds". 0 = no queueing per-IP (fast-fail behaviour). |
| ``MAX_QUEUE_GLOBAL`` | Extra requests across all IPs that may wait. | Total system buffer for bursts. Beyond this -> instant 503. |
| ``MAX_QUEUE_WAIT_SECONDS`` | How long any one request waits in queue before 429. | Match to typical request length; ``wait + work + slack`` should stay under your edge timeout (Cloudflare free is 100s). |

### Behaviour

| Situation | Response |
|---|---|
| Slot available | Run immediately (no queue overhead). |
| Slot taken, queue not full | Wait up to ``MAX_QUEUE_WAIT_SECONDS`` for a slot, then run, OR 429 if the wait expired. |
| Per-IP queue full at arrival | 503 immediately (this user is hammering). |
| Global queue full at arrival | 503 immediately (server is overloaded). |

Both 429 and 503 include a ``Retry-After: 10`` header so the frontend
can show a friendly "server busy, retry in 10s" spinner instead of a
hard error.

### Tuning profiles for the tigzig fleet

Three workload types cover most apps. Pick the closest profile, set the
env vars accordingly, and tune from there if needed.

#### Profile A: fast IO-bound (1-5s per request)
For yfin, mdtopdf, llama-parse, markitdown, neon-proxy, supabase-proxy,
and any wrapper that mostly does network I/O + light pandas.

```
MAX_INFLIGHT_PER_IP=6
MAX_INFLIGHT_GLOBAL=15
MAX_QUEUE_PER_IP=4
MAX_QUEUE_GLOBAL=10
MAX_QUEUE_WAIT_SECONDS=5
```

Why: requests are quick, so being aggressive with concurrency is safe
(no memory pressure). Short wait window because end users expect fast
responses; queue absorbs occasional bursts.

#### Profile B: slow LLM / compute-bound (10-45s per request)
For TA, FFN/SPR, QuantStats, codeinter (Python sandbox), codeinter-gemini.

```
MAX_INFLIGHT_PER_IP=2
MAX_INFLIGHT_GLOBAL=6
MAX_QUEUE_PER_IP=3
MAX_QUEUE_GLOBAL=8
MAX_QUEUE_WAIT_SECONDS=30
```

Why: each request hogs a CPU or holds an LLM connection for a long time.
Tighter in-flight cap to avoid memory blowup. Longer wait because users
already expect slow; an extra 15-30s of queue is invisible vs the 30s of
actual work. Total ``wait + work`` stays under Cloudflare 100s.

#### Profile C: heavy memory / database (DuckDB queries, Flowise)
For duckdb-backend, duckdb-dashboards-backend, flowise-docker-custom.

```
MAX_INFLIGHT_PER_IP=2
MAX_INFLIGHT_GLOBAL=4
MAX_QUEUE_PER_IP=2
MAX_QUEUE_GLOBAL=4
MAX_QUEUE_WAIT_SECONDS=15
```

Why: a single heavy DuckDB query can pin a CPU AND eat a few hundred MB
of memory. Tight global cap matters more than per-IP fairness. Bump the
container memory limit if you raise the global cap.

### Container-limit reminder

These knobs only protect the server if the container itself has enough
CPU and memory. For the tigzig fleet's standard:

```
limits_cpus=2.0
limits_memory=1024m   (or 2048m for memory-heavy or many-concurrent profiles)
limits_memory_swap=2048m
```

A queue with ``MAX_INFLIGHT_GLOBAL=15`` on a container capped at
``cpus=0.25`` will just queue everything and run them slowly. The CPU
bump is the bigger lever.

## Multiple queues in one app

If one endpoint is much heavier than others, you can run two queues
with different settings via the ``prefix`` argument:

```python
light_queue = BoundedQueue.from_env(prefix="LIGHT_", name="light")
heavy_queue = BoundedQueue.from_env(prefix="HEAVY_", name="heavy")
```

Then set ``LIGHT_MAX_INFLIGHT_PER_IP``, ``HEAVY_MAX_INFLIGHT_PER_IP`` etc.
in your environment.

## Health / introspection

The queue exposes its current state via a ``stats`` property. Useful
for ``/healthz`` endpoints:

```python
@app.get("/healthz/queue")
def queue_health():
    return queue.stats
```

Returns:

```json
{
  "name": "queue",
  "limits": {"inflight_per_ip": 2, "inflight_global": 6, ...},
  "current": {"inflight_global": 3, "queued_global": 1, "queued_per_ip": {"1.2.3.4": 1}}
}
```

## Reference: header chain that ``get_real_client_ip`` reads

In order:

1. ``X-Original-Client-IP`` (canonical for trusted upstream backends)
2. ``CF-Connecting-IP`` (Cloudflare)
3. ``X-Forwarded-For`` (first IP if comma-separated)
4. ``X-Real-IP``
5. ``request.client.host`` (socket peer; usually docker proxy)

Returns ``"unknown"`` if no IP can be resolved.

## Why ``X-Original-Client-IP`` over ``X-Forwarded-For``

XFF gets rewritten by every hop (Cloudflare, Caddy, nginx). For trusted
internal forwarding it's safer to use a custom header that only the
trusted upstream sets. ``X-Original-Client-IP`` is the convention across
the tigzig fleet; the helper sets both to maximize compatibility but
the canonical reader prefers the custom header.

## Versioning

Semantic versioning. Breaking changes to function signatures or env-var
names bump the major version. Tuning-profile recommendations may evolve
across minor versions; check the README of the version you're using.

## License

MIT.
