Metadata-Version: 2.4
Name: forge-infer-cloud
Version: 0.1.3
Summary: OpenAI-compatible reasoning-aware inference proxy for Qwen3.6
Author: ArkaD171717
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: forge,inference,proxy,qwen,reasoning,thinking
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: aiosqlite>=0.20
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: qwen-think>=0.1.0
Requires-Dist: uvicorn[standard]>=0.27
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# forge-cloud

OpenAI-compatible reasoning-aware inference proxy for Qwen3.6.

Point your OpenAI client at forge-cloud instead of directly at vLLM/SGLang/Ollama. The proxy routes thinking mode based on query complexity, swaps sampling parameters to match the mode, normalizes backend flags, and tags responses with routing metadata.

## What it does

1. Receives a standard `/v1/chat/completions` request
2. Classifies query complexity (simple/moderate/complex)
3. Decides thinking mode (think vs no_think) with correct sampling params
4. Normalizes the `enable_thinking` flag for the target backend (vLLM nested, DashScope top-level, llama.cpp server-side)
5. Forwards to the user's configured backend
6. Tags the response with routing metadata and estimated token split (thinking vs response)

The proxy does not run inference. It configures and monitors it.

## Install

```bash
pip install forge-cloud
```

## Quick start

```bash
# Set admin key and backend URL
export FORGE_ADMIN_KEY=my-secret
export FORGE_BACKEND_URL=http://localhost:8000
export FORGE_BACKEND_TYPE=vllm

# Start the proxy
forge-cloud
```

Create an API key:

```bash
curl -X POST http://localhost:8741/v1/keys \
  -H "Authorization: Bearer my-secret" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-app"}'
# Returns: {"key": "fk-...", "name": "my-app", "tier": "free", ...}
```

Use it like any OpenAI endpoint:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8741/v1",
    api_key="fk-..."  # key from above
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[{"role": "user", "content": "refactor this module"}],
)
print(response.choices[0].message.content)
```

## Response metadata

Every response includes a `forge` field with routing metadata and estimated token counts:

```json
{
  "id": "chatcmpl-test",
  "choices": ["..."],
  "usage": {"...": 0},
  "forge": {
    "thinking_mode": "think",
    "complexity": "complex",
    "backend": "vllm",
    "sampling_profile": "thinking",
    "thinking_tokens": 450,
    "response_tokens": 120
  }
}
```

## Endpoints

| Method | Path | Description |
|--------|------|-------------|
| POST | `/v1/chat/completions` | Proxied chat completion with forge routing |
| POST | `/v1/keys` | Create API key (admin auth required) |
| GET | `/health` | Proxy health check |

## Configuration

All settings are environment variables with `FORGE_` prefix:

| Variable | Default | Description |
|----------|---------|-------------|
| `FORGE_HOST` | `127.0.0.1` | Bind address |
| `FORGE_PORT` | `8741` | Port |
| `FORGE_BACKEND_URL` | `http://localhost:8000` | Default backend URL |
| `FORGE_BACKEND_TYPE` | `vllm` | Backend type: vllm, sglang, dashscope, llamacpp |
| `FORGE_FREE_DAILY_LIMIT` | `1000` | Free tier requests per day |
| `FORGE_ADMIN_KEY` | (empty) | Admin key for creating API keys |
| `FORGE_DB_PATH` | `forge.db` | SQLite database path |
| `FORGE_REQUEST_TIMEOUT` | `120.0` | Backend request timeout (seconds) |

## Per-key backend override

Each API key can have its own backend URL and type:

```bash
curl -X POST http://localhost:8741/v1/keys \
  -H "Authorization: Bearer my-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sglang-user",
    "tier": "paid",
    "backend_url": "http://sglang-server:30000",
    "backend_type": "sglang"
  }'
```

## Tiers

- **Free**: 1,000 requests/day, single backend target
- **Paid**: no rate limit, per-key backend routing

## Streaming

Streaming is supported. Set `stream: true` in the request and the proxy forwards the SSE stream from the backend.

**Note:** Streaming responses do not include forge metadata (`thinking_mode`, `complexity`, token counts). The proxy cannot inspect the response until the stream completes, so the `forge` field is only present in non-streaming responses.

## Dependencies

- [qwen-think](https://pypi.org/project/qwen-think/) -- thinking session manager (routing, budget, sampling)
- FastAPI + uvicorn
- httpx -- async HTTP client for backend forwarding
- aiosqlite -- async SQLite for API keys and usage tracking

## License

Apache-2.0
