Metadata-Version: 2.4
Name: matyan-frontier
Version: 0.3.0
Summary: Ingestion gateway: WebSocket + presigned blob URLs, publishes to Kafka
Author-email: Tigran Grigoryan <grigoryan.tigran119@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/4gt-104/matyan-core
Project-URL: Repository, https://github.com/4gt-104/matyan-core
Project-URL: Documentation, https://4gt-104.github.io/matyan-core/
Keywords: experiment-tracking,ingestion,websocket,kafka
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Typing :: Typed
Requires-Python: <4,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi~=0.115
Requires-Dist: uvicorn[standard]~=0.34
Requires-Dist: pydantic~=2.0
Requires-Dist: pydantic-settings~=2.0
Requires-Dist: loguru~=0.7.3
Requires-Dist: aiokafka~=0.13
Requires-Dist: click~=8.0
Requires-Dist: prometheus-client~=0.21
Requires-Dist: aioboto3>=7.0.0
Requires-Dist: boto3>=1.42.53
Requires-Dist: google-cloud-storage~=2.14
Requires-Dist: gcloud-aio-storage~=9.3.0
Requires-Dist: azure-storage-blob~=12.24
Requires-Dist: azure-identity~=1.19
Requires-Dist: matyan-api-models~=0.2.0
Dynamic: license-file

# Matyan Frontier

Ingestion gateway between training clients and the rest of the Matyan stack. Clients connect via **WebSocket** for metrics, params, and logs, and via **REST** for presigned URLs (S3, GCS, or Azure Private SAS); the frontier publishes to **Kafka**. The UI and backend do not talk to the frontier. Part of the Matyan experiment-tracking stack (fork of Aim).

## Layout

- **`src/matyan_frontier/`** — Python package: FastAPI app, WebSocket handler, REST presign endpoint, Kafka producer, config, health, metrics.
- **Entrypoints**: `app.py` (lifespan: start Kafka producer, create blob storage clients, ensure bucket; shutdown: flush producer, close clients).
- **Routes**: `GET /api/v1/ws/runs/{run_id}` (WebSocket), `POST /api/v1/rest/artifacts/presign`, `GET /health/ready/`, `GET /health/live/`, `GET /metrics/` (Prometheus).

## Prerequisites

- Python 3.12+. The package uses `uv` in the repo: `uv run matyan-frontier` or install then `matyan-frontier` CLI.
- **Runtime dependencies**: Kafka (bootstrap reachable) and an blob store (e.g. RustFS in dev, AWS S3/GCS/Azure Blob Store in prod). The smoke test and local dev assume Kafka and S3 are up (e.g. via docker-compose).

## Run (production-like)

From the frontier package directory: `uv run matyan-frontier start` (or `matyan-frontier start` if installed).

Options: `--host`, `--port` (defaults: `0.0.0.0`, `53801`). The CLI uses these option defaults; config also defines `host`/`port` for other entry points.

## Configuration (environment variables)

| Variable | Default | Purpose |
|----------|---------|---------|
| `MATYAN_ENVIRONMENT` / `ENVIRONMENT` | `development` | When `production`, S3/Kafka must be non-dev (validated at startup). |
| `LOG_LEVEL` | `INFO` | Log level (loguru + uvicorn). |
| `HOST` | `0.0.0.0` | Bind address. |
| `PORT` | `53801` | Bind port. |
| `KAFKA_BOOTSTRAP_SERVERS` | `localhost:9092` | Kafka broker list. |
| `KAFKA_DATA_INGESTION_TOPIC` | `data-ingestion` | Topic for ingestion messages. |
| `KAFKA_SECURITY_PROTOCOL` / `KAFKA_SASL_*` | (empty) | Optional Kafka SASL. |
| `BLOB_BACKEND_TYPE` | `s3` | Storage backend: `s3`, `gcs`, or `azure`. |
| `S3_ENDPOINT` | `http://localhost:9000` | S3 API endpoint (e.g. RustFS). |
| `S3_PUBLIC_ENDPOINT` | `""` | Optional; used for presigned URLs if different from `S3_ENDPOINT`. |
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | (dev defaults) | S3 credentials. |
| `S3_BUCKET` | `matyan-artifacts` | Bucket for artifacts (S3). |
| `S3_REGION` | `us-east-1` | S3 region (default: `us-east-1`). |
| `GCS_BUCKET` | `matyan-artifacts` | Bucket for artifacts (GCS). |
| `AZURE_CONTAINER` | `matyan-artifacts` | Container for artifacts (Azure). |
| `AZURE_CONN_STR` | `""` | Azure connection string. |
| `AZURE_ACCOUNT_URL` | `""` | Azure account URL (for SAS user delegation key). |
| `S3_PRESIGN_EXPIRY` | `3600` | URL/SAS expiry (seconds). |
| `SHUTDOWN_FLUSH_TIMEOUT` | `5.0` | Seconds to wait for Kafka flush on shutdown. |
| `METRICS_ENABLED` | `true` | Expose Prometheus `/metrics/`. |
| `CORS_ORIGINS` | (localhost list) | Allowed origins (comma-separated or repeated). |

Source of truth: [config.py](src/matyan_frontier/config.py).

## Development and smoke test

- **Development**: Run Kafka and S3 (e.g. `docker compose up kafka kafka-init rustfs` or equivalent), then `uv run matyan-frontier start`. Clients (e.g. matyan-client) point at the frontier URL for WebSocket and presign.
- **Smoke test**: From `extra/matyan-frontier`, run `uv run python scripts/smoke_test.py`. Prerequisites: Kafka, S3, and frontier running. The script covers WebSocket (create_run, log_metric, log_hparams, finish_run, etc.) and REST presign, and optionally verifies Kafka consumption.

See [scripts/smoke_test.py](scripts/smoke_test.py) for the exact command and what it checks.

## Deployment

- **Docker**: Use [Dockerfile.dev](Dockerfile.dev) or [Dockerfile.prod](Dockerfile.prod) (context from repo root as needed; align with how the repo builds frontier images).

The UI talks only to the backend, not to the frontier; the frontier is for training clients.

## Related

- **Backend**: matyan-backend serves the REST API and consumes from Kafka (ingestion + control workers); it does not receive client traffic from the frontier directly.
- **Client**: matyan-client sends tracking data to the frontier (WebSocket + presign REST).
- **Monorepo**: This package lives under `extra/matyan-frontier` in the matyan-core repo.
