Metadata-Version: 2.4
Name: keble-data-infra-api
Version: 0.1.0
Summary: Sole FastAPI composition root for Keble raw provider data.
Author-email: zhenhao-ma <bob0103779@gmail.com>
Requires-Python: <3.14,>=3.13
Requires-Dist: aiohttp<4,>=3
Requires-Dist: fastapi<1,>=0.110
Requires-Dist: httpx<1,>=0.28
Requires-Dist: keble-amz<3,>=2.0.0
Requires-Dist: keble-data-infra-contract<1,>=0.1.0
Requires-Dist: keble-db<2,>=1.9.0
Requires-Dist: keble-helpers<2,>=1.12.2
Requires-Dist: keble-keepa<2,>=1.3.0
Requires-Dist: keble-tiktok<1,>=0.5.0
Requires-Dist: motor<4,>=3
Requires-Dist: pymongo<5,>=4.10
Requires-Dist: redis<7,>=5
Requires-Dist: sentry-sdk<3,>=2
Requires-Dist: uvicorn<1,>=0.29
Description-Content-Type: text/markdown

# keble-data-infra-api

Install the composed raw-data API with
`pip install "keble-data-infra-api>=0.1.0,<1"`. The published distribution
resolves the released Amazon and TikTok providers; editable local sources in
the uv workspace are development-only.

The sole FastAPI composition root for raw Amazon and TikTok provider data.
Provider packages export framework-free services and manifests; this package
owns route generation, exception mapping, runtime resources, and ingestion.

The Stand Out platform consumes this service over HTTP and does not import its
contracts or provider packages.

## Local provider environment

Build an ignored API-local `.env` from the existing Keble developer env files.
The helper copies only the required named settings and prints key names, never
credential values:

```bash
python scripts/build_local_env.py
uv run --env-file .env pytest -q tests/live -m live
```

The live suite uses independent `RUN_KEEPA_LIVE`, `RUN_FASTMOSS_LIVE`, and
`RUN_LIVE_ECHOTIK` gates. A gate is written only when its complete credential
set exists. Each configured provider runs as a separate canary, so one missing
credential cannot hide another provider's result. Provider failures are test
failures; missing opt-in credentials are the only valid reason to skip. Live
providers honor `PROVIDER_HTTP_TRUST_ENV` through `ApiSettings` instead of
hard-coding direct network access, matching the production runtime boundary.
The pinned TikTok provider includes HTTPX's SOCKS transport, so production
SOCKS4/5 proxy settings remain executable rather than failing during client
construction. The image also pins the timezone-aware/warning-free keble-db
checkpoint used by the local-full persistence proof.

## Runtime ownership

`keble_data_infra_api.main:create_app` constructs the only raw FastAPI app.
`api.router:create_router` builds discovery and provider routes from the
registry, and `api.error_handlers` performs typed exception conversion. The
Stand Out adapter resolves the single product-search capability through
`GET /v1/{channel}/providers` and then calls its generated operation. The old
`/v1/commerce/search` route and compatibility DTO snapshot are deleted.

API settings and immutable application values reuse
`keble-helpers.PydanticModelConfig` or the contract-owned `ContractModel`.
Durable aggregates use `SchemaBase`/`MongoObjectBase`; do not introduce a local
`ConfigDict` or direct persistence `BaseModel` policy.

Mongo/Redis settings enable the API-owned provider infrastructure. The local
env builder copies only `MONGO_DB_URI`, `REDIS_URI`, and named provider
credentials; API-owned database/collection/namespace defaults stay isolated.
Startup creates cache indexes before provider handlers run and shutdown closes
all sync/async Mongo and Redis clients.

Video Enrichment projection is also API-owned and requires Mongo whenever its
Result origin is configured. One document stores the source event, owner-token
claim, attempts, failure state, and optional projection. Unique event/result
indexes plus expiring claims make concurrent consumers safe; Redis entries are
acknowledged only after the document atomically reaches `completed`. The old
replica-local JSON state file is not supported.

The same typed gateway centralizes submission and job/result correlation:

```text
POST /internal/video-enrichments:submit
GET  /internal/video-enrichment-jobs/{job_id}
GET  /internal/video-enrichment-jobs/{job_id}/projection
```

The projection lookup uses the API-owned job/status/completion-time compound
index. Consumers never call the independent media service or infer `result_id`
from provider storage. A projection 404 means durable completion ingestion is
still pending and is safe for a bounded worker retry.

The submission gateway forwards the expected media SHA-256 as a dedicated
integrity field. When an authorized provider transcript exists, the contract binds
its normalized content checksum, source reference, and source media checksum before
the independent service may skip ASR. The API never treats a provider title or video
description as spoken captions.

The producer's Job/Result payload contains additional stage, artifact, usage,
frame, warning, and error fields. Data Infra validates the typed stable subset
through the contract's source-projection inheritance and ignores only those
additive producer fields. All submission and public normalized models retain
`extra="forbid"`; compatibility does not weaken command validation.

Configured provider routes include:

```text
POST /v1/amazon/providers/keepa/products/lookup
POST /v1/amazon/providers/keepa/products/search
POST /v1/amazon/providers/keepa/products/rank
POST /v1/tiktok/providers/fastmoss/products/search
POST /v1/tiktok/providers/echotik/products/rank
```

Run the isolated all-provider generated-route canary with:

```bash
python scripts/build_local_env.py
uv run --env-file .env pytest -q tests/live/test_provider_routes_live.py -m live
```

## Container and observability

The multi-stage image installs exact pushed provider commits and receives the
GitHub token only as a BuildKit secret. It runs as UID 10001, exposes `/health`
for liveness and `/ready` for configured Mongo/Redis readiness, and closes all
runtime clients through FastAPI lifespan.

Build the credential-preserving Docker env and start the profile on the existing
`keble_default` network:

```bash
uv run --directory keble-data-infra-api python scripts/build_local_env.py
uv run --directory keble-data-infra-api python -m scripts.build_local_docker_env
GH_TOKEN="$(gh auth token)" docker compose \
  --env-file keble-data-infra-api/.env.docker \
  -f deploy/compose.raw-api.yaml --profile raw-api up --build
```

`.env`, `.env.docker`, and the BuildKit secret stay outside Git and image
layers. Set `SENTRY_DSN`, `SENTRY_ENVIRONMENT`, and
`SENTRY_TRACES_SAMPLE_RATE` in the source env to enable error/performance
telemetry. Registry discovery, generated operations, cache outcomes, lease
contention, upstream work, batches, and video ingestion emit structural spans;
request/response content and credentials are not attached.
