Metadata-Version: 2.4
Name: keble-idea
Version: 0.4.1
Summary: Provider-neutral idea extraction, concept graph, generation, and search contracts.
Author-email: zma <bob0103779@gmail.com>
Requires-Python: >=3.13
Requires-Dist: keble-helpers<2,>=1.52.2
Requires-Dist: pydantic<3.0.0,>=2.10.0
Description-Content-Type: text/markdown

# Keble Idea

`keble-idea` owns the provider-neutral contracts and application behavior for
public content snapshots, standout ideas, semantic concepts, relationships,
immutable generations, durable work ledgers, and concept-first hybrid search.

It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant,
Neo4j, raw providers, or `keble-data-infra`. The host application implements
the protocols under `keble_idea.application.ports` and converts raw source
responses into idea-owned schemas at its outer boundary.

Version 0.4.0 adds immutable model-execution provenance beside the canonical
job-failure and recovery-policy contract. Every model-backed envelope, receipt,
and accepted relationship freezes a secret-free `ModelExecutionSnapshot` with
one model revision, selector identity, purpose, and generation. Platform must
resolve the snapshot before admission; retries may not substitute a process-
global model, and the removed `model_id`, `error_code`, and
`retry_classification` fields have no compatibility aliases.

## Architecture

```text
raw provider HTTP responses
        -> platform-owned adapters
        -> keble-idea schemas and services
        -> platform-owned persistence/projection adapters
```

MongoDB is canonical in the host application. Qdrant stores canonical concept
vectors only, Neo4j stores the concept graph only, Redis stores TTL search
frontiers, and RabbitMQ delivers identifiers for Mongo-backed queue envelopes.
Ideas are retrieved through scalar idea-to-concept postings rather than idea
vectors or multikey story-array scans.

Cost-bearing extraction and relationship protocols receive an
`EnvelopeExecutionIdentity` copied from the atomically leased Mongo envelope.
The envelope key identifies logical work; `attempt_number` identifies the
physical broker/worker retry. Optional `queued_at` and `leased_at` preserve the
exact durable queue interval without introducing provider or pricing ownership.
Financial adapters include the identity in provider attempt keys so redelivery
cannot overwrite or hide a second possible spend.

Model identity is a separate immutable value. `ModelExecutionSnapshot` stores
purpose, model revision, provider/model/deployment/region/billing selector, and
generation without endpoints, credentials, quotas, or prices. `EXTRACT`,
`INFER_RELATIONSHIP`, `PROJECT_CONCEPT`, and `REINDEX` envelopes require their
matching purpose; non-model jobs reject the snapshot. Monetary pricing remains
owned by the Platform ledger and resolves from these frozen selector fields.

## Development

Python 3.13 and `uv` are required.

```bash
uv sync
uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
npx --yes pyright .
uv build
```

The portable test lane uses only reusable in-memory adapters from
`keble_idea.testing`. Stateful integration adapters belong to the platform and
must isolate and clean MongoDB, Redis, Qdrant, Neo4j, RabbitMQ, and artifact
state after each test.

## Contract rules

- Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
- Write exactly one minimal receipt for every content/generation terminal state.
- Keep receipts provider-neutral: physical token/image/duration usage remains in
  `ProcessingUsage`, while monetary pricing and per-content allocation belong
  exclusively to the platform-owned financial ledger. Receipt schema version 3
  rejects the removed `cost_micros` and loose `model_id` fields.
- Use the eight finite semantic facets; provider/source metadata is not a concept.
- Use `standalone_concept_error` at model boundaries so invalid facet labels
  receive a typed correction before publication applies the same strict policy.
- Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
- Vectorize concepts only unless a separately approved retrieval eval changes the design.
- Use immutable generations and active pointers; never mutate active indexes in place.
- Persist named, collection-owned integer-version migration checkpoints so outer
  workers resume by stable record key and never migrate a corpus at startup.
- Preserve original ASR and OCR independently from English-normalized fields;
  unknown-language OCR belongs in `ocr_text_original`, never guessed into
  `ocr_text_en` from the transcript language.
- Cutover only verified generations through compare-and-swap pointers and preserve
  the immediately previous verified generation for exact rollback.
- Stamp relationship coverage with the exact embedding, neighborhood, and
  relationship generations plus bounded neighbor similarities; a valid
  null/no-relation result still completes coverage. Persisted current/previous
  neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs
  comparison.
- Keep `relationship_priority_score` and `relationship_priority_at` as bounded,
  provider-neutral scheduling hints. The platform derives them idempotently from
  posting quality/time so Stage-B graph budget reaches high-value/recent concepts
  first; they never change semantic concept identity or Stage-A search readiness.
- Compact alias postings asynchronously in bounded resumable pages; merge and
  unmerge correctness must never depend on compaction finishing first. Derived
  rows retain `origin_concept_key`, so unmerge removes them without deleting the
  original posting identities.
- Search is bounded, concept-first, count-free, and cursor-session based.
- Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
- Operator lifecycle is canonical and typed (schema version 2): content moves
  `ACTIVE -> ARCHIVED -> ACTIVE` and enters `PURGING` terminally; ideas move
  `ACTIVE -> RETIRED -> ACTIVE` and enter `PURGING` terminally. `archived_at` /
  `retired_at` must match the state, transitions go through
  `assert_lifecycle_transition`, and public reads exclude non-ACTIVE rows.
- Operator job commands use the canonical status sets: retry targets
  `TERMINAL_JOB_STATUSES`, cancel targets `CANCELLABLE_JOB_STATUSES`
  (never LEASED work), and purge work routes through `IdeaJobType.PURGE`.

## Durable provider recovery

Each queue envelope snapshots the complete active
`ProviderRecoveryProfileSnapshot`, its matching `attempt_limit`, and an absolute
deadline. Automatic retry retains that snapshot. A manual retry also preserves
it unless the operator explicitly requests the active profile. Profile records
are append-only and activation uses a revision-guarded pointer; changing the
pointer affects future admissions only.

The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second
fallback doubled to a one-hour cap, a 24-hour maximum provider hint,
deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five-
minute open circuit, one half-open probe, and blocked release batches of 100.
Those are defaults for the first persisted profile, not module constants that
silently mutate existing jobs.

`retry_at_for_failure` anchors delay and jitter to the failure's immutable
observation time. Re-evaluating the same envelope attempt therefore returns the
same not-before timestamp. A provider hint above the submitted 24-hour policy
becomes `BLOCKED`; attempt/deadline exhaustion or permanent failure becomes
`DEAD_LETTERED`; operator action becomes `BLOCKED`. Only a retryable failure
within both budgets becomes `RETRY_SCHEDULED`.

`UpstreamFailureEvidence` retains bounded status/code/request identity, opaque
quota scope, exact capacity, physical-attempt count, usage, and a completeness
bit. Missing or `USAGE_ONLY` evidence on a physical call is billing unknown.
Payment requires a reviewed billing-code/account-status source; HTTP 402,
negative capacity, quota, and rate limiting are never sufficient by themselves.

Side effects if changes:

- Platform Mongo queue/profile/incident adapters and central indexes implement
  these exact field paths and compare-and-swap semantics;
- Data Platform job and commerce-operation APIs expose the same failure/policy
  facts through generated clients and translated frontend labels;
- finance records failed physical usage exactly once while pre-provider gate
  rejection remains zero spend;
- `EnvelopeProcessResult` must continue to contain exactly one result reference
  or failure snapshot.

## Images are a critical asset

Content payloads treat visuals as first-class evidence:
`ProductContentPayload.image_artifacts` carries model-ready product photos,
`CommerceProfileContentPayload.image_artifacts` carries acquired creator or
seller portraits, `ShortVideoContentPayload.thumbnail_artifacts` carries safe
ranked-video thumbnails without transient playback URLs, and
`VideoContentPayload` preserves uploaded video bytes plus key frames.
`IdeaSearchMedia.from_payload` projects the representative image/video (with
`IdeaSearchSourceSummary` for type-specific display facts) onto every search
item so consuming platforms can render media-rich cards. Artifact references
stay checksum-addressed (`ArtifactRef`) — delivery URLs are always resolved by
the serving platform at read time.

## Commerce discovery content

Trending commerce ingestion stays provider-neutral after the platform boundary:

```python
profile = CommerceProfileContentPayload(
    profile_type=CommerceProfileType.CREATOR,
    display_name="Rocket Lab Notes",
    image_artifacts=[acquired_profile_image],
)

video = ShortVideoContentPayload(
    title="A compact model-rocket launch guide",
    thumbnail_artifacts=[acquired_thumbnail],
)
```

Products keep `ProductContentPayload`; creators and sellers share the typed
profile payload with a stable `profile_type`; channel-wide ranked videos use a
separate short-video payload. Provider names, provider response bags, volatile
playback URLs, and raw enum labels do not enter these durable contracts.

## Language and standout evidence

`ContentBody.original_language` remains the backward-compatible ISO code.
Optional `ContentLanguageProvenance` explains whether it was source-declared,
text/media detected, or unresolved; a missing provenance on `und` identifies a
legacy row. Provider-neutral `ContentStandoutEvidence` records why a source
surfaced content without claiming profitability. Performance observations are
identified by metric kind, unit, observed time, and window: equal duplicates
collapse to the standout-linked evidence, while conflicting values are rejected.
