Metadata-Version: 2.4
Name: keble-idea
Version: 0.6.2
Summary: Provider-neutral idea extraction, concept graph, generation, and search contracts.
Author-email: zma <bob0103779@gmail.com>
Requires-Python: >=3.13
Requires-Dist: keble-helpers<2,>=1.53.0
Requires-Dist: pydantic<3.0.0,>=2.10.0
Description-Content-Type: text/markdown

# Keble Idea

`keble-idea` owns the provider-neutral contracts and application behavior for
public content snapshots, standout ideas, semantic concepts, relationships,
immutable generations, durable work ledgers, and concept-first hybrid search.

It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant,
Neo4j, raw providers, or `keble-data-infra`. The host application implements
the protocols under `keble_idea.application.ports` and converts raw source
responses into idea-owned schemas at its outer boundary.

Version 0.5.0 changes immutable model execution from one physical deployment
to one Infisical-catalog logical model. Every model-backed envelope, receipt,
and accepted relationship freezes its purpose, `ProcessingTier`, selection
rule, logical model/provider keys, model revision, catalog fingerprint, and
generation. Every real provider request separately records a secret-free
`ProviderAttemptBinding` with endpoint-pool revision, endpoint, deployment,
region, and billing mode.

Version 0.6.0 aligns durable failure evidence with the universal raw contract:
a physical HTTP rejection may truthfully have complete zero billable usage
(for example reviewed authentication, permission, payment, invalid-request, or
rate-limit rejection). A zero-attempt local gate still cannot carry usage, and
`USAGE_ONLY` evidence remains financially incomplete.

Version 0.6.1 makes physical-attempt budgeting explicit on the immutable
failure snapshot. A zero-attempt `PLATFORM_RUNTIME` observation—such as waiting
for an already-paid completion projection—does not consume another provider
attempt. Provider/adapter failures and missing evidence remain conservative and
consume the budget. The wall-clock deadline still terminates local polling.

Version 0.6.2 completes the REVIEW2 replay boundary. Publication repositories
return one canonical hydrated result for both first publication and receipt
replay, so a redelivered paid extraction retains every idea, concept, and
posting identity needed for downstream projections. Durable upstream failure
evidence also keeps the authoritative classification origin and provider
mapping revision, while separately metered identity correction remains owned by
`ExtractionService` and is included exactly once in terminal receipt usage.

`INGEST` and `ENRICH` envelopes carry the tier before model selection;
model-backed envelopes carry it only inside `ModelExecutionSnapshot`. Platform
must resolve the logical snapshot before admission, select a physical endpoint
per attempt, and never substitute another logical model or tier during retry.
The removed physical snapshot fields and prior `model_id`, `error_code`, and
`retry_classification` contracts have no compatibility aliases.

## Architecture

```text
raw provider HTTP responses
        -> platform-owned adapters
        -> keble-idea schemas and services
        -> platform-owned persistence/projection adapters
```

MongoDB is canonical in the host application. Qdrant stores canonical concept
vectors only, Neo4j stores the concept graph only, Redis stores TTL search
frontiers, and RabbitMQ delivers identifiers for Mongo-backed queue envelopes.
Ideas are retrieved through scalar idea-to-concept postings rather than idea
vectors or multikey story-array scans.

Cost-bearing extraction and relationship protocols receive an
`EnvelopeExecutionIdentity` copied from the atomically leased Mongo envelope.
The envelope key identifies logical work; `attempt_number` identifies the
physical broker/worker retry. Optional `queued_at` and `leased_at` preserve the
exact durable queue interval without introducing provider or pricing ownership.
Financial adapters include the identity in provider attempt keys so redelivery
cannot overwrite or hide a second possible spend.

`ExtractionService` is the single batch-identity correction owner. The first
typed extraction response may contain missing, unknown, or duplicate content
keys; the service issues at most one separately metered corrective request,
merges both calls' `ProcessingUsage` by submitted content key, and rejects a
second mismatch with `BatchIdentityError`. Platform adapters may retain in-run
facet validation, but must not add a second identity `ModelRetry` loop or the
application correction and its billing identity become unreachable.

Publication replay has one canonical repository contract:
`aget_publication_result` returns the stored receipt plus the complete idea,
concept, and posting-key result for both the initial publish and every receipt
replay. `ExtractionService` never reconstructs a reduced replay result from the
receipt alone. Hosts must hydrate this contract from canonical persistence so a
paid retry cannot falsely close downstream projection work with empty concept
keys.

Side effects if changes:

- initial and replayed extraction return values must remain structurally equal;
- Platform Stage-A work derives its expected projection parts from these keys;
- host reconciliation repairs historical projections separately from replay
  prevention and must not mutate immutable receipts.

Model identity is a separate immutable value. `ModelExecutionSnapshot` stores
purpose, processing tier, selection rule, logical model/provider identity,
catalog fingerprint, and generation without endpoints, credentials, quotas,
or prices. `EXTRACT`, `INFER_RELATIONSHIP`, `PROJECT_CONCEPT`, and `REINDEX`
envelopes require their matching purpose; non-model jobs reject the snapshot.
`INGEST` and `ENRICH` retain an explicit tier until that snapshot exists.
Monetary pricing remains owned by the Platform ledger and resolves after one
physical endpoint is selected into `ProviderAttemptBinding`.
If no active purpose binding exists, Platform must raise
`ModelRouteUnavailableError` before broker publication. Configuration absence
is distinct from a generation-verification or provider-execution failure.
Every new search resolves and freezes one `SearchEmbeddingBinding` containing
the embedding execution snapshot and vector resource. That same binding enters
concept retrieval and the cursor session identity, so a host cannot read a
model route and Qdrant alias independently.

## Development

Python 3.13 and `uv` are required.

```bash
uv sync
uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
npx --yes pyright .
uv build
```

The portable test lane uses only reusable in-memory adapters from
`keble_idea.testing`. Stateful integration adapters belong to the platform and
must isolate and clean MongoDB, Redis, Qdrant, Neo4j, RabbitMQ, and artifact
state after each test.

## Contract rules

- Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
- Write exactly one minimal receipt for every content/generation terminal state.
- Hydrate the complete canonical publication result on receipt replay; never
  return an empty derived-key set for already-published work.
- Keep receipts provider-neutral: physical token/image/duration usage remains in
  `ProcessingUsage`, while monetary pricing and per-content allocation belong
  exclusively to the platform-owned financial ledger. Receipt schema version 3
  rejects the removed `cost_micros` and loose `model_id` fields.
- Keep batch-identity correction in `ExtractionService`: one corrective call,
  one ordinal owned by the host, and one merged receipt usage total.
- Use the eight finite semantic facets; provider/source metadata is not a concept.
- Use `standalone_concept_error` at model boundaries so invalid facet labels
  receive a typed correction before publication applies the same strict policy.
- Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
- Vectorize concepts only unless a separately approved retrieval eval changes the design.
- Use immutable generations and active pointers; never mutate active indexes in place.
- Persist named, collection-owned integer-version migration checkpoints so outer
  workers resume by stable record key and never migrate a corpus at startup.
- Preserve original ASR and OCR independently from English-normalized fields;
  unknown-language OCR belongs in `ocr_text_original`, never guessed into
  `ocr_text_en` from the transcript language.
- Cutover only verified generations through compare-and-swap pointers and preserve
  the immediately previous verified generation for exact rollback.
- Stamp relationship coverage with the exact embedding, neighborhood, and
  relationship generations plus bounded neighbor similarities; a valid
  null/no-relation result still completes coverage. Persisted current/previous
  neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs
  comparison.
- Keep `relationship_priority_score` and `relationship_priority_at` as bounded,
  provider-neutral scheduling hints. The platform derives them idempotently from
  posting quality/time so Stage-B graph budget reaches high-value/recent concepts
  first; they never change semantic concept identity or Stage-A search readiness.
- Compact alias postings asynchronously in bounded resumable pages; merge and
  unmerge correctness must never depend on compaction finishing first. Derived
  rows retain `origin_concept_key`, so unmerge removes them without deleting the
  original posting identities.
- Search is bounded, concept-first, count-free, and cursor-session based.
- Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
- Operator lifecycle is canonical and typed (schema version 2): content moves
  `ACTIVE -> ARCHIVED -> ACTIVE` and enters `PURGING` terminally; ideas move
  `ACTIVE -> RETIRED -> ACTIVE` and enter `PURGING` terminally. `archived_at` /
  `retired_at` must match the state, transitions go through
  `assert_lifecycle_transition`, and public reads exclude non-ACTIVE rows.
- Operator job commands use the canonical status sets: retry targets
  `TERMINAL_JOB_STATUSES`, cancel targets `CANCELLABLE_JOB_STATUSES`
  (never LEASED work), and purge work routes through `IdeaJobType.PURGE`.
- Processing tiers are exactly `ECONOMY`, `BALANCED`, and `QUALITY`. They
  express requested cost/quality position, not provider identity. A retry keeps
  the frozen tier; changing it requires explicit reprocessing.

## Durable provider recovery

Each queue envelope snapshots the complete active
`ProviderRecoveryProfileSnapshot`, its matching `attempt_limit`, and an absolute
deadline. Automatic retry retains that snapshot. A manual retry also preserves
it unless the operator explicitly requests the active profile. Profile records
are append-only and activation uses a revision-guarded pointer; changing the
pointer affects future admissions only.

The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second
fallback doubled to a one-hour cap, a 24-hour maximum provider hint,
deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five-
minute open circuit, one half-open probe, and blocked release batches of 100.
Those are defaults for the first persisted profile, not module constants that
silently mutate existing jobs.

`retry_at_for_failure` anchors delay and jitter to the failure's immutable
observation time. Re-evaluating the same envelope attempt therefore returns the
same not-before timestamp. A provider hint above the submitted 24-hour policy
becomes `BLOCKED`; attempt/deadline exhaustion or permanent failure becomes
`DEAD_LETTERED`; operator action becomes `BLOCKED`. Only a retryable failure
within both budgets becomes `RETRY_SCHEDULED`.

`JobFailureSnapshot.consumes_attempt_budget` is the sole classification for
whether a leased retry counts as external work. Queue decision and persistence
must use it together: a zero-attempt Platform observation is evaluated against
the prior physical-attempt count and its lease increment is refunded atomically.
This preserves earlier paid attempts while allowing local projection polling
until the immutable deadline.

`UpstreamFailureEvidence` retains bounded status/code/request identity, opaque
quota scope, exact capacity, physical-attempt count, usage, and a completeness
bit. Empty usage is known zero only when the producer explicitly marks it
complete; incomplete empty usage and every `USAGE_ONLY` event remain billing
unknown.
Data Infra evidence also retains a provider-neutral classification origin plus
the upstream adapter's mapping revision separately from the Platform mapping
revision. This lets operators distinguish provider, contract, and Data Infra
runtime classification without making Idea depend on a provider package.
Payment requires a reviewed billing-code/account-status source; HTTP 402,
negative capacity, quota, and rate limiting are never sufficient by themselves.

Side effects if changes:

- Platform Mongo queue/profile/incident adapters and central indexes implement
  these exact field paths and compare-and-swap semantics;
- Data Platform job and commerce-operation APIs expose the same failure/policy
  facts through generated clients and translated frontend labels;
- finance records failed physical usage exactly once while pre-provider gate
  rejection remains zero spend;
- queue decision and persistence must never duplicate or disagree with
  `consumes_attempt_budget`;
- `EnvelopeProcessResult` must continue to contain exactly one result reference
  or failure snapshot.

## Images are a critical asset

Content payloads treat visuals as first-class evidence:
`ProductContentPayload.image_artifacts` carries model-ready product photos,
`CommerceProfileContentPayload.image_artifacts` carries acquired creator or
seller portraits, `ShortVideoContentPayload.thumbnail_artifacts` carries safe
ranked-video thumbnails without transient playback URLs, and
`VideoContentPayload` preserves uploaded video bytes plus key frames.
`IdeaSearchMedia.from_payload` projects the representative image/video (with
`IdeaSearchSourceSummary` for type-specific display facts) onto every search
item so consuming platforms can render media-rich cards. Artifact references
stay checksum-addressed (`ArtifactRef`) — delivery URLs are always resolved by
the serving platform at read time.

## Commerce discovery content

Trending commerce ingestion stays provider-neutral after the platform boundary:

```python
profile = CommerceProfileContentPayload(
    profile_type=CommerceProfileType.CREATOR,
    display_name="Rocket Lab Notes",
    image_artifacts=[acquired_profile_image],
)

video = ShortVideoContentPayload(
    title="A compact model-rocket launch guide",
    thumbnail_artifacts=[acquired_thumbnail],
)
```

Products keep `ProductContentPayload`; creators and sellers share the typed
profile payload with a stable `profile_type`; channel-wide ranked videos use a
separate short-video payload. Provider names, provider response bags, volatile
playback URLs, and raw enum labels do not enter these durable contracts.

## Language and standout evidence

`ContentBody.original_language` remains the backward-compatible ISO code.
Optional `ContentLanguageProvenance` explains whether it was source-declared,
text/media detected, or unresolved; a missing provenance on `und` identifies a
legacy row. Provider-neutral `ContentStandoutEvidence` records why a source
surfaced content without claiming profitability. Performance observations are
identified by metric kind, unit, observed time, and window: equal duplicates
collapse to the standout-linked evidence, while conflicting values are rejected.
