Metadata-Version: 2.4
Name: lexigram-ai-workers
Version: 0.1.1
Summary: AI background workers for the Lexigram Framework — batch embedding, document ingestion, DLQ, maintenance
Project-URL: Homepage, https://github.com/lexigram-dev/lexigram
Project-URL: Repository, https://github.com/lexigram-dev/lexigram
Project-URL: Documentation, https://docs.lexigram.dev
Project-URL: Issues, https://github.com/lexigram-dev/lexigram/issues
Project-URL: Changelog, https://github.com/lexigram-dev/lexigram/blob/main/CHANGELOG.md
Author-email: Lexigram Framework Team <team@lexigram.dev>
Maintainer-email: Lexigram Framework Team <team@lexigram.dev>
License: MIT
License-File: LICENSE
Keywords: ai,async,background-workers,framework,lexigram,python,workers
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: lexigram-contracts>=0.1.0
Requires-Dist: lexigram>=0.1.1
Requires-Dist: typing-extensions>=4.0.0
Provides-Extra: dev
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: lexigram-testing>=0.1.1; extra == 'test'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
Requires-Dist: pytest-mock>=3.10.0; extra == 'test'
Requires-Dist: pytest>=8.0.0; extra == 'test'
Description-Content-Type: text/markdown

# lexigram-ai-workers

AI background workers for the Lexigram Framework — batch embedding, document ingestion, DLQ, maintenance

---

## Overview

AI background workers for the Lexigram Framework. Handles the heavy-lifting off the request path: document ingestion, batch embedding generation, periodic maintenance, and dead-letter-queue recovery — all with progress tracking, exponential backoff, and health reporting. Zero-config usage starts with sensible defaults.


> Full documentation: [docs.lexigram.dev](https://docs.lexigram.dev)
## Install

```bash
uv add lexigram-ai-workers
```

## Quick Start

```python
from lexigram import Application
from lexigram.di.module import Module, module

from lexigram.ai.workers import WorkersModule
from lexigram.ai.workers.config import WorkersConfig

@module(imports=[
    WorkersModule.configure(
        WorkersConfig(
            batch_embedding_concurrency=3,
            document_ingestion_concurrency=3,
            enable_maintenance=True,
            dlq_check_interval=60,
        )
    )
])
class AppModule(Module):
    pass

app = Application(modules=[AppModule])
if __name__ == "__main__":
    app.run()
```

## Configuration

> **Zero-config usage:** Call `WorkersModule.configure()` with no arguments to use defaults.

### Option 1 — YAML file

```yaml
# application.yaml
ai_workers:
  enabled: true
  batch_embedding_concurrency: 3
  document_ingestion_concurrency: 3
  enable_maintenance: true
  dlq_check_interval: 60
```

### Option 2 — Profiles + Environment Variables *(recommended)*

```bash
export LEX_AI_WORKERS__BATCH_EMBEDDING_CONCURRENCY=5
# Environment variables for each field
```

### Option 3 — Python

```python
from lexigram.ai.workers.config import WorkersConfig
from lexigram.ai.workers import WorkersModule

config = WorkersConfig(
    enabled=True,
    batch_embedding_concurrency=5,
    document_ingestion_concurrency=3,
    enable_maintenance=True,
    dlq_check_interval=60,
)
WorkersModule.configure(config)
```

### Config reference

| Field | Default | Env var | Description |
|-------|---------|---------|-------------|
| `enabled` | `True` | `LEX_AI_WORKERS__ENABLED` | Master on/off switch for all background workers |
| `batch_embedding_concurrency` | `3` | `LEX_AI_WORKERS__BATCH_EMBEDDING_CONCURRENCY` | Concurrent embedding batch tasks |
| `document_ingestion_concurrency` | `3` | `LEX_AI_WORKERS__DOCUMENT_INGESTION_CONCURRENCY` | Concurrent document processing tasks |
| `enable_maintenance` | `True` | `LEX_AI_WORKERS__ENABLE_MAINTENANCE` | Enable vector-store and cache maintenance |
| `dlq_check_interval` | `60` | `LEX_AI_WORKERS__DLQ_CHECK_INTERVAL` | Seconds between DLQ recovery sweeps |

## Module Factory Methods

| Method | Description |
|--------|-------------|
| `WorkersModule.configure(config, enable_scheduler)` | Configure with explicit config |
| `WorkersModule.stub(config)` | Minimal config for testing |

## Key Features

- **Document ingestion worker**: Parse PDF, DOCX, TXT, HTML, Markdown into chunks for vector store
- **Batch embedding worker**: Process chunks in configurable batches with in-memory embedding cache
- **Dead letter queue worker**: Handle failed jobs with failure classification and exponential backoff
- **Maintenance worker**: Periodic vector store index optimization, cache cleanup, document cleanup
- **Progress tracking**: Job progress monitoring with cache hit rate statistics
- **Adapters**: RAGAdapter, TasksAdapter, LoaderWorker for ecosystem integration

## Testing

```python
async with Application.boot(modules=[WorkersModule.stub()]) as app:
    # your test code
    ...
```

## Key Source Files

| File | What it contains |
|------|-----------------|
| `src/lexigram/ai/workers/module.py` | `WorkersModule.configure()`, `.stub()` |
| `src/lexigram/ai/workers/config.py` | `WorkersConfig` |
| `src/lexigram/ai/workers/document_ingestion/worker.py` | `DocumentIngestionWorker` |
| `src/lexigram/ai/workers/batch_embedding/worker.py` | `BatchEmbeddingWorker` |
| `src/lexigram/ai/workers/dlq/worker.py` | `DeadLetterQueueWorker` |
| `src/lexigram/ai/workers/maintenance/worker.py` | `MaintenanceWorker` |
| `src/lexigram/ai/workers/types.py` | `DLQItem`, `DLQStats`, `MaintenanceTask`, `MaintenanceResult` |
| `src/lexigram/ai/workers/di/provider.py` | `WorkersProvider` |
