Metadata-Version: 2.4
Name: pycraftcore
Version: 2.1.0
Summary: Provides reusable technical infrastructure and adapter implementations for Python applications
Author-email: Toufik Saouchi <toufik.saouchi@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/s-toufik/pycraftcore
Project-URL: Repository, https://github.com/s-toufik/pycraftcore
Project-URL: Issues, https://github.com/s-toufik/pycraftcore/issues
Keywords: infrastructure,hexagonal-architecture,configuration,logging,opentelemetry,resilient-http,repository,sandbox,asyncio
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: authentication
Provides-Extra: context
Provides-Extra: logger
Provides-Extra: profiler
Requires-Dist: pyinstrument; extra == "profiler"
Provides-Extra: query-language
Requires-Dist: sqlglot; extra == "query-language"
Provides-Extra: computation-engine
Requires-Dist: numpy; extra == "computation-engine"
Requires-Dist: scipy; extra == "computation-engine"
Provides-Extra: serializer
Requires-Dist: orjson; extra == "serializer"
Requires-Dist: msgpack; extra == "serializer"
Requires-Dist: pydantic>=2; extra == "serializer"
Provides-Extra: file-handler
Requires-Dist: orjson; extra == "file-handler"
Requires-Dist: pyyaml; extra == "file-handler"
Provides-Extra: runtime
Requires-Dist: pydantic>=2; extra == "runtime"
Provides-Extra: circuit-breaker
Requires-Dist: aiobreaker; extra == "circuit-breaker"
Requires-Dist: pycraftcore[logger]; extra == "circuit-breaker"
Provides-Extra: retry
Requires-Dist: tenacity; extra == "retry"
Requires-Dist: pycraftcore[logger]; extra == "retry"
Provides-Extra: http
Requires-Dist: aiohttp; extra == "http"
Requires-Dist: httpx; extra == "http"
Requires-Dist: orjson; extra == "http"
Requires-Dist: starlette; extra == "http"
Requires-Dist: pycraftcore[context,logger]; extra == "http"
Provides-Extra: resilient-http
Requires-Dist: aiohttp; extra == "resilient-http"
Requires-Dist: httpx; extra == "resilient-http"
Requires-Dist: pycraftcore[circuit_breaker,http,logger,retry,telemetry]; extra == "resilient-http"
Provides-Extra: application-configuration
Requires-Dist: omegaconf; extra == "application-configuration"
Requires-Dist: pydantic>=2; extra == "application-configuration"
Requires-Dist: python-dotenv; extra == "application-configuration"
Requires-Dist: pycraftcore[authentication,http,logger]; extra == "application-configuration"
Provides-Extra: telemetry
Requires-Dist: opentelemetry-api; extra == "telemetry"
Requires-Dist: opentelemetry-sdk; extra == "telemetry"
Requires-Dist: opentelemetry-exporter-otlp; extra == "telemetry"
Requires-Dist: opentelemetry-instrumentation-logging; extra == "telemetry"
Requires-Dist: pycraftcore[application_configuration,context]; extra == "telemetry"
Provides-Extra: repository
Requires-Dist: aiosqlite; extra == "repository"
Requires-Dist: pymongo; extra == "repository"
Requires-Dist: psycopg[binary]; extra == "repository"
Requires-Dist: psycopg-pool; extra == "repository"
Requires-Dist: orjson; extra == "repository"
Requires-Dist: pydantic>=2; extra == "repository"
Requires-Dist: pycraftcore[application_configuration,authentication]; extra == "repository"
Provides-Extra: runtime-sandbox
Requires-Dist: pandas; extra == "runtime-sandbox"
Requires-Dist: numpy; extra == "runtime-sandbox"
Requires-Dist: matplotlib; extra == "runtime-sandbox"
Requires-Dist: python-dateutil; extra == "runtime-sandbox"
Requires-Dist: pycraftcore[runtime]; extra == "runtime-sandbox"
Provides-Extra: all
Requires-Dist: pycraftcore[application_configuration]; extra == "all"
Requires-Dist: pycraftcore[authentication]; extra == "all"
Requires-Dist: pycraftcore[circuit_breaker]; extra == "all"
Requires-Dist: pycraftcore[computation_engine]; extra == "all"
Requires-Dist: pycraftcore[context]; extra == "all"
Requires-Dist: pycraftcore[file_handler]; extra == "all"
Requires-Dist: pycraftcore[http]; extra == "all"
Requires-Dist: pycraftcore[logger]; extra == "all"
Requires-Dist: pycraftcore[profiler]; extra == "all"
Requires-Dist: pycraftcore[query_language]; extra == "all"
Requires-Dist: pycraftcore[repository]; extra == "all"
Requires-Dist: pycraftcore[resilient_http]; extra == "all"
Requires-Dist: pycraftcore[retry]; extra == "all"
Requires-Dist: pycraftcore[runtime]; extra == "all"
Requires-Dist: pycraftcore[runtime-sandbox]; extra == "all"
Requires-Dist: pycraftcore[serializer]; extra == "all"
Requires-Dist: pycraftcore[telemetry]; extra == "all"
Dynamic: license-file

# pycraftcore

The shared technical foundation of the homelab's Python services: configuration, logging, telemetry, resilient HTTP, database repositories, safe SQL and a sandbox for running untrusted Python. Small, swappable building blocks rather than a framework — every module is an interface (a port) plus the implementations behind it (adapters).

**[Using pycraftcore](#using-pycraftcore)** — install it and use each module in your service.
**[Working on pycraftcore](#working-on-pycraftcore)** — how it is organised and how to release it.

```text
  your service (agent-orchestrator, agent-toolbox, …)
        │
        ▼
  ┌─ pycraftcore ─────────────────────────────────────────────────────────────┐
  │                                                                           │
  │  set up      application_configuration · logger · telemetry · context     │
  │  talk out    http · resilient_http (retry + circuit_breaker)              │
  │  store       repository: SQLite · PostgreSQL · MongoDB                    │
  │  run safely  query_language (read-only SQL) · runtime (Python sandbox)    │
  │  utilities   file_handler · serializer · computation_engine · profiler    │
  │  …           new modules, same port + adapter shape                       │
  │                                                                           │
  └───────────────────────────────────────────────────────────────────────────┘
```

---

## Using pycraftcore

### Install

Requires Python 3.14. Each module's third-party dependencies sit behind an extra of the same name, so a service installs only what it uses:

```bash
uv add "pycraftcore[http,repository]"     # or: pip install "pycraftcore[http,repository]"
uv add "pycraftcore[all]"                 # every module
```

A bare `pycraftcore` installs no third-party packages: only `authentication`, `context` and `logger` work without an extra. For any other module, importing its adapters without the extra fails with `ModuleNotFoundError`.

| Extra | Third-party packages | Also brings in |
|---|---|---|
| `authentication`, `context`, `logger` | none | — |
| `profiler` | pyinstrument | — |
| `query_language` | sqlglot | — |
| `computation_engine` | numpy, scipy | — |
| `serializer` | orjson, msgpack, pydantic | — |
| `file_handler` | orjson, pyyaml | — |
| `runtime` | pydantic | — |
| `runtime-sandbox` | pandas, numpy, matplotlib, python-dateutil | `runtime` |
| `circuit_breaker` | aiobreaker | `logger` |
| `retry` | tenacity | `logger` |
| `http` | aiohttp, httpx, orjson, starlette | `context`, `logger` |
| `application_configuration` | omegaconf, pydantic, python-dotenv | `http`, `logger`, `authentication` |
| `telemetry` | opentelemetry (api, sdk, otlp exporter, logging instrumentation) | `application_configuration`, `context` |
| `resilient_http` | aiohttp, httpx | `circuit_breaker`, `http`, `retry`, `telemetry` |
| `repository` | aiosqlite, pymongo, psycopg, psycopg-pool, orjson, pydantic | `application_configuration`, `authentication` |

List every module your code imports, even when one extra already brings in another. That way your service stays correct if these links change. For the Python sandbox, use `runtime-sandbox` rather than `runtime`: the sandboxed code runs on your service's interpreter, so the libraries it may import (pandas, numpy, matplotlib, dateutil) must be installed there.

Development builds are published to TestPyPI; use them for testing only. Their dependencies come from PyPI, so add it as an extra index:

```bash
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ "pycraftcore[http]"
```

### What's inside

Each module stands on its own: use the ones you need. The modules today (new ones follow the same shape, see [Working on pycraftcore](#working-on-pycraftcore)):

| Package | What it gives you |
|---|---|
| `application_configuration` | YAML configuration (connectors and operations) driven by environment variables, loaded into typed objects |
| `authentication` | The auth models a connector can use: none, basic, token |
| `logger` | One log format for the whole process, with the request id on every line |
| `context` | `request_id_context`, the request id shared by logs, telemetry and outgoing calls |
| `telemetry` | OpenTelemetry traces, metrics and logs, exported to an OTLP collector (or nothing) |
| `http` | Async HTTP clients (httpx, aiohttp), request-id and request-context middlewares, retry policies |
| `resilient_http` | An HTTP transport with retries and a circuit breaker built in |
| `retry`, `circuit_breaker` | The two policies on their own, for any async call |
| `repository` | Async repositories built from a database connector — today SQLite (pooled), PostgreSQL and MongoDB |
| `query_language` | Checks that a SQL statement is read-only, and translates between dialects, before it reaches a database |
| `runtime` | Runs untrusted Python in a separate process with memory, time and module limits; optionally lets it call your functions |
| `file_handler` | Read and write files by extension (today csv, json, yml, md, txt, svg); a new format is a new reader/writer pair |
| `serializer` | JSON, dict and binary (msgpack) serialisation of dataclasses |
| `computation_engine` | Numerical helpers: financial arithmetic, integration, interpolation |
| `profiler` | `@profiled`, an async profiling decorator |

### Configuration

A service's configuration is a folder of YAML files, with values taken from environment variables (`${oc.env:NAME}`):

| Path | What it holds |
|---|---|
| `root.yml` | The environment and the list of connectors and operations in use |
| `<env>/connector/*.yml` | Where things are: APIs, databases, files, MCP servers, telemetry |
| `<env>/operation/*.yml` | What you do with them; each operation points at a connector |

```yaml
# <env>/connector/api.yml
connector:
  weather:
    name: weather
    type: api
    base_url: https://<api-host>
    timeout: 5
    retry: 3
    auth:
      type: token
      key_name: apikey
      key_value: ${oc.env:WEATHER_API_KEY}
```

```yaml
# <env>/operation/weather.yml
operation:
  forecast:
    name: forecast
    type: api
    connector: ${connector.weather}
    endpoint: /forecast
    method: GET
    parameters:
      days: 3
```

```yaml
# root.yml
application_configuration:
  env: ${oc.env:APP_ENV,debug}
  run: async
  connector:
    api:
      weather: ${connector.weather}
  operation:
    forecast: ${operation.forecast}
```

```python
from pathlib import Path

from pycraftcore.application_configuration.adapter import (
    LoadApplicationConfiguration,
    OmegaConfigurationReader,
)
from pycraftcore.application_configuration.enum import RunTypeEnvironment
from pycraftcore.logger.adapter import StandardLogger

reader = OmegaConfigurationReader(RunTypeEnvironment.debug, Path("config"))
config = LoadApplicationConfiguration(reader, StandardLogger()).load()

connector = config.connector.api("weather")  # ApiConnector
operation = config.operation.api("forecast")  # ApiOperation
```

`config.connector` returns typed connectors per kind (`.api()`, `.database()`, `.file()`, `.mcp()`, `.telemetry()`) and `config.operation` typed operations (`.api()`, `.file()`); the `type` field of each entry decides which fields it takes.

### Logging and the request id

```python
from pycraftcore.logger import configure_logging
from pycraftcore.logger.adapter import StandardLogger

configure_logging("INFO")
logger = StandardLogger()
logger.info("booted")
```

```text
2026-09-27 10:06:17.679 | INFO     | 1bddfefb-… | controller:execute:51 - stream request accepted
```

- The third column is `request_id_context`. In a web service, `RequestIDMiddleware` (`pycraftcore.http.middleware`) sets it from the `X-Request-ID` header; every line logged during that request shows it, `-` otherwise. Don't write it in your messages.
- Uvicorn logs use the same format; chatty libraries (httpx, mcp, pymongo…) stay at `WARNING`.
- Calling `configure_logging` again replaces the handler rather than adding a second one.

### Telemetry

```python
from pycraftcore.application_configuration.enum import RunTypeEnvironment
from pycraftcore.telemetry.adapter import OpenTelemetryProvider

telemetry = OpenTelemetryProvider(
    service_name="my-service",
    environment=RunTypeEnvironment.debug,
    otlp_endpoint="<collector-host>:<port>",  # None: nothing is exported
)
tracer = telemetry.tracer("my-service")
```

Spans and log records carry the request id as an attribute, so logs and traces line up in Grafana.

### Resilient HTTP

A transport that retries what is worth retrying and stops calling a backend that keeps failing:

```python
from pycraftcore.circuit_breaker.configuration import CircuitBreakerSettings
from pycraftcore.http.configuration import HttpClientSettings, LimitsSettings
from pycraftcore.http.policy.http_error_policy import is_business_error, is_retryable
from pycraftcore.resilient_http.adapter import ResilientTransportFactory
from pycraftcore.resilient_http.configuration import ResilientHttpSettings
from pycraftcore.retry.configuration import RetrySettings

settings = ResilientHttpSettings(
    http=HttpClientSettings(limits=LimitsSettings(timeout=10)),
    retry=RetrySettings(
        retry_count=3, retry_delay=1, max_retry_delay=20, should_retry=is_retryable
    ),
    circuit_breaker=CircuitBreakerSettings(
        failure_threshold=3, recovery_timeout=30, is_excluded=is_business_error, name="weather"
    ),
)
client = ResilientTransportFactory(
    settings=settings, trace_manager=tracer, logger=logger
).create_async_client()
```

`client` is an `httpx.AsyncClient`: pass it to any library that accepts one.

### Databases

Repositories are built from a database connector and share one interface, `AsyncRepository.execute(statement)`:

```python
from pycraftcore.repository.adapter import SqliteRepositoryFactory, SqliteSettingsMapper

factory = SqliteRepositoryFactory(SqliteSettingsMapper(config.connector.database("users"))())
repository = await factory.connect()
rows = await repository.execute("SELECT id, email FROM users LIMIT 10")
await factory.disconnect()
```

`PostgresRepositoryFactory` / `PostgresSettingsMapper` and `MongoRepositoryFactory` / `MongoSettingsMapper` work the same way, and so will any database added later: a factory, a settings mapper and a repository behind the same `AsyncRepository` port.

### Read-only SQL

```python
from pycraftcore.query_language.adapter import SqlHandlerFactory

statement = SqlHandlerFactory()("SELECT * FROM users", dialect="sqlite").transpile()
```

`transpile()` raises when the statement writes, deletes or changes the schema, so only reads reach the database.

### Running untrusted Python

```python
from pycraftcore.runtime.adapter import PythonSafeCodeFactory
from pycraftcore.runtime.schema import CodeResult, SafeCodeSettings

factory = PythonSafeCodeFactory(settings=SafeCodeSettings(code_timeout=30, max_memory_mb=256))
output = await factory(code="import math\nresult = math.sqrt(2)").execute()
print(CodeResult.from_stdout(output.stdout).value)  # 1.4142135623730951
```

- The code runs in its own process; only the modules in `PYTHON_ALLOWLIST` can be imported (pandas, numpy, matplotlib, datetime…).
- `working_directory` in `SafeCodeSettings` confines file access to one folder.
- With a `HostBridgeServer`, the code can call functions of your service as if they were local — this is how agent-toolbox lets sandboxed code use its other tools.

---

## Working on pycraftcore

### How a module is organised

Every package follows the same shape, so a new backend is a new adapter, never a change for the services that use the port:

```text
  pycraftcore/<module>/
  ├── port/            Protocols the services depend on (AsyncRepository, Code, Logger…)
  ├── adapter/         Implementations (SqliteRepository, PythonSafeCode, StandardLogger…)
  ├── configuration/   Pydantic settings objects (RetrySettings, CircuitBreakerSettings…)
  ├── schema/          Data passed across the port (CodeResult, SafeCodeSettings…)
  └── enum/            Fixed choices (ConnectorType, HttpMethod…)
```

Not every module needs every folder: add a folder when it has something to hold.

### Design rules

- Services import the **port** and receive an adapter; they never construct a third-party client themselves.
- Adapters are thin: they translate between the port and the library, and keep library types out of the port.
- Settings are validated pydantic models; nothing reads environment variables except the configuration loader.
- Every public module is typed: `ty` runs in strict mode.

### Adding a module or an adapter

1. Define the protocol in `<module>/port/`.
2. Implement it in `<module>/adapter/`, with its settings in `configuration/` and the data it exchanges in `schema/`.
3. Export the public names from the package's `__init__.py`.
4. Declare its third-party packages as an extra named after the module in `[project.optional-dependencies]` (pull in other modules with `pycraftcore[<module>]`). Then add it to `all`, to the [Install](#install) table, and to the `extras` matrix in `.github/workflows/cicd.yml`, which installs each extra alone and imports its package.
5. Add tests under `tests/<module>/`, and the module to [What's inside](#whats-inside) — with a short example under Using pycraftcore if services will use it directly.

### Tests and checks

```bash
make install_dev    # uv sync --group dev --all-extras
make check          # ruff + ty + pytest
make test
make lint / make fix / make format / make typecheck
uv run pytest --cov=pycraftcore --cov-report=term-missing   # coverage per module
```

Pre-commit runs the test suite before every commit and push:

```bash
uv run pre-commit install --hook-type pre-commit --hook-type pre-push
```

### Releasing

1. Bump `version` in `pyproject.toml`.
2. `make build`.
3. `make publish_dev` to TestPyPI (needs `TEST_PYPI_TOKEN`), try it in a service, then `make publish` to PyPI (needs `RELEASE_PYPI_TOKEN`).
4. Raise the minimum version in the services that need the change (`pycraftcore[<extras>]>=…` in their `pyproject.toml`), then `uv lock --upgrade-package pycraftcore`.

To work on pycraftcore and a service at the same time, point the service at your checkout with an editable path dependency in its `pyproject.toml`: `pycraftcore = { path = "../pycraftcore", editable = true }` under `[tool.uv.sources]`.
