Metadata-Version: 2.4
Name: exonware-xwsystem
Version: 0.9.0.49
Summary: Enterprise-grade Python framework with AI-powered performance optimization, 24 serialization formats, military-grade security, automatic memory leak prevention, circuit breakers, and production monitoring - replaces 50+ dependencies
Project-URL: Homepage, https://exonware.com
Project-URL: Repository, https://github.com/exonware/xwsystem
Project-URL: Documentation, https://github.com/exonware/xwsystem#readme
Project-URL: Subtree, https://github.com/exonware/xwsystem.git
Author: eXonware Backend Team
License: Apache-2.0
License-File: LICENSE
Keywords: ai-optimization,bson,cbor,circuit-breaker,crypto,enterprise,exonware,framework,json,memory-management,monitoring,msgpack,object-pool,performance,security,serialization,threading,yaml
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Security :: Cryptography
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.12
Requires-Dist: typing-extensions>=4.0.0
Provides-Extra: dev
Requires-Dist: aiofiles>=23.0.0; extra == 'dev'
Requires-Dist: aiohttp>=3.9.0; extra == 'dev'
Requires-Dist: bcrypt>=4.0.0; extra == 'dev'
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: cbor2>=5.0.0; extra == 'dev'
Requires-Dist: cryptography>=3.4.0; extra == 'dev'
Requires-Dist: defusedxml>=0.7.0; extra == 'dev'
Requires-Dist: dicttoxml>=1.7.0; extra == 'dev'
Requires-Dist: fastavro; extra == 'dev'
Requires-Dist: httpx>=0.23.0; extra == 'dev'
Requires-Dist: hyperlight-hyperjson>=1.0.0; extra == 'dev'
Requires-Dist: ijson>=3.2.0; extra == 'dev'
Requires-Dist: isort>=5.12.0; extra == 'dev'
Requires-Dist: json5>=0.9.0; extra == 'dev'
Requires-Dist: jsonpatch>=1.32; extra == 'dev'
Requires-Dist: jsonpointer>=2.3; extra == 'dev'
Requires-Dist: jsonschema>=4.17.0; extra == 'dev'
Requires-Dist: lxml>=4.9.0; extra == 'dev'
Requires-Dist: lz4>=4.0.0; extra == 'dev'
Requires-Dist: msgpack>=1.0.0; extra == 'dev'
Requires-Dist: msgspec>=0.18.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: opentelemetry-api>=1.15.0; extra == 'dev'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.15.0; extra == 'dev'
Requires-Dist: opentelemetry-exporter-zipkin>=1.15.0; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.15.0; extra == 'dev'
Requires-Dist: orjson>=3.8.0; extra == 'dev'
Requires-Dist: psutil>=5.9.0; extra == 'dev'
Requires-Dist: pymongo>=3.12.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest-benchmark>=4.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: pywin32>=300; (platform_system == 'Windows') and extra == 'dev'
Requires-Dist: pyyaml-include>=1.3.0; extra == 'dev'
Requires-Dist: pyyaml>=5.4.0; extra == 'dev'
Requires-Dist: requests>=2.28.0; extra == 'dev'
Requires-Dist: rtoml>=0.9.0; extra == 'dev'
Requires-Dist: ruamel-yaml>=0.17.0; extra == 'dev'
Requires-Dist: setuptools>=65.0.0; extra == 'dev'
Requires-Dist: toml>=0.10.2; extra == 'dev'
Requires-Dist: tomli-w>=1.0.0; extra == 'dev'
Requires-Dist: tomli>=2.0.0; extra == 'dev'
Requires-Dist: xmlschema>=2.0.0; extra == 'dev'
Requires-Dist: xmltodict>=0.13.0; extra == 'dev'
Provides-Extra: full
Requires-Dist: aiofiles>=23.0.0; extra == 'full'
Requires-Dist: aiohttp>=3.9.0; extra == 'full'
Requires-Dist: bcrypt>=4.0.0; extra == 'full'
Requires-Dist: cbor2>=5.0.0; extra == 'full'
Requires-Dist: cryptography>=3.4.0; extra == 'full'
Requires-Dist: defusedxml>=0.7.0; extra == 'full'
Requires-Dist: dicttoxml>=1.7.0; extra == 'full'
Requires-Dist: fastavro; extra == 'full'
Requires-Dist: httpx>=0.23.0; extra == 'full'
Requires-Dist: hyperlight-hyperjson>=1.0.0; extra == 'full'
Requires-Dist: ijson>=3.2.0; extra == 'full'
Requires-Dist: json5>=0.9.0; extra == 'full'
Requires-Dist: jsonpatch>=1.32; extra == 'full'
Requires-Dist: jsonpointer>=2.3; extra == 'full'
Requires-Dist: jsonschema>=4.17.0; extra == 'full'
Requires-Dist: lxml>=4.9.0; extra == 'full'
Requires-Dist: lz4>=4.0.0; extra == 'full'
Requires-Dist: msgpack>=1.0.0; extra == 'full'
Requires-Dist: msgspec>=0.18.0; extra == 'full'
Requires-Dist: opentelemetry-api>=1.15.0; extra == 'full'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.15.0; extra == 'full'
Requires-Dist: opentelemetry-exporter-zipkin>=1.15.0; extra == 'full'
Requires-Dist: opentelemetry-sdk>=1.15.0; extra == 'full'
Requires-Dist: orjson>=3.8.0; extra == 'full'
Requires-Dist: psutil>=5.9.0; extra == 'full'
Requires-Dist: pymongo>=3.12.0; extra == 'full'
Requires-Dist: pywin32>=300; (platform_system == 'Windows') and extra == 'full'
Requires-Dist: pyyaml-include>=1.3.0; extra == 'full'
Requires-Dist: pyyaml>=5.4.0; extra == 'full'
Requires-Dist: requests>=2.28.0; extra == 'full'
Requires-Dist: rtoml>=0.9.0; extra == 'full'
Requires-Dist: ruamel-yaml>=0.17.0; extra == 'full'
Requires-Dist: setuptools>=65.0.0; extra == 'full'
Requires-Dist: toml>=0.10.2; extra == 'full'
Requires-Dist: tomli-w>=1.0.0; extra == 'full'
Requires-Dist: tomli>=2.0.0; extra == 'full'
Requires-Dist: xmlschema>=2.0.0; extra == 'full'
Requires-Dist: xmltodict>=0.13.0; extra == 'full'
Provides-Extra: lazy
Requires-Dist: exonware-xwlazy==1.0.1.82; extra == 'lazy'
Provides-Extra: xw
Description-Content-Type: text/markdown

# 🧰 xwsystem

**One install instead of 50+.** Serialization (24+ formats), caching, security, validation, HTTP, IPC, monitoring - same APIs everywhere. The base every other eXonware package builds on.

**Company:** eXonware.com · **Author:** eXonware Backend Team · **Email:** connect@exonware.com  

[![Status](https://img.shields.io/badge/status-beta-blue.svg)](https://exonware.com)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)

---

## 📦 Install

**Requires Python 3.12+.** (See [pyproject.toml](pyproject.toml) `requires-python`.)

| Install | What you get | When to use |
|---------|--------------|-------------|
| `pip install exonware-xwsystem` | **Lite** - core only, zero optional deps | Minimal footprint, or you install what you need. |
| `pip install exonware-xwsystem[lazy]` | **Lazy** - core + xwlazy; missing deps install on first import | Development; optional formats without pre-installing everything. |
| `pip install exonware-xwsystem[full]` | **Full** - core + common optionals pre-installed | Production or CI when you want all formats and features up front. |

Same package; `[lazy]` and `[full]` are extras. You can also install as `xwsystem` (same thing).

---

## 🚀 Quick start

**Serialization (same API for every format):**

```python
from exonware.xwsystem import JsonSerializer

s = JsonSerializer()
text = s.dumps({"name": "Alice", "age": 30})
data = s.loads(text)
```

**Lazy install:** With `[lazy]`, use normal imports; if a format backend is missing, xwlazy installs it on first use. No try/except import blocks.

**Windows console UTF-8:**

```python
from exonware.xwsystem.console.cli import ensure_utf8_console
ensure_utf8_console()
```

**Config:** `xwconfig.toml` at project root; in code: `from exonware.xwsystem import settings` then `settings.current()`.

---

## ✨ What you get

| Area | What’s in it |
|------|----------------|
| **One dependency** | Replaces dozens of imports (json, yaml, msgpack, crypto, requests, path validation, caching, …) with one API surface. |
| **24+ serialization formats** | Text (JSON, YAML, TOML, XML, CSV, INI, properties, .env), binary (MsgPack, BSON, CBOR, Pickle), database (Sqlite3, Dbm, Shelve), tabular (DataFrame, Excel, CSV dialects). Enterprise/scientific (~10 formats: Avro, Protobuf, Parquet, ORC, Arrow, etc.) integrate via xwformats. |
| **20+ caching variants** | LRU, LFU, TTL (sync + async), read-through/write-through/write-behind, two-tier (memory+disk), secure/encrypted caches, observable caches, Prometheus export. |
| **Security** | Path validation, file security, audit helpers, crypto utilities, validators, policies. |
| **Validation** | Pydantic-style XModel, schema discovery. |
| **Runtime** | Config (performance modes, logging), console (CLI, colors, progress, tables), monitoring (memory, performance, tracing), IPC (process fabric, queues, pipes, shared memory), HTTP client (HTTP/2, retries, streaming). |
| **Patterns** | Circuit breakers, retry, object pool, context managers. Plus: operations (diff, merge, patch), structures (tree walker, circular ref detection), threading (locks, thread-safe factory), utils (datetime, paths, string, web), data structures (trie, union-find), grammar/syntax (50+ languages, Monaco), plugins, query registry. |
| **Indexing** | Pluggable index backends (`io/indexing`): InMemory, FileBacked, **BTreeIndexBackend** (ordered, range_scan), **FullTextIndexBackend** and **FileBackedFullTextIndexBackend** (term index, persistent). See [docs/REF_15_API.md](docs/REF_15_API.md) (Indexing module). |

Lite = zero optional deps. Lazy = xwlazy installs format backends on first use. Full = common optionals pre-installed.

---

## 🎯 Why xwsystem? (Highlights)

**Automatic format detection (no guessing)** - One API for any file. `AutoSerializer` and `detect_format` pick the right codec from **file extension**, **magic bytes**, or **content patterns** (confidence-scored). No need to know whether it’s JSON, YAML, or MessagePack; pass a path or bytes and get the right serializer.

```python
from exonware.xwsystem import AutoSerializer, detect_format
from pathlib import Path

# By path: extension tells us the format
fmt = detect_format(Path("data.json"))   # → JSON

# One serializer for everything: format from path (extension)
auto = AutoSerializer()
data = auto.load_file(Path("config.yaml"))
auto.save_file(data, Path("out.toml"))
```

**JSON: 3x+ faster than stdlib** - Pluggable parsers: **orjson** (3-4x faster), optional **hybrid** (msgspec for read, orjson for write). Measured: 1MB serialize &lt;45ms, deserialize &lt;52ms (see [REF_54_BENCH](docs/REF_54_BENCH.md)).

**Caching: O(1), benchmark-backed** - LRU/LFU/TTL with sub-microsecond get/set (10K items). Default uses the fastest Python cache baseline (~33k GET ops/sec, ~3.3k MIXED in our benchmarks). SLAs and NFRs in [REF_54_BENCH](docs/REF_54_BENCH.md).

**Serialization benchmark (1MB data)** - From [REF_15_API](docs/REF_15_API.md) / [REF_54_BENCH](docs/REF_54_BENCH.md):

| Format     | Serialize | Deserialize |
|-----------|-----------|-------------|
| MessagePack | 23 ms   | 28 ms       |
| JSON       | 45 ms   | 52 ms       |
| YAML       | 123 ms  | 145 ms      |

**Benchmarks** - Campaigns live in [benchmarks/](benchmarks/) (per [GUIDE_54_BENCH](../docs/guides/GUIDE_54_BENCH.md)). Run from project root; full index: [benchmarks/INDEX.md](benchmarks/INDEX.md).

**Benchmarks summary (10-Feb-2026)** - Condensed from latest runs. All campaigns: JSON, caching, serialization, atomic I/O, operations, data structures, object pool, validation, threading locks, async fabric.

| Area | Best / representative result |
|------|------------------------------|
| **JSON** | JsonSerializer (hybrid/hyperjson): small ~798k decode / 506k encode ops/s; with hyperjson installed matches hyperjson. orjson/msgspec in same tier. |
| **Caching** | PylruCache (default): get 2.99M/s, put 2.83M/s, mixed 2.49M/s. CacheboxCache get 3.72M/s. TwoTierCache now included (put/get contract). |
| **Serialization** | MsgPackSerializer small: 862k decode / 301k encode. YamlSerializer medium: 15 decode / 39 encode. PickleSerializer small: 422k decode / 357k encode. Competitive with msgpack/PyYAML/stdlib pickle. |
| **Atomic I/O** | AtomicFileWriter: 205-357 writes/s (1KB-100KB). Plain write: 504-743 writes/s. Atomic adds ~2-3x latency for crash-safe commit. |
| **Operations** | diff (structural/content/full) small: 35k-48k ops/s. merge (deep/shallow/overwrite) small: 59k-70k ops/s. patch_apply small: 64k ops/s. |
| **Data structures** | TrieNode lookup 899k ops/s; dict lookup 26M ops/s. UnionFind (make_set+union+find) 573 runs/s. trie insert batch 49k ops/s. |
| **Object pool** | ObjectPool get+release (thread_safe=True) 350k ops/s; (thread_safe=False) 287k ops/s. direct SimpleObj() 2.93M ops/s. Pool overhead ~8x for reuse. |
| **Validation** | check_data_depth shallow 1.11M ops/s, deep 229k ops/s. validate_path_input (safe) 2.58M ops/s. |
| **Locks** | EnhancedRLock (track_stats=True) 1.53M ops/s; (track_stats=False) 3.37M ops/s. threading.RLock 7.14M ops/s. Use track_stats=False for fast path. |
| **Async fabric** | Submit 1k tasks ~3.27k ops/s; queue latency ~0.02 ms; shared memory reuse read ~0.006 ms (see benchmarks/20260210-benchmark xwsystem async fabric). |

**Using these results:** `create_cache()` already defaults to **PylruCache** when pylru is installed (~2.5-3M mixed ops/s). For hot-path locking without timeout or stats, use **`fast_lock()`** or **`EnhancedRLock(track_stats=False)`** (~3.4M ops/s). Use **TwoTierCache** when you need memory+disk; use **ObjectPool** when object creation cost outweighs pool overhead. See [REF_54_BENCH](docs/REF_54_BENCH.md) trade-offs table.

---

## 🦀 Rust and the rest of the stack

**Rust:** xwsystem is the reference implementation. Multi-language (TypeScript, Rust, Go) is planned via contracts/specs. Right now Python performance is comparable to what we would expect from a Rust port on the hot paths, so no conversion yet. Getting here took several rewrites; the current design is the one we kept.

**Ecosystem:** xwsystem is the foundation for 12+ eXonware projects (xwstorage, xwformats, xwjson, xwnode, xwdata, xwauth, xwquery, xwchat, xwui, *-server). They get one dependency instead of 50+, same APIs everywhere, and Lite/Lazy/Full so each package can ship lean or full. One place to fix bugs and add features for the whole stack.

---

## 📚 Full feature list and examples

For a complete feature tour, code samples, and platform notes see **[README_LONG.md](README_LONG.md)**.

---

## 📖 Docs and tests

Content in this README is aligned with the project REFs and [docs/GUIDE_01_USAGE.md](docs/GUIDE_01_USAGE.md) (per [GUIDE_63_README](../../docs/guides/GUIDE_63_README.md)).

- **Start:** [docs/INDEX.md](docs/INDEX.md) - doc index and quick links.
- **Use it:** [docs/GUIDE_01_USAGE.md](docs/GUIDE_01_USAGE.md) - installation, codecs, caching, runtime, production.
- **Requirements and status:** [docs/REF_01_REQ.md](docs/REF_01_REQ.md), [docs/REF_22_PROJECT.md](docs/REF_22_PROJECT.md).
- **API and design:** [docs/REF_15_API.md](docs/REF_15_API.md), [docs/REF_13_ARCH.md](docs/REF_13_ARCH.md).
- **DX and quality:** [docs/REF_14_DX.md](docs/REF_14_DX.md), [docs/REF_50_QA.md](docs/REF_50_QA.md), [docs/REF_54_BENCH.md](docs/REF_54_BENCH.md), [docs/REF_51_TEST.md](docs/REF_51_TEST.md).
- **Ideas and planning:** [docs/REF_12_IDEA.md](docs/REF_12_IDEA.md), [docs/REF_21_PLAN.md](docs/REF_21_PLAN.md).
- **Compliance:** [docs/compliance/](docs/compliance/). **Evidence:** [docs/logs/](docs/logs/) (changes, tests, benchmarks, plans).
- **Benchmarks:** [benchmarks/INDEX.md](benchmarks/INDEX.md) - run scripts in `benchmarks/<campaign>/scripts/` (JSON, caching, serialization, atomic I/O, operations, data structures, object pool, validation, locks). See [REF_54_BENCH](docs/REF_54_BENCH.md).

**Tests:** 4-layer suite (0.core, 1.unit, 2.integration, 3.advance). Run via project test runner or pytest. See [docs/REF_51_TEST.md](docs/REF_51_TEST.md).

---

## 🧭 Where xwsystem fits

`xwsystem` provides the shared infrastructure layer for other eXonware packages (xwstorage, xwformats, xwjson, xwnode, xwdata, xwauth, xwquery, xwchat, xwui, *-server). They depend on its serializers, caches, security helpers, IPC layer, and runtime services instead of re-implementing them.

Approximate surface area (as of 2026-02-07, see REFs and benchmarks for exact lists):

- **Formats:** 24+ core serialization formats in this package, plus ~10 enterprise/scientific formats via xwformats.
- **Caching:** 20+ cache variants built from core implementations and wrappers (LRU/LFU/TTL, two-tier, observable, secure, async, read-through/write-through/write-behind).
- **Grammar/syntax:** 50+ language grammars with Monaco integration (see `grammar/` and [REF_13_ARCH](docs/REF_13_ARCH.md)).
- **IPC:** 4+ IPC primitives (process pool, message queue, shared memory, async fabric facade) wired into a single Async Process Fabric.
- **Benchmarks:** 10+ benchmark campaigns (JSON, caching, serialization, atomic I/O, operations, data structures, object pool, validation, locks, async fabric) tracked in `benchmarks/` and [REF_54_BENCH](docs/REF_54_BENCH.md).

Downstream libraries consume these services via stable APIs exported from `exonware.xwsystem`, so fixes and performance improvements in xwsystem flow through the rest of the ecosystem.

---

## 🌐 Ecosystem functional contributions

`xwsystem` is the provider layer for nearly every XW package; this table clarifies what downstream libs consume functionally.
You can use `xwsystem` standalone as a general-purpose runtime foundation in non-XW projects.
Adopting the full XW ecosystem is optional and primarily useful for enterprise and mission-critical platforms that want one self-managed infrastructure baseline across many services.

| Downstream XW lib | What xwsystem provides to it | Functional requirement xwsystem covers |
|------|----------------|----------------|
| **XWStorage** | Core serializer/runtime/security/caching utilities and base contracts. | Storage engine consistency, reliability, and shared ops tooling. |
| **XWFormats / XWJSON / XWData** | Codec registry, format detection, utility/runtime infrastructure. | Multi-format serialization and data pipeline foundations. |
| **XWNode / XWQuery** | Shared runtime, structures/utilities, and support primitives. | Predictable execution behavior for structure and query engines. |
| **XWAuth / XWAPI / *-API packages** | Security helpers, OAuth error parts, logging/config/runtime foundations. | Standardized auth/API behavior and transport-neutral operational consistency. |
| **XWEntity / XWModels / XWBase** | Base object model and cross-package infrastructure semantics. | Uniform object lifecycle, shared contracts, and reduced duplicated plumbing. |
| **XWChat / XWAI / XWBots** | Common runtime services (config, monitoring, utilities, serialization). | Reusable infrastructure for agent/chat automation services. |

This is the strategic advantage: improvements in one foundational runtime surface propagate to the full XW ecosystem without per-library rewrites.

---

## 📜 License and links

Apache-2.0 - see [LICENSE](LICENSE).

- **Homepage:** https://exonware.com  
- **Repository:** https://github.com/exonware/xwsystem  

Part of the eXonware ecosystem - one foundation for all of it.

## ⏱️ Async Support

<!-- async-support:start -->
- xwsystem includes asynchronous execution paths in production code.
- Source validation: 284 async def definitions and 210 await usages under src/.
- Use async APIs for I/O-heavy or concurrent workloads to improve throughput and responsiveness.
<!-- async-support:end -->
Version: 0.9.0.43 | Updated: 25-Apr-2026

*Built with ❤️ by eXonware.com - Revolutionizing Python Development Since 2025*
