Metadata-Version: 2.3
Name: venvmux
Version: 1.0.0
Summary: Run Python functions in persistent, warm subprocesses inside isolated virtual environments.
Keywords: venv,subprocess,workers,isolation,autoscale,uv,pip
Author: Simon Waloschek
Author-email: Simon Waloschek <waloschek@pm.me>
License: MIT License
         
         Copyright (c) 2025 Simon Waloschek
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
         in the Software without restriction, including without limitation the rights
         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
         copies of the Software, and to permit persons to whom the Software is
         furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all
         copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-timeout>=2.1 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
Requires-Dist: ruff>=0.4 ; extra == 'dev'
Requires-Dist: mypy>=1.10 ; extra == 'dev'
Requires-Dist: pre-commit>=3.5 ; extra == 'dev'
Requires-Dist: orjson>=3.9 ; extra == 'speed'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/sonovice/venvmux
Project-URL: Repository, https://github.com/sonovice/venvmux
Provides-Extra: dev
Provides-Extra: speed
Description-Content-Type: text/markdown

venvmux
=======

Run Python functions in persistent, warm subprocesses inside isolated virtual environments.

### Motivation
- Microservice-like boundaries for Python code on a single machine, without deploying a service.
- Cleanly separate libraries with diverging dependencies by running them in dedicated virtual environments.
- Keep worker processes warm to avoid repeated heavy imports and initialization costs.
- Simple, local orchestration: your application calls functions; workers execute them in isolated subprocesses via JSONL over stdio.
- Great for: side-by-side versions (e.g., `numpy` v1 and v2), incremental upgrades, and safer experimentation.

### Features
 - Stdlib-only core, optional `uv` support if installed
 - Pyproject-first dependency resolution, fallbacks to requirements or inline packages
 - JSONL protocol over stdio; simple `Pool` API with timeouts and restarts
 - Context manager, async and batch calls; inflight concurrency control
 - Autoscaling workers by default (`workers="auto"`), fixed-size workers optional

### Limits and guarantees
- Trusted code only; no sandboxing or privilege dropping.
- Return values are sent as JSONL. venvmux applies a best-effort encoder so your worker functions don’t need decorators or library changes:
  - Dataclasses → dict
  - Enums → value (fallback: name)
  - datetime/date → ISO-8601 string
  - Decimal/UUID/Path → str
  - bytes → `{ "__bytes__": true, "b64": "..." }` (base64)
  - set/tuple → list
  - Optional: numpy.ndarray → list, pandas.DataFrame/Series → dict (if installed)
  - Duck-typed: `.model_dump()` / `.dict()` / `__json__()` if available
  - Unknown objects → `repr(obj)`
- Large results: optionally spill to a temp file to avoid large JSON frames.

### Install
- Requires Python 3.11+ (CPython). Supported platforms: macOS, Linux, Windows.
- Normal install:
  ```bash
  pip install venvmux
  ```
- With speed extra (optional; enables faster serialization via orjson if available):
  ```bash
  pip install "venvmux[speed]"
  ```
  - Behavior: when `Pool(serializer="orjson")` is set, workers attempt to use `orjson`; if not installed, they gracefully fall back to stdlib `json`.
- Editable for local dev:
   ```bash
   uv pip install -e .
   ```
   or with pip:
   ```bash
   python -m pip install -e .
   ```
- Editable with dev extras (contributors; installs pytest/ruff/mypy/pre-commit tooling):
  ```bash
  uv pip install -e .[dev]
  # or
  python -m pip install -e .[dev]
  ```
  - You can combine extras, e.g. `.[dev,speed]` to install both tooling and the speed extra.

### Quickstart (multiple environments)
- Create worker modules that expose plain Python functions (no decorators required)

```python
# file: yourpkg_v1/entry.py (module loaded by the worker process)
def compute(n: int) -> dict:
    return {"version": "v1", "n": n}

# file: yourpkg_v2/entry.py (module loaded by the worker process)
def compute(n: int) -> dict:
    return {"version": "v2", "n": n}
```

- Call those functions from your application using named environments

```python
# file: main.py (your application)
from venvmux import EnvSpec, Pool

pool = Pool.from_envs(
    {
        "v1": (EnvSpec(workers=1), "yourpkg_v1.entry"),
        "v2": (EnvSpec(workers=2), "yourpkg_v2.entry"),
    }
)

pkg_v1 = pool.venv("v1")
pkg_v2 = pool.venv("v2")

print(pkg_v1.compute(n=1))
print(pkg_v2.compute(n=2))
pool.close()
```

### Notes
- The `worker_module` (here `yourpkg_v1.entry` and `yourpkg_v2.entry`) must be importable in the worker process.
  - If your project has a `pyproject.toml`, venvmux will (by default) build an isolated venv and install your project there.
  - For local development, you can reuse the current interpreter: `EnvSpec(python=sys.executable)` and ensure your packages are on `PYTHONPATH` or use `worker_paths`.
    ```python
    import sys
    from venvmux import EnvSpec

    EnvSpec(python=sys.executable, worker_paths=["/abs/path/to/src"])  # example
    ```

### Lifecycle
- Use `pool.close()` to fully stop all environments and background threads; `close()` is an alias for `stop()`.
- To stop only a single environment without tearing down the pool, call `pool.stop_env("env_name")`.
- To gracefully restart all workers for one environment while keeping sizing, use `pool.reload_env("env_name")`.
- `pool.stop(name="env_name")` is also supported to stop a specific env.

### Context manager and async/batch
```python
from venvmux import EnvSpec, Pool

with Pool.from_envs({"env": (EnvSpec(workers=2), "yourpkg_v2.entry")}) as pool:
    # async
    fut = pool.call_async("compute", {"n": 21}, env="env")
    print(fut.result())

    # batch
    results = pool.venv("env").map("compute", [{"n": i} for i in range(3)], max_workers=2)
    print(results)
```

### Concurrency
- `EnvSpec.inflight` limits concurrent in-flight calls per worker process.

### Workers and autoscaling
- By default, `EnvSpec.workers = "auto"` and the pool autoscales:
  - Starts with a small number of workers (min 1 by default).
  - Grows when all current workers are saturated (up to `max_workers`, default: max(2, CPU count)).
  - Scales down after inactivity (`scale_down_idle_s`, default: 120s) back to at least `min_workers`.
- Tuning knobs on `EnvSpec` (optional):
  - `min_workers`: minimum warmed workers when using auto
  - `max_workers`: maximum workers when using auto
  - `autogrow`: enable/disable growth even if `workers="auto"`
  - `scale_down_idle_s`: idle time before scaling down
  - `saturation_window_len`: number of recent checks before growing (default 5)
  - `scale_up_cooldown_s`: minimum seconds between scale-ups (default 2.0)
- Fixed-size mode: set an integer, e.g. `EnvSpec(workers=4)`, to always run exactly 4 workers (autogrow disabled).

#### Autoscaling knobs (selected)
- `min_workers`/`max_workers`: bounds for auto mode.
- `inflight`: concurrent in-flight calls per worker process.
- `autogrow`: enable/disable growth when `workers="auto"`.
- `scale_down_idle_s`: idle time before shrinking back down.
 - `saturation_window_len`: number of recent checks to consider.
 - `scale_up_cooldown_s`: minimum seconds between scale-ups.

#### Autoscaling: quick reference
- Growth: requires sustained saturation (recent majority of checks show all workers at inflight limit).
- Shrink: after `scale_down_idle_s` of inactivity, oldest workers stop until `min_workers` remain.

### Payload size
- You can set `Pool(max_payload_bytes=...)` to spill large payloads to a temp file automatically.

### Result size
- You can set `Pool(max_result_bytes=...)` to spill large results to a temp file automatically. The worker returns a small manifest; the pool reads and returns the actual data transparently.

### Serializer
- Default serializer is stdlib `json`. For faster serialization, set `Pool(serializer="orjson")`. The pool passes `VENV_MUX_SERIALIZER=orjson` to workers; they gracefully fall back to `json` if `orjson` is unavailable.

### Logging
- Control via env vars:
  - `VENV_MUX_LOG=DEBUG` (or INFO/WARN/ERROR)
  - `VENV_MUX_LOG_JSON=1` to emit JSON logs
  - `VENV_MUX_LOG_FORMAT` to customize text logs

### Errors
- Calls raise subclasses of `CallError` with a structured `code` and optional `traceback`:
  - `NotFoundError`: function not found in worker module
  - `InvalidJsonError`: worker received invalid JSON input
  - `RemoteExceptionError`: remote function raised; includes traceback

### Protocol compatibility
- venvmux uses a simple JSONL protocol with a shared `PROTOCOL_VERSION`. The pool validates the worker's reported version during startup and raises a clear error if there is a mismatch.

#### Convenience: preload multiple environments
See Quickstart above for the preferred multi-env usage pattern.

### Environment resolution
 - Priority: pyproject.toml (uv.lock + uv sync if available) > requirements files > inline packages
 - Venv path: `~/.venvmux/envs/{hash}/`
 - Hash inputs: interpreter/version, files content, inline packages, platform/arch
 - Also includes `PIP_INDEX_URL`, `PIP_EXTRA_INDEX_URL`, and `UV_INDEX` if set

#### Home and cache
- Default home: `~/.venvmux` (envs under `~/.venvmux/envs/{hash}`)
- Override via `Pool(home="/path/to/.venvmux")` or environment variable `VENV_MUX_HOME`
- Cleanup: stop the pool, then you can safely remove stale env directories under the home

### Worker import paths
- If your worker module is not importable by default, set `EnvSpec(worker_paths=["/abs/path/to/src"])` to extend the worker's `PYTHONPATH`.
- Security note: worker paths are trusted; only point to directories you control.

### Public API
- `EnvSpec`
- `Pool`
- `VenvHandle`

 

### Security
 - For trusted code only; no sandboxing.

### Troubleshooting
- If `uv` not available, the library uses `pip`.
 - Set `VENV_MUX_LOG=DEBUG` for verbose logs.
 - If using custom indices, note hashing includes index URLs; changing them creates a new env.

### Tests
- Unit tests mock installers and rely on the local interpreter.

### Examples
- See `examples/two_envs_demo.py`
- See `examples/typed_remote_demo.py`

