Metadata-Version: 2.4
Name: emfc
Version: 0.1.0rc1
Summary: CUDA-native EM field compression and self-contained simulation scene storage
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://witwin.ai
Project-URL: License, https://witwin.ai/license
Keywords: cuda,electromagnetics,compression,pytorch,maxwell,emfc
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Microsoft :: Windows :: Windows 10
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.10
Requires-Dist: numpy>=1.26
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# EMFC

`emfc` reads and writes complex electromagnetic fields directly as CUDA PyTorch
tensors. `.emfc` (Electromagnetic Field Container) files support bit-exact or encoder-verified
relative-L2 field compression plus bit-exact, self-contained simulation scenes.

The package has no CPU field codec and never compiles code at runtime. Install a precompiled wheel
matching the supported Windows/Linux, CUDA, and GPU architecture matrix.

## Quick start

```python
import emfc

# One call returns the primary field on the requested CUDA device.
field = emfc.load("field.emfc", device="cuda:0")

# Omit tolerance for bit-exact primary-field storage.
emfc.save("field-lossless.emfc", field)

# Or request a verified global relative-L2 bound for the primary field.
emfc.save("field-1pct.emfc", field, tolerance=0.01)

# The same 1% error bound expressed in decibels: 20 * log10(0.01) = -40 dB.
emfc.save("field-minus-40db.emfc", field, tolerance_db=-40)

# Also verify 0.1% complex error independently in each 20 dB band from -20 to -100 dB.
emfc.save(
    "field-weak-preserving.emfc", field,
    tolerance=0.001,
    preserve_weak_fields=True,
)
```

Paths and binary file objects are accepted. The `.emfc` suffix is conventional; the EMFC magic
and exact `1.0` version stored in the file are authoritative. Bare private codec streams are not
accepted by this public file API.

## Containers without files

`encode` and `decode` are the same codec with the file step removed, for containers that live in
memory — an object store, a socket, a database column, or a dataset pipeline:

```python
# Every save() argument works here; the result is exactly the bytes save() would write.
blob = emfc.encode(field, tolerance_db=-40, scene=scene)

# Every load() argument except the file; bytes, bytearray, and memoryview are accepted.
field, scene = emfc.decode(blob, device="cuda:0", return_scene=True)

# Metadata without decoding any field or scene array.
info = emfc.inspect(blob)
```

Because `save` and `load` are thin wrappers over these two, containers cross freely between the
paths: `decode` reads a saved file's bytes and `load` reads encoded bytes. The tensor stays on its
CUDA device throughout; only the finished container is host memory.

## Complete simulation scenes

Use `SceneData` to store the complete compiled material fields and declarative simulation input.
The definition can be a WiTwin Maxwell `Scene`, a dataclass tree, or mappings/lists containing
analytic geometries, triangle meshes, sources, monitors, ports, boundaries, and grids.

```python
scene_data = emfc.SceneData.from_scene(
    scene,
    eps_r=eps_r,
    mu_r=mu_r,
    sigma_e=sigma_e,
    simulation=simulation,  # solver method and configuration, if separate from scene
    metadata={"run": "reference"},
)

emfc.save("result.emfc", electric, tolerance=1e-3, scene=scene_data)

electric, restored_scene = emfc.load(
    "result.emfc", device="cuda", return_scene=True
)
assert restored_scene is not None
eps_r = restored_scene.eps_r
```

`load()` reads the file once. It returns a Tensor by default and returns `(tensor, scene)` when
`return_scene=True`; the scene value is `None` for a container without one. `tolerance` and
`tolerance_db` apply only to the primary E/H field. Permittivity, permeability, conductivity, mesh
vertices/faces, custom grid coordinates, source datasets, and observables always retain their
original dtype and bytes. Scene arrays may use lossless Deflate compression and carry SHA-256
checksums.

Analytic geometry and other Python objects load as safe `ObjectRecord` trees containing their
qualified type names and state. The reader does not use pickle, dynamically import named modules,
or instantiate classes from an untrusted file.

An EMSample-shaped object with `relative_permittivity`, `relative_permeability`, grid,
boundary, sources, and material table attributes can be passed directly as `scene=`.

## Shard datasets for training

`emfc.shard` packs complete containers into fixed-volume `.emfcs` shards (4 GiB by default), so a
training job can plan an epoch from one small index and fetch a sample with one read. Records are
stored and returned byte-identically — the shard layer never re-encodes — and samples that share a
scene store its arrays once per shard, when those samples land in the same shard and the shared
suffix is at least `dedup_min_bytes` (64 KiB).

```python
from emfc.shard import ShardWriter, index, loader, verify

# Write: encode straight into shards, or append already-encoded containers with write_bytes().
with ShardWriter("dataset/", target_bytes=1 << 30) as writer:
    for sample_id, field in enumerate(fields):
        writer.write(field, sample_id=sample_id, tolerance_db=-40, scene=scene_data)

for path in writer.sealed_shards:
    verify(path)          # crc32 of every reconstructed container, checksums, sidecars

# Index once, after every writer has finished: merges the per-shard sidecars into
# dataset.json/dataset.idx without opening a single shard.
dataset = index("dataset/")
print(len(dataset), "records in", len(dataset.shards), "shards")

# Train: workers read bytes, this process decodes on a private CUDA stream.
for batch in loader("dataset/", batch_size=8, num_workers=2, device="cuda", epoch=0):
    fields = batch.fields          # (B, C, X, Y, Z) complex64 CUDA tensor when shapes agree
    ...
```

`ShardReader(path).read(i)` returns exactly the bytes `emfc.encode` produced, so
`emfc.shard.extract(path, i, "sample.emfc")` writes an ordinary `.emfc` file. `emfc.shard.inspect`
summarises a shard without decoding, and `pack`/`repack` import `.emfc` files or re-shard at a new
volume, both with zero re-encode. There is no CPU codec, so a DataLoader worker must never decode;
`loader` enforces that split. See [docs/SHARD.md](docs/SHARD.md) for the byte-level format.

A dataset can be grown incrementally and safely. A later session resumes the sample-id numbering
instead of restarting it, several ranks fan out with a stride, a crashed writer's `.emfcs.tmp` is
salvaged with `recover`, and the daily small shards are consolidated with `compact` — none of it
re-encodes a container:

```python
from emfc.shard import ShardWriter, index, recover, compact

# A later session continues past the sample ids already in the dataset.
with ShardWriter("dataset/", resume=True) as writer:
    for field in todays_fields:
        writer.write(field, tolerance_db=-40, scene=scene_data)

# Several writers fan out with distinct ranks; rank r emits ids r, r + world, r + 2*world, …
with ShardWriter("dataset/", rank=r, sample_id_stride=world) as writer:
    ...

dataset = index("dataset/")        # refuses a dataset that reissued a sample id; ignores .emfcs.tmp
print(dataset.sample_id_max, dataset.shard_count, dataset.pending)

for tmp in index("dataset/").pending:
    recover(tmp)                    # rebuild a sealed shard from a crash's unsealed .emfcs.tmp

compact("dataset/")                 # merge under-filled shards in place, then re-run index()
```

`index` is the dataset-level safety net: every record's `sample_id` is its identity, and a session
that forgot `resume=True` and reissued ids is refused loudly rather than trained on.

Durability is a deliberate trade-off. A writer buffers appends in `buffer_bytes` (64 MiB by
default) and only sealed shards are on disk, so a hard crash — `SIGKILL`, power loss — loses the
records still in that buffer; `recover` salvages only what already reached the `.emfcs.tmp`. Lower
`buffer_bytes` to flush more often (down to one write per record) when the tail of an interrupted
session must survive, and pay the extra I/O; keep it large for throughput when losing the current
batch on a crash is acceptable. A crash never corrupts a sealed shard either way. See
[docs/SHARD.md](docs/SHARD.md) section 10 for the incremental workflow, concurrency, and crash
semantics.

## Binary file objects

Paths are the shortest form, while already-open binary files work as expected:

```python
with open("result.emfc", "wb") as file:
    emfc.save(file, electric, tolerance_db=-60, scene=scene_data)

with open("result.emfc", "rb") as file:
    electric, restored_scene = emfc.load(
        file, device="cuda:0", return_scene=True
    )
```

## Install

Install a CUDA-compatible PyTorch build first, followed by the tested wheel:

```powershell
python -m pip install "torch>=2.10" --index-url https://download.pytorch.org/whl/cu128
python -m pip install emfc-0.1.0rc1-cp310-abi3-win_amd64.whl
```

Release wheels target Windows and manylinux x86-64, CPython 3.10+, Torch 2.10+, CUDA 12.8, and the
GPU architectures embedded by the release workflow. A missing or incompatible `_C` extension
raises an installation error; it never falls back to Python or invokes a compiler.

## Stable Python API

```python
emfc.load(file, *, device="cuda", return_scene=False) -> torch.Tensor
emfc.load(file, *, device="cuda", return_scene=True) -> tuple[torch.Tensor, SceneData | None]
emfc.save(
    file, tensor, *, tolerance=None, tolerance_db=None, scene=None,
    preserve_weak_fields=False, weak_tolerance=None, weak_floor_db=-100, weak_ceiling_db=-20,
) -> None
emfc.inspect(file) -> ContainerInfo
```

- `tensor` is complex CUDA `(C, X, Y)` or `(C, X, Y, Z)` data.
- `tolerance=None` preserves complex64 or complex128 IEEE words exactly.
- A finite `tolerance` in `[0, 1)` selects the verified lossy frontier for complex64 input.
- A finite negative `tolerance_db` is converted with `10 ** (tolerance_db / 20)`; for example,
  `-40` dB is `0.01`. Pass only one tolerance form.
- `preserve_weak_fields=True` additionally verifies complex relative-L2 in each 20 dB
  GT-relative amplitude band from `weak_ceiling_db` down to `weak_floor_db`; `weak_tolerance`
  defaults to the global tolerance. Bands already within tolerance cost nothing; failed bands
  use complete-byte selection over structured Tucker and exact-support point repairs. The opt-in
  stream retains weak-field amplitude, phase, and support.
- `SceneData` uses the conventional material names `eps_r`, `mu_r`, `sigma_e`, and `sigma_m`.
- `inspect()` reads metadata without decoding field or scene tensors.
- `emfc.__version__` reports the encoder package version (`0.1.0rc1`).

Low-level codec and stream classes remain public for stream accounting, certificates, and
lossless square/cubic power-of-two patch decode.

## Format and execution guarantees

- EMFC stores format version `1.0` in both its fixed binary header and manifest, and records the
  encoder package version (`0.1.0rc1`) in the manifest.
- Its lossless/lossy primary-field stream is a private payload, alongside named bit-exact scene
  arrays.
- Lossless field streams use independent `16^3` blocks (`16x16x1` for direct 2D), checksums, and
  complete-container MDL selection over raw, rANS, byte shuffle, Lorenzo, PCIX, and DESL.
- Lossy field streams select the smallest verified native SVDQ/Tucker stream with raw fallback.
- The public file API accepts complete EMFC 1.0 files only.
- Weak-preserving lossy streams independently repair only failed log-amplitude bands, selecting
  complete bytes over masked Tucker and group-quantized point repairs with indexed or bit-packed
  support, then verify the real decode in every declared band.
- Field-sized encode/decode work stays in registered CUDA, cuBLAS, and cuSOLVER operators.
- No field or encoded payload tensor crosses to CPU during codec execution; explicit file and
  scene serialization are the host I/O boundary.

## Build and test

Building requires MSVC x64 or a compatible Linux C++ toolchain, CUDA Toolkit/NVCC, Ninja,
PyTorch 2.10, and an explicit architecture list. Runtime JIT builds are intentionally unsupported.

```powershell
$env:TORCH_CUDA_ARCH_LIST = "12.0"
python -m build --wheel --no-isolation
python -m pytest -q -p no:cacheprovider --basetemp .tmp/pytest-wheel
python -m ruff check src tests setup.py
python -m twine check dist/*.whl
```

The release workflow builds Windows and manylinux `cp310-abi3` wheels with CUDA 12.8.1 and
PyTorch 2.10, audits their native imports, and loads the identical binaries across Python
3.10–3.14 and PyTorch 2.10–2.12. It uploads GitHub Actions artifacts only; it does not publish
GitHub Releases or PyPI packages.


See [docs/FORMAT.md](docs/FORMAT.md), [docs/SHARD.md](docs/SHARD.md),
[CHANGELOG.md](CHANGELOG.md), and [LICENSE](LICENSE).
