<div align="center">


# An À La Carte Data Model — from Dataclass to Symbolic Programs

[**Documentation**](https://free-solo.github.io/pygx/)
| [**Why PyGX**](#why-pygx)
| [**Performance**](#performance)
| [**When to use**](#when-pygx-is-useful)
| [**Installation**](#install)
| [**What's in the box**](#whats-in-the-box)

</div>

## Why PyGX

**PyGX is a data model with a slider.** At one end it is a plain, typed,
*fast* dataclass; at the other it is a full symbolic object model that
algorithms can inspect, rewrite, and search over. Between the two you slide
notch by notch — and the capabilities are **à la carte**: order exactly the
ones you need, and what you don't order costs nothing. Every capability you
don't turn on is structurally absent, not a skipped branch, and the hot paths
run in a native Rust core that keeps the bottom of the slider at
dataclass-class performance while beating Pydantic v2 on construction (see
[Performance](#performance)).

> [!IMPORTANT]
> **Breaking change (pre-1.0): the `pg.Object` default flipped from
> `sym=True` to `sym=False`.** A plain `class C(pg.Object)` is now a flat,
> validated dataclass — construction and assignment still validate, JSON
> round-trip / equality / hashing / diffing / hooks all still work — but it
> is **not** a symbolic-tree node. The semantic change to watch for: nested
> assignment of an already-parented object used to **clone** it into the new
> location (the single-parent invariant); at the new default it holds a
> plain **reference**, and tree operations (`sym_rebind`, `sym_setparent`)
> raise `SymbolicModeError`. Add `sym=True` to your class statement to get
> the previous behavior — it inherits to subclasses. PyGX's own tree-based
> classes (functors, hyper primitives, DNA, ...) pin `sym=True` internally
> and are unaffected.

### Notch 0 — a typed struct, at dataclass speed

Declare fields with annotations, exactly as you would with `@dataclass` or
Pydantic. This is the default posture — plain reference semantics: no tree,
no tracking, just a validated value object:

```python
import pygx as pg

class Rect(pg.Object):
    width: int
    height: int

r = Rect(width=2, height=3)          # ~290 ns — faster than pydantic v2
Rect(width="2", height=3)            # TypeError — validated at construction
pg.eq(r, Rect(width=2, height=3))    # True — value equality
r.to_json_str()                      # '{"_type": "...Rect", "width": 2, "height": 3}'
```

Equality, hashing, cloning, diffing, readable repr, and typed JSON
round-trip all come with the struct — no boilerplate, no separate schema
language.

### Notch 1 — richer validation, same struct

When an annotation can't express the constraint, attach a value spec. It is
enforced at construction *and* on every later assignment:

```python
from typing import Annotated

class Rect(pg.Object):
    width: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]
    height: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]

Rect(width=-1, height=3)   # ValueError: Value -1 is out of range (min=1).
```

### Notch 2 — the symbolic tree

Opt in with `sym=True` and the construction arguments are no longer thrown
away: the object becomes a node in a mutable, parent-aware, self-validating
**tree**. Nested objects know their path; assigning an object into a tree
*adopts* it (cloning it if it already has a parent, preserving the
single-parent invariant); mutation re-validates and fires change events;
objects can be partial, sealed, or contextual. `sym=True` inherits to
subclasses, so one symbolic base is enough for a whole hierarchy:

```python
class Rect(pg.Object, sym=True):
    width: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]
    height: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]

    def on_sym_ready(self):              # derived state, recomputed on change
        super().on_sym_ready()
        self.area = self.width * self.height

r = Rect(width=2, height=3)              # r.area == 6
r.sym_rebind(width=5)                     # in-place, re-validated, area -> 15

p = Rect.partial(width=2)                 # late binding: height unbound
p.sym_rebind(height=3)                    # now complete -> on_sym_ready fires
Rect(width=2, height=3).sym_seal()        # immutable from here on

cfg = pg.Dict(a=r, b=Rect(width=4, height=5))
cfg.a.sym_path                            # 'a' — every node knows where it lives
cfg.sym_rebind({'a.height': 6})           # deep-path edits, batched + one notify
pg.query(cfg, where=lambda v: isinstance(v, Rect))
```

### Notch 3 — symbolic programs

Because the object *is* a tree, programs become data. Swap any field for a
*space* of values and enumerate, sample, or evolve it; patch trees
declaratively; detour construction in code you don't own; bind functions
partially and rewrite the binding later:

```python
space = Rect(width=pg.oneof([2, 4]), height=pg.oneof([3, 5]))
list(pg.iter(space))                      # 4 concrete Rects
pg.patch_on_key(cfg, "width", 1)          # every width in the tree -> 1

@pg.functor()
def scale(rect, factor=2):
    return rect.area * factor

f = scale(factor=3)                       # a symbolic, rebind-able call
f.sym_rebind(factor=10)
```

This is the notch AutoML, program search, and evolutionary computation live
on — and it is the same object model, not a separate spec language.

### Capabilities are à la carte

Each capability is a class keyword, orthogonal to the rest — `sym`,
`slots` (inline-slot storage for memory-critical classes), `frozen`, `eq`,
`order`, `validate`, `attr_read` / `attr_write` — and an option you don't
enable costs nothing at runtime: no tree state on a `sym=False` (the
default) leaf, no
validation machinery on `validate=False` fields, no per-instance `__dict__`
on a `slots=True` record. Context-aware fields (`pg.ContextualObject`),
URI-based construction (`pg.from_uri`), and schema-driven meta-programming
(every class carries a queryable `__schema__`) layer on the same model.

### More meta-programming toolkits

Beyond the data model, PyGX treats program construction itself as something you
can reach into:

- **Functors — advanced function binding.** `pg.functor` turns a function into a
  symbolic, partially-bindable call: bind some arguments now, supply or `sym_rebind`
  the rest later, and inspect or serialize the call before it ever runs.

  ```python
  @pg.functor()
  def scale(rect, factor=2):
      return rect.width * rect.height * factor

  f = scale(factor=3)              # bind factor now; rect supplied at call time
  f(Rect(width=2, height=3))       # 18
  f.sym_rebind(factor=10)              # sym_rebind a bound argument later
  ```
- **Detour the logic of an existing class.** `pg.detour` redirects construction
  to another class inside a context manager — intercept object creation in code
  you don't own. (`pg.symbolize` / `pg.wrap` / `pg.wrap_module` similarly opt
  plain classes, functions, and whole modules into the symbolic world without a
  rewrite.)

  ```python
  with pg.detour([(ResNet, MobileNet)]):
      model = build_model()          # internal ResNet(...) now yields a MobileNet
  ```
- **Patch existing objects.** `pg.patch`, `pg.patcher`, and the `pg.patch_on_*`
  family compose declarative rewrites by key, path, value, type, or member.

  ```python
  cfg = pg.Dict(a=Rect(width=2, height=3), b=Rect(width=4, height=5))
  pg.patch_on_key(cfg, "width", 1)   # every width in the tree -> 1
  ```

### Handy utilities

- **Portable logging, metrics, and IO.** Project-scoped logging (`pg.logging`),
  metric collection (`pg.monitoring`), and lightweight timing (`pg.timeit`),
  plus an fsspec-backed IO layer so `pg.save` / `pg.load` / `pg.open_jsonl`
  work transparently over local, GCS, S3, and friends — rounded out by parallel
  `pg.concurrent.execute` / `map` (retries, timeouts, progress bar) and
  `pg.reload`, which follows the module graph instead of leaving stale
  references behind.

  ```python
  pg.concurrent.execute(lambda r: r.area, [Rect(width=2, height=3)], max_workers=8)
  pg.save(Rect(width=2, height=3), "gs://bucket/rect.json")   # local, GCS, S3, ...
  ```
- **Code generation with permission control.** `pg.coding` compiles and runs
  Python source under a declared permission set, with optional subprocess
  sandboxing.

  ```python
  pg.coding.evaluate("1 + 1")        # 2
  pg.coding.evaluate("import os", permission=pg.coding.CodePermission.BASIC)
  # CodeError: `import` is not allowed.
  ```

## Performance

The symbolic model used to be a tax; it isn't anymore. PyGX's hot paths —
construction, attribute access, storage, and per-node state — run in a
native Rust core (`pygx-core`, installed automatically), while the
pure-Python implementation remains the executable specification: the full
test suite runs against **both** cores in CI, so they cannot diverge.

Median ns/op on a 3-field object (Apple Silicon; read ratios, not
absolutes — the [full report](docs/reports/perf.md) covers ~50 operations
across scales):

| operation | pygx `sym=False` (default) | pygx `sym=True` | `@dataclass` | pydantic v2 |
| --- | --- | --- | --- | --- |
| construct (kwargs) | 291 | 274 | 189 | 508 |
| attr get | 43 | 44 | 39 | 45 |
| clone (deep) | 808 | 813 | 2,210 | 1,620 |
| to dict/json | 379 | 404 | 652 | 527 |
| from dict/json | 634 | 622 | 200 | 659 |

Construction is ~1.8× faster than Pydantic v2 *with the whole symbolic
model attached*; attribute reads are at dataclass speed; deep clone and
serialization beat both. (A few cold ops — `eq`, `hash` — still trail a
bare dataclass; the report tracks them honestly.) Wheels ship for Linux
(glibc + musl, x86_64 + aarch64), macOS (Intel + Apple Silicon), and
Windows on CPython 3.12–3.14 — plus, from pygx-core ≥ 0.1.2, a wheel
for the free-threaded 3.14t build that is genuinely free-threaded (the
GIL stays off; crash-freedom and per-operation — per-node, per-field —
atomicity per the threading contract in `docs/design/gil-free.md` §3); on anything else PyGX
falls back to the pure-Python core with identical behavior.

## When PyGX is useful

Reach for PyGX when your problem has any of these shapes:

- **You need extensive Python meta-programming.** Symbolizing existing
  classes, functions, or whole modules; redirecting `__new__` with
  `pg.detour`; reloading transitive module graphs with `pg.reload`;
  controlled `parse` / `evaluate` / `run` of generated source — PyGX
  treats program construction itself as a first-class activity.
- **You need a mutable data model.** Symbolic objects are deeply
  mutable, parent-aware, and self-validating. `sym_rebind` propagates change
  events up and down the tree; `on_sym_change` / `on_sym_ready` hooks let
  derived state recompute incrementally; partial / late binding,
  declarative patching, and deep diffing fall out for free.
- **You need to manipulate objects with algorithms.** Once a program is a
  symbolic tree, algorithms can operate on it directly: enumerate or
  sample a search space (`pg.iter`, `pg.random_sample`); drive
  distributed search (`pg.sample`); evolve populations with composable
  selectors and mutators (`pg.algo.evolution`); synthesize functions
  from scratch (`pg.algo.mutfun`); or just `pg.traverse` / `pg.query` /
  `pg.transform` over the tree yourself.

Concrete domains where these properties cash out:

- **Configurable, composable applications.** Replace ad-hoc config systems
  (YAML/JSON/argparse glue) with typed, validated, serializable Python
  objects that *are* the configuration.
- **Machine learning at team scale.** Models, pipelines, and experiments
  expressed as symbolic trees are easy to log, diff, share, and recombine —
  with no separate spec language.
- **AutoML, program search, and evolutionary computing.** Drop search into an
  existing Python program by replacing scalars with hyper primitives, then reach
  for `pg.sample` + `pg.algo.evolution` to drive distributed search. Populations
  are just lists of symbolic programs; mutations are direct rewrites; selectors
  operate on object metadata.
- **Daily Python work.** Late/partial binding, deep direct manipulation,
  context-aware components, declarative patching, and an interactive HTML
  inspector — useful well outside ML.

## Install

```bash
pip install pygx
```

Nightly build:

```bash
pip install pygx --pre
```

Optional extras:

```bash
pip install "pygx[io]"          # fsspec-backed remote IO (GCS, S3, ...)
pip install "pygx[concurrent]"  # parallel execute/map with retries + progress
```

Requires Python 3.12+.

## What's in the box

PyGX is a small core plus focused sub-packages — and every one of them is a
consumer of the *same* symbolic model: `pg.hyper` turns fields into search
spaces, `pg.geno`/`pg.tuning` drive distributed search over them,
`pg.algo.evolution` mutates and recombines symbolic programs directly,
`pg.patching`/`pg.detouring` rewrite trees and construction, and `pg.views`
renders any node as an interactive HTML tree. Nothing downstream needs a
second representation of your program. The most common entry points are
re-exported at the top level as `pg.*`.

### Symbolic object model — `pg.symbolic`

- `pg.Object` — schema-driven, dataclass-like classes with a synthesized
  keyword-only `__init__`, PEP 681 `dataclass_transform` support
  (pyright/Pylance/mypy understand it), and runtime field validation.
- `pg.Dict`, `pg.List` — mutable symbolic containers that participate in the
  same tree as `pg.Object`.
- `pg.symbolize`, `pg.functor`, `pg.wrap`, `pg.wrap_module` — opt existing
  classes, functions, or whole modules into the symbolic world without
  rewriting them.
- `pg.field` — dataclass-style field descriptor for defaults, factories,
  per-field flags (`init=False`, `repr=False`, `compare=False`, ...), and
  inline value-spec overrides.
- Symbolic operations: `pg.eq`, `pg.ne`, `pg.lt`, `pg.gt`, `pg.hash`,
  `pg.clone`, `pg.diff`, `pg.traverse`, `pg.query`, `pg.contains`.
- Serialization: `pg.to_json` / `pg.from_json` (and `_str` variants),
  `pg.save` / `pg.load`, `pg.open_jsonl`.
- Lifecycle hooks: `on_sym_ready`, `on_sym_bound`, `on_sym_change`, `on_sym_parent_change`,
  `on_sym_path_change` for incremental update and parent-aware behavior.
- Per-class behavioral knobs via class-statement kwargs (immutability,
  freezing, equality semantics, registry membership).

### Runtime typing — `pg.typing`

- `pg.typing.Int`, `Float`, `Str`, `List`, `Dict`, `Object`, `Union`,
  `Callable`, ... — composable value specifications.
- Bounds, regex, default values, custom converters via
  `pg.typing.register_converter`.
- Annotations are automatically lowered to value specs, so the common case
  needs no explicit `pg.typing.*` usage. Reach for it when you need
  constraints the annotation can't express.

### Search & program generation — `pg.hyper`, `pg.geno`, `pg.tuning`

- **Hyper primitives** turn any symbolic value into a *space* of values:
  `pg.oneof`, `pg.manyof`, `pg.permutate`, `pg.floatv`, `pg.evolve`.
- **Genotypes (`DNA`/`DNASpec`)** decouple the algorithms that *generate*
  programs from the user code that *consumes* them.
- `pg.iter`, `pg.random_sample`, `pg.materialize` — local enumeration and
  sampling.
- `pg.sample` — distributed search with a pluggable backend, work groups,
  failover, and result polling.

### Algorithms — `pg.algo`

- `pg.algo.evolution` — compositional evolutionary algorithms built from
  selectors, mutators, and recombinators wired with the `>>` operator.
  Ready-made constructors: `hill_climb`, `neat`, `nsga2`,
  `regularized_evolution`.
- `pg.algo.mutfun` — mutable symbolic programs (`Function`, `Assign`,
  `Var`, ...) for symbolic regression and from-scratch program synthesis.
- `pg.algo.scalars` — scalar schedules.
- `pg.algo.early_stopping` — early-stopping policies for tuning.

### Detour and patching — `pg.detouring`, `pg.patching`

- `pg.detour([(A, B)])` — redirect `A.__new__` to `B.__new__` inside a
  context manager. Works on *non-symbolic* classes; useful for
  intercepting object creation inside code you don't own.
- `pg.patch`, `pg.patcher`, `pg.patch_on_key` / `_path` / `_value` / `_type`
  / `_member` — declarative, composable rewrites over a symbolic tree.
- `pg.from_uri` — build (or patch) symbolic objects from URI-like strings.

### Context-aware components — `pg.contextual`

- Thread-local stack of variable bindings installed by
  `pg.contextual.override` and read via `pg.contextual.get`.
- `pg.ContextualObject` — symbolic fields that resolve from the surrounding
  context when unset, propagating into child threads via
  `pg.contextual.with_override`.

### Visualization — `pg.views`

- `pg.view`, `pg.view_options` — pluggable rendering for symbolic values.
- `pg.to_html` / `pg.to_html_str` — self-contained, interactive HTML
  tree-view of any symbolic object (rendered inline in Jupyter or saved
  to disk).
- `pg.controls` — small interactive widgets composable into HTML views.

### Concurrent execution — `pg.concurrent`

- `pg.concurrent.execute` / `execute_async` — parallel map preserving input
  order, with per-item and overall timeouts.
- `pg.concurrent.map` — lazy `(input, output, error)` stream with a
  thread-safe progress bar.
- `pg.concurrent.with_retry` / `with_retry_async` — bounded retries with
  optional exponential backoff.
- `ExecutorPool` keyed by resource id so global rate limits are honored
  across nested calls.
- Sync/async bridging that propagates `pg.contextual` overrides across the
  boundary.

### Code generation — `pg.coding`

- `pg.coding.parse` / `evaluate` / `run` — compile and run Python source
  under a declared permission set (assignment, function definition,
  imports, ...).
- `pg.coding.make_function` — synthesize a callable from a source string.
- `pg.coding.sandbox_call` / `maybe_sandbox_call` — run a callable in a
  child process with a timeout.

### Pluggable IO — `pg.io`

- Abstract `FileSystem` / `File` / `Sequence` interfaces.
- In-memory backend for tests, plus an
  [`fsspec`](https://filesystem-spec.readthedocs.io/) bridge that picks up
  any registered filesystem (local, GCS, S3, ...).
- Used by `pg.save`, `pg.load`, and `pg.open_jsonl`.

### Instrumentation — `pg.instrument`

- `pg.logging` — project-scoped logger.
- `pg.monitoring` — metric collection.
- `pg.timeit` — lightweight scoped timing.

### Errors and topology — `pg.error_handling`, `pg.topology`, `pg.formatting`

- `pg.catch_errors`, `pg.match_error`, `pg.ErrorInfo` — pattern-matched
  exception suppression and structured, JSON-serializable error capture.
- `pg.KeyPath`, `pg.MISSING_VALUE`, `pg.traverse`, `pg.JSONConvertible` —
  the tree/value vocabulary the rest of the library is built on.
- `pg.format`, `pg.print`, `pg.colored` — readable, optionally-colored
  rendering of symbolic values.

### Importing — `pg.importing`

- `pg.reload` — module reload that follows dependency edges, refreshing
  transitively-affected modules instead of leaving stale references behind.

## Background

PyGX was originally built at **Google Brain / DeepMind** by Daiyi Peng to power automated machine
learning (AutoML), under the name [**PyGlove**](https://github.com/google/pyglove).
The abstraction underneath — **symbolic object-oriented programming (SOOP)** — turned out to be
much more general than AutoML, and PyGlove grew into a toolkit for advanced Python
programming used well beyond ML. The original PyGlove paper was
[published](https://proceedings.neurips.cc/paper/2020/file/012a91467f210472fab4e11359bbfef6-Paper.pdf)
at NeurIPS 2020.

## Documentation & examples

- [Bird's-eye view of PyGX](https://free-solo.github.io/pygx/notebooks/intro/birdview/)
  — 5–10 minute tour of the core ideas.
- [Authoring `pg.Object` subclasses](docs/guide/general/pg_object_style.md) —
  style guide for positional `__init__`, field defaults, and `ClassVar` config.
- [User Guide](https://free-solo.github.io/pygx/guide/) — topic-specific
  tracks for Python, ML, AutoML, and Evolution.
- [Learning PyGX](https://free-solo.github.io/pygx/learn/) — conceptual
  material on Symbolic Object-Oriented Programming and Symbolic Detour.
- [API Reference](https://free-solo.github.io/pygx/api/) — generated from
  source.

Runnable notebooks live in [`docs/notebooks/`](docs/notebooks/) and source
examples in [`examples/`](examples/).

## Citing PyGX

```
@inproceedings{peng2020pyglove,
  title={PyGlove: Symbolic programming for automated machine learning},
  author={Peng, Daiyi and Dong, Xuanyi and Real, Esteban and Tan, Mingxing and Lu, Yifeng and Bender, Gabriel and Liu, Hanxiao and Kraft, Adam and Liang, Chen and Le, Quoc},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={33},
  pages={96--108},
  year={2020}
}
```

## License

Apache License 2.0. PyGX is derived from PyGlove (also Apache 2.0); see
[`LICENSE`](LICENSE) and the per-file copyright headers for attribution.

PyGX is developed by Daiyi Peng.
