Metadata-Version: 2.4
Name: pypersistable
Version: 1.0.2
Summary: Core Object Model
Author-email: Nick Young <nick.young.github@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Nick Young
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: homepage, https://github.com/nickyoung-github/persistable
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anyschema
Requires-Dist: duckdb
Requires-Dist: fsspec
Requires-Dist: narwhals
Requires-Dist: orjson
Requires-Dist: pyarrow
Requires-Dist: pydantic>2.0
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Dynamic: license-file

# pypersistable: persist python objects with minimal boilerplate

Any object that can be handled by `pydantic.TypeAdaptor` will work (pydantic models, dataclasses etc.)
Objects can be persisted as JSON or parquet (with some caveats).


## Defining persistable objects

### `TypeRegistry`
Persistable types must be registered with `TypeRegistry`. This can be done explicitly or using entry points.
While we could have avoided a registry, the alternative is persisting implementation-specific info,
such as the module and class names, making code refactoring very difficult.
You can optionally supply a `root` parameter to group types, e.g., you might specify a root of `Animals`
for classes `Dog`, `Cat`, `Bunyip` etc. See [DiscriminatedUnion](#discriminatedunion) for another use of root.

### `Persistable`
`Persistable` is the abstract base class for persistable objects. All it does it provide an `object_id`
for serialisation and ways to form that id for retrieval. There are two species of `Persistable`:

- `Unique` - As the name suggests, this is the base for types where all fields form the id. E.g.,
```python
class Address(Unique):
    street_number: str  # Could be 221B, for example
    street_name: str
    town: str
    postcode: str
    country: Country
```
- `Identifiable` - A base class for objects where a subset of fields form their id. Such fields
are declared using `IdField`. E.g.,
```python
class Customer(Identifiable):
    name: IdField[str]
    address: Address
```

### `Reference`
In classes which have user-defined types as attributes, do we want those nested objects to be persisted
by value in the parent object's payload or saved separately, with the parent persisting a reference?
The answer is very much, "it depends". If you definitely want the nested payload to be embedded, that is
easy: simply declare your type like `Customer` above. However, you could instead re-write it as:

```python
class Customer(Identifiable):
    name: IdField[str]
    address: Reference[Address]
```

This means that if we had multiple customers (e.g. family members) living at the same address,
when persisting those customers, the `Address` would be persisted once and shared, with a reference being
saved in the `Customer` payload.

You can force references to be serialised by value via the `by_reference` paramter on `ObjectStore.write`.
`ObjectStore.read` will always check for persisted references.

### `DiscriminatedUnion`
>`DiscriminatedUnion` works on all types, they do not need to be Persistable

Polymorphism in persisted classes tends to work quite badly and should be avoided. Persistable objects
often also get used in contexts where we need to know a schema, such as web services.
`fastapi` based services, using pydantic models is a good example.

A better approach is to declare the valid types using a python `Union`. pydantic can try and infer the
actual type on deserialisation but it can be preferable to indicate that explicitly. I got tired of trying to use
half-baked implementations from Stack Overflow so I wrote `DiscriminatedUnion`, which does this by:
- Adding a type specifier to the JSON schema 
- Adding that type specifier to the serialisation paylod (the type's module and qualified name - no need for the type registry as the union tells us the type)
- Extracting and removing the type specifier from the deserialisation payload and selecting the appropriate type

`DiscriminatedUnion` can be combined with `__subclasses__` to implement a rather hacky polymorphism, with two
major caveats:
1. The static type checkers like pyright do not like it as the type is determined at runtime
2. You need to ensure that all subclasses or your base have been imported

For `Persistable` types `TypeRegistry.types_for_root` can help with 2., e.g.:

```python
Animals = DiscriminatedUnion[TypeRegistry.types_for_root("Animals")]
```

## Storing persistable objects

`ObjectStore` is an abstract base class for implementing an object store, such as a database.
A file system-based store `FSStore` is provided. Objects are written into directories, formed
from the store's root, the `TypeRegistry` root and registered name.
Each write creates a new file, whose name is the timestamp of the write, which corresponds to
`MetaData.entry_time` in the persisted record.

Reading uses `duckdb` to read the stored parquet/json files. Currently only querying by an object's id
is supported, but this approach also allows for querying by attributes of the object in future.

A nightly job to compact the timestamped files into a single file will improve read performance.

`FSStore` uses [fsspec](https://pypi.org/project/fsspec/), which supports a wide range of file systems.
For parquet schema generation it uses [anyschema](https://pypi.org/project/anyschema/), extended with
some handling for dataclasses and references.

> [Note]
> Parquet cannot handle union types natively. Currently they are only support via a `Reference`, e.g.:

```python
class Foo:
    animal: Reference[Union[Dog, Cat, Wildebeest]]
```

### Writing

```python
store = FSStore(r"file:///users/hades/ObjectStore", format=FSStoreFormat.Parquet)
best_boy = Dog(name="Cerberus")
store.write([best_boy])
```

Will result in:

```
/users/hades/ObjectStore                               # FSStore root
    /Animals                                           # TypeRegistry root
        /Dog                                           # TypeRegistry registered_name
            /2026-11-08T13:01:55.249821+00:00.parquet  # File name
```

If you save an object which contains references, multiple objects may be created:

```python
blofeld = Customer(
    name="Blofeld, Ernst S.",
    address=Address(
        street_numer=1,
        street_name="The Hollowed Out Volcano",
        town="",
        postcode="HV1 1ZZ",
        country=Country.Krakatoa
    )
)
store.write([blofeld])
```

This would create:

```
/users/hades/ObjectStore                           # FSStore root
    /Customer                                      # TypeRegistry registered_name
        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
    /Address                                       # TypeRegistry registered_name
        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
```

#### Reading
Objects are read by creating an id with `make_object_id`:

```python
best_boy_id = Dog.make_object_id(name="Cerberus")
loaded = store.read([best_boy_id])
best_boy = loaded[best_boy_id]
```
