Metadata-Version: 2.5
Name: zerobucket
Version: 0.23.0
Summary: Database-native image storage. Your database. Your images. Zero buckets.
Project-URL: Homepage, https://github.com/KedarGhadyalji/ZeroBucket
Project-URL: Repository, https://github.com/KedarGhadyalji/ZeroBucket
Project-URL: Issues, https://github.com/KedarGhadyalji/ZeroBucket/issues
Project-URL: Changelog, https://github.com/KedarGhadyalji/ZeroBucket/blob/main/CHANGELOG.md
Author-email: Kedar Ghadyalji <your-email@example.com>
License-Expression: MIT
Keywords: bytea,image,postgresql,s3-alternative,storage
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Multimedia :: Graphics
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pillow>=10.0
Requires-Dist: psycopg-pool>=3.2
Requires-Dist: psycopg[binary]>=3.1
Provides-Extra: dev
Requires-Dist: aiomysql>=0.2; extra == 'dev'
Requires-Dist: aiosqlite>=0.19; extra == 'dev'
Requires-Dist: boto3>=1.34; extra == 'dev'
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: numpy>=1.26; extra == 'dev'
Requires-Dist: pillow-heif>=0.13; extra == 'dev'
Requires-Dist: pymysql<1.2.0,>=1.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: scikit-image>=0.22; extra == 'dev'
Provides-Extra: heic
Requires-Dist: pillow-heif>=0.13; extra == 'heic'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: mysql-async
Requires-Dist: aiomysql>=0.2; extra == 'mysql-async'
Requires-Dist: pymysql<1.2.0; extra == 'mysql-async'
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Provides-Extra: sqlite-async
Requires-Dist: aiosqlite>=0.19; extra == 'sqlite-async'
Description-Content-Type: text/markdown

# ZeroBucket

[![PyPI version](https://img.shields.io/pypi/v/zerobucket.svg)](https://pypi.org/project/zerobucket/)
[![Python versions](https://img.shields.io/pypi/pyversions/zerobucket.svg)](https://pypi.org/project/zerobucket/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/LICENSE)
[![CI](https://github.com/KedarGhadyalji/ZeroBucket/actions/workflows/ci.yml/badge.svg)](https://github.com/KedarGhadyalji/ZeroBucket/actions/workflows/ci.yml)

**Your database. Your images. Zero buckets.**

ZeroBucket is a database-native image storage library. It lets you store
and retrieve images using the PostgreSQL database you already have,
instead of standing up a separate object-storage service like S3.

```python
from zerobucket import ZeroBucket

images = ZeroBucket(database_url="postgresql://...")

image_id = images.put("avatar.jpg")

image = images.get(image_id)
print(image.mime_type)   # "image/jpeg"
print(image.size_bytes)  # 1116478
print(image.data)        # raw bytes, ready to serve
```

Full documentation, architecture notes, and benchmark results live in the
[GitHub repository](https://github.com/KedarGhadyalji/ZeroBucket).

## Installation

```bash
pip install zerobucket
```

Requires Python 3.10+ and PostgreSQL 13+ (uses `gen_random_uuid()`, built
in since Postgres 13).

## Quick reference

| Method                                                                                        | What it does                                                                                                                                                                                 |
| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `images.put(image, filename=None, optimize=False, max_width=None, format=None, quality=None)` | Validates, optionally optimizes, checksums, and stores an image. Accepts a file path, raw `bytes`, or a file-like object (including framework upload objects). Returns the new image's id.   |
| `images.get(image_id, context=None)`                                                          | Returns an `Image` (data, mime_type, filename, width, height, size_bytes, checksum_sha256). Raises `ImageNotFoundError` if missing, or `AccessDeniedError` if a `before_get` hook denies it. |
| `images.get_stream(image_id, chunk_size=1MB)`                                                 | Like `get()`, but returns an iterator of chunks instead of one `bytes` object -- avoids holding the full image in Python memory at once. Raises `ImageNotFoundError` if missing.             |
| `images.stream_to(image_id, destination, chunk_size=1MB)`                                     | Writes an image's bytes directly to `destination` (anything with `.write(bytes)`), chunk by chunk. Returns total bytes written.                                                              |
| `images.metadata(image_id)`                                                                   | Same fields as `get()` but without the raw bytes -- cheap existence/info check.                                                                                                              |
| `images.exists(image_id)`                                                                     | Returns `True`/`False`.                                                                                                                                                                      |
| `images.delete(image_id)`                                                                     | Deletes the image. Returns `True` if it existed.                                                                                                                                             |
| `images.close()`                                                                              | Releases database connections. `ZeroBucket` also works as a context manager.                                                                                                                 |

```python
from zerobucket import ZeroBucket, ImageNotFoundError, ImageValidationError

images = ZeroBucket(
    database_url="postgresql://user:pass@localhost/mydb",
    max_bytes=8 * 1024 * 1024,  # default: 8MB
)

image_id = images.put("photo.jpg")
image_id = images.put(open("photo.jpg", "rb"))
image_id = images.put(request.files["avatar"])  # framework upload objects

image = images.get(image_id)
info = images.metadata(image_id)

try:
    images.get("nonexistent-id")
except ImageNotFoundError:
    ...

try:
    images.put("not-actually-an-image.txt")
except ImageValidationError:
    ...
```

### Serving from a web API

```python
from flask import Flask, Response

app = Flask(__name__)
images = ZeroBucket(database_url=DATABASE_URL)

@app.route("/images/<image_id>")
def serve_image(image_id):
    image = images.get(image_id)
    return Response(image.data, mimetype=image.mime_type)
```

### Streaming reads for large files

```python
for chunk in images.get_stream(image_id, chunk_size=1024 * 1024):
    response.write(chunk)

# or:
total_bytes = images.stream_to(image_id, response, chunk_size=1024 * 1024)
```

Avoids holding the full image in Python memory at once (one chunk at a
time instead), via ranged `substring()` queries. This is a Python-side
memory optimization only -- Postgres still handles the full stored value
the way it always has, and this isn't HTTP range/partial-content
support. See the [full explanation](https://github.com/KedarGhadyalji/ZeroBucket#streaming-readswrites-for-large-files)
on GitHub, including the mid-stream-delete safety behavior and how
`put()` bounds memory for rejected oversized file-like uploads.

### Access control

```python
def before_get(image_id: str, context: dict | None) -> bool:
    return context is not None and owns(context["user_id"], image_id)

def before_put(context: dict | None) -> bool:
    return context is not None and context.get("user_id") is not None

images = ZeroBucket(
    database_url=DATABASE_URL,
    before_get=before_get,
    before_put=before_put,
)

images.get(image_id, context={"user_id": current_user.id})  # -> AccessDeniedError if denied
```

No built-in ownership/permissions model by default -- these hooks let
you plug in your own check. Denied calls raise `AccessDeniedError` and
never reach the database. A hook that raises fails _closed_ (the
exception propagates, never treated as an implicit allow) -- see the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#access-control)
on GitHub for exactly what's gated (`get`/`get_many`/`get_stream`/
`stream_to`/`metadata`, NOT `exists`) and the `put_many`/`get_many`
batch evaluation semantics.

### Object-storage tiering

```python
from zerobucket import ZeroBucket, ObjectStorage

store = ObjectStorage("my-bucket", region_name="us-east-1")  # bucket must already exist
images = ZeroBucket(database_url=DATABASE_URL, object_storage=store)

image_id = images.put("large_scan.jpg")
images.tier_to_object_storage(image_id)   # explicit -- put() never auto-tiers

image = images.get(image_id)              # works exactly the same, fetched from S3
images.delete(image_id)                   # cleans up both the Postgres row and the S3 object
```

Opt-in escape hatch for images that outgrow keeping everything in
Postgres -- doesn't change anything for images you don't tier. S3-
compatible via `boto3` (`pip install zerobucket[s3]`, not installed by
default). Explicit trigger only (no automatic size-based tiering in
`put()` yet), transparent reads (`get`/`get_many`/`get_stream`/
`stream_to`/`metadata`/`exists` all work identically either way), and
not available with `dedup=True` yet. The upload happens inside the same
DB transaction as the row update, so a failed upload leaves the row
completely untouched. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#object-storage-tiering)
on GitHub for the transactional-safety details and the row-lock tradeoff.

### Optimizing images (compression)

Off by default -- `put()` stores your exact input bytes unless you opt in:

```python
image_id = images.put(
    "photo.jpg",
    optimize=True,
    max_width=1600,      # downscale if wider, aspect ratio preserved
    format="webp",       # optional re-encode target: "jpeg", "png", "webp"
    quality=90,           # 1-100, JPEG/WebP only; omit for data-backed defaults
)
```

Quality defaults (JPEG=90, WebP=88) are backed by measured SSIM data
across multiple content types -- typical photos see 70-95% size
reduction with no visible quality loss. One thing this data caught:
**don't target `format="jpeg"` for flat/graphic content** (screenshots,
logos) -- it can make them larger, not smaller. See
[`COMPRESSION_RESULTS.md`](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/COMPRESSION_RESULTS.md)
on GitHub for the full methodology.

### Async support

```python
from zerobucket import AsyncZeroBucket

images = AsyncZeroBucket(database_url="postgresql://user:pass@localhost/mydb")
image_id = await images.put("photo.jpg")
image = await images.get(image_id)

stream = await images.get_stream(image_id)  # note the await
async for chunk in stream:
    ...

await images.close()  # or: async with AsyncZeroBucket(...) as images:
```

Built on psycopg3's own native async mode -- not the third-party
`asyncpg` package, despite that name having sat on the roadmap for a
while. Zero new dependencies. First-pass scope: core operations +
streaming reads, classic mode only (no `dedup=`, hooks, `on_operation`,
`optimize=`/`validator=`, `connection=`, or retry yet -- tracked
honestly, not silently missing). See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#async-support)
on GitHub, including why `get_stream()` needs an `await` before you can
iterate it, and a **Windows note**: psycopg3's async mode needs a
`SelectorEventLoop`, not the default `ProactorEventLoop` --
`AsyncZeroBucket` raises a clear error telling you how to fix this if
you hit it, instead of a confusing timeout.

### SQLite support (experimental, in progress)

```python
from zerobucket import ZeroBucket, SQLiteBackend

images = ZeroBucket(backend=SQLiteBackend("images.db"))
image_id = images.put("photo.jpg")
```

```python
from zerobucket import AsyncZeroBucket, AsyncSQLiteBackend  # pip install zerobucket[sqlite-async]

images = AsyncZeroBucket(backend=AsyncSQLiteBackend("images.db"))
```

Sync: core CRUD, `get_stream()`, `tier_to_object_storage()`, and
`dedup=True` all work. Async (`AsyncSQLiteBackend`): core CRUD +
`get_stream()`, classic mode only -- same scope as `AsyncPostgresBackend`,
kept consistent on purpose. Three behaviors genuinely differ across
backends (not just theoretically) -- tiering locks the whole database
file rather than one row; sync `get_stream()` survives a concurrent
delete mid-stream thanks to WAL-mode snapshot isolation, while async
`get_stream()` raises instead (no `blobopen()` in `aiosqlite`), matching
Postgres. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#sqlite-support)
on GitHub for why, and what's still missing.

### MySQL support

```python
from zerobucket import ZeroBucket, MySQLBackend  # pip install zerobucket[mysql]

images = ZeroBucket(backend=MySQLBackend("mysql://user:pass@localhost:3306/mydb"))
image_id = images.put("photo.jpg")
```

```python
from zerobucket import AsyncZeroBucket, AsyncMySQLBackend  # pip install zerobucket[mysql,mysql-async]

images = AsyncZeroBucket(backend=AsyncMySQLBackend("mysql://user:pass@localhost:3306/mydb"))
image_id = await images.put("photo.jpg")
```

Core CRUD, `get_stream()` (ranged `SUBSTRING()` queries),
`tier_to_object_storage()` (`object_storage=`, also needs
`zerobucket[s3]`), `dedup=True` (content-addressed storage with
reference counting), and now async support all work as of 0.23.0 --
`AsyncMySQLBackend` matches `AsyncPostgresBackend`/`AsyncSQLiteBackend`'s
scope (core CRUD + streaming, classic mode only). Connection pooling
(sync and async) is the remaining explicit follow-up work. No
`RETURNING` (real MySQL 8.0 doesn't support it on any statement,
unlike MariaDB 10.5+), so `delete_many()`/`tier_to_object_storage()`/
dedup's ref-count decrement use `SELECT ... FOR UPDATE` then the
following statement inside one transaction instead. A genuine
improvement over SQLite's tiering: InnoDB has real per-row locking, so
tiering one image never blocks writes to any other row (SQLite has to
lock the whole database file for the same guarantee). Dedup mode's
blob upsert uses MySQL's `ON DUPLICATE KEY UPDATE` (not Postgres's/
SQLite's `ON CONFLICT`) -- verified atomic under 20 concurrent
threads, not just assumed. `AsyncMySQLBackend` uses `aiomysql`, pinned
against `PyMySQL<1.2.0` in the `zerobucket[mysql-async]` extra to work
around a real, verified incompatibility in the (unmaintained) aiomysql
library with newer PyMySQL releases -- see the full explanation for
what was actually hit and how it was root-caused. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#mysql-support)
on GitHub for every verified design divergence.

## What it validates

- **Format**: JPEG, PNG, WebP built in, plus HEIC/HEIF (iPhone photos) via
  the optional `pip install zerobucket[heic]` extra -- detected from
  actual file content, never from filename extension or a client-supplied
  `Content-Type` header.
- **Corruption**: truncated or malformed images are decoded and rejected
  before they reach the database.
- **Decompression bombs**: a tiny compressed file that decodes to an
  enormous pixel grid is rejected, not silently allocated.
- **Size**: configurable via `max_bytes` (default 8MB) -- see the
  [benchmark results](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/RESULTS.md)
  for why.

## Transactions

By default, `put()`/`get()`/`delete()` each use their own independent
database connection -- **not** your application's own transaction, even
against the same database. Pass your own open `psycopg` connection via
`connection=` to make a write participate in your transaction (e.g. "user

- avatar, atomically, or neither"). See the
  [Transactions section](https://github.com/KedarGhadyalji/ZeroBucket#transactions)
  on GitHub for a worked example -- this was verified by direct experiment
  during development, not assumed.

## Batch operations

```python
results = images.put_many([open("a.jpg", "rb"), open("b.jpg", "rb")])
fetched = images.get_many([id1, id2])
deleted = images.delete_many([id1, id2])
```

Best-effort, not all-or-nothing -- check `.success`/`.error` per item.
`get_many`/`delete_many` are genuine single-query batch operations; see
the [full docs](https://github.com/KedarGhadyalji/ZeroBucket#batch-operations)
on GitHub for what's actually batched vs. still per-item.

## Retry behavior

Transient errors (connection drops, deadlocks, serialization failures)
are automatically retried with exponential backoff (`max_retries=3` by
default). **Important:** passing your own `connection=` disables
automatic retry for that call -- see the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#retry-behavior)
on GitHub for why that's a deliberate safety rule, not an oversight.

## Custom content types (PDFs and beyond)

```python
from zerobucket.validators.pdf import PDFValidator

doc_id = images.put(pdf_bytes, validator=PDFValidator())
doc = images.get(doc_id)  # no special handling needed -- ever
```

Everything (transactions, retry, batch ops) works identically regardless
of which validator produced a row. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#custom-content-types-pdfs-and-beyond)
on GitHub for why this is a pluggable hook rather than native PDF
support built into the core.

## CLI

```bash
zerobucket init      # create the schema if missing
zerobucket info      # image count, total size, breakdown by format
zerobucket verify    # re-checksum every image to detect corruption
zerobucket tier --all --bucket my-bucket   # move untiered images into S3 (needs zerobucket[s3])
```

Takes `--database-url` or reads `ZEROBUCKET_DATABASE_URL` from the
environment. `verify`/`tier` exit non-zero on any failure, so both are
usable in cron/CI. See the
[full CLI docs](https://github.com/KedarGhadyalji/ZeroBucket#cli) on
GitHub.

## Deduplication

```python
images = ZeroBucket(database_url=DATABASE_URL, dedup=True)
id1 = images.put("photo.jpg")
id2 = images.put("photo.jpg")  # identical content -- stored exactly once, referenced twice
```

Opt-in (`dedup=True`), uses separate tables from classic mode, so it's
safe to add later without touching existing data. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#deduplication)
on GitHub, including the migration path for existing classic-mode data.

## Tuning and observability

```python
images = ZeroBucket(
    database_url=DATABASE_URL,
    pool_max_size=10,       # default: 5
    on_operation=lambda e: print(e.operation, e.duration_seconds, e.success),
)
```

Pool sizing (`pool_min_size`/`pool_max_size`/`pool_timeout`) was
previously hardcoded, now configurable. `on_operation` fires after every
storage operation with timing, retry count, and success/failure --
callback exceptions are caught and never break a real operation. See the
[full explanation](https://github.com/KedarGhadyalji/ZeroBucket#tuning-and-observability)
on GitHub.

## Limitations (read before using in production)

- **Not built for large files or high-volume media.** `get_stream()`
  avoids holding a full image in Python memory during a read (see
  above), but Postgres itself still handles the full stored value the
  same way it always has -- no true server-side memory reduction, no
  HTTP range requests, no CDN.
- **Deduplication is opt-in, not automatic.** Default `dedup=False`
  stores every upload as a separate row; pass `dedup=True` for
  content-addressed, reference-counted storage (see above).
- **PostgreSQL only, for now.** The storage layer is abstracted for
  future adapters, but only Postgres exists today.

See the full
[README](https://github.com/KedarGhadyalji/ZeroBucket#readme) and
[roadmap](https://github.com/KedarGhadyalji/ZeroBucket#roadmap) on
GitHub for more detail.

## License

MIT -- see [LICENSE](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/LICENSE).
