Metadata-Version: 2.5
Name: bookshelf
Version: 1.0.0b10
Summary: Python SDK for the Bookshelf data platform
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: gitpython>=3.1
Requires-Dist: httpx>=0.27
Requires-Dist: keyring>=25
Requires-Dist: pandas>=2.3
Requires-Dist: platformdirs>=4
Requires-Dist: pyarrow>=18
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: dataframes
Provides-Extra: figures
Requires-Dist: matplotlib>=3.10; extra == 'figures'
Provides-Extra: publish
Requires-Dist: nbconvert>=7.0; extra == 'publish'
Requires-Dist: nbformat>=5.0; extra == 'publish'
Provides-Extra: scmrun
Requires-Dist: scmdata>=0.18; extra == 'scmrun'
Provides-Extra: test
Requires-Dist: keyring>=25; extra == 'test'
Requires-Dist: matplotlib>=3.10; extra == 'test'
Requires-Dist: polars>=1.8; extra == 'test'
Requires-Dist: pytest-asyncio>=0.23; extra == 'test'
Requires-Dist: pytest-cov>=6; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Requires-Dist: pyyaml>=6.0; extra == 'test'
Requires-Dist: typer>=0.12; extra == 'test'
Requires-Dist: types-pyyaml>=6.0; extra == 'test'
Description-Content-Type: text/markdown

# Bookshelf Python SDK

The `bookshelf` package is the official Python SDK for the Bookshelf data platform.
It provides synchronous and asynchronous facades for consuming published data,
producing managed resources,
and running record and replay publishing workflows.
It also includes the `bookshelf` command line interface for authentication,
discovery,
and local cache management.

This migration replaces the legacy Bookshelf consumer library.

## Installation

Install the SDK from PyPI:

```bash
uv add bookshelf
```

The SDK requires Python 3.12 or newer.
It ships with pandas and PyArrow, so `as_df()` and `as_arrow()` work out of the box.
`as_polars()` uses Polars if you have it installed.

Install optional integrations as needed:

```bash
uv add "bookshelf[scmrun,publish]"
```

For local development,
install the workspace package and all extras with the lock file enforced:

```bash
uv sync --locked --package bookshelf --all-extras
```

A local wheel can also be built and installed directly:

```bash
uv build --project packages/bookshelf --out-dir /tmp/bookshelf-sdk-dist
uv pip install /tmp/bookshelf-sdk-dist/bookshelf-*.whl
```

## Consuming published data

The `bookshelf` package provides synchronous and asynchronous facades.
Book coordinates resolve the latest published edition unless `edition=` pins one.
Indexing a `Book` returns a `BookEntry` with book scoped exploration helpers.

`search_volumes()` finds volumes by free text plus discovery filters,
and `volume()` resolves one, carrying the versions and editions it has published.
Both are available on either facade, and neither needs credentials for public data.

```python
from bookshelf import Bookshelf

with Bookshelf() as bs:
    found = bs.search_volumes("emissions", deprecated=False)
    volume = bs.volume("primap-hist")
    versions, latest = volume.versions, volume.latest
```

```python
with Bookshelf() as bs:
    entry = bs.book("rcmip-emissions", "v5.1.0")["magicc"]
    frame = entry.as_df(year_min=2020, year_max=2100, region="World")
    preview = entry.query(year_min=2020, top_n=5, drop_constant=True)
    facets = entry.facets()
```

`as_df()` reads the whole resource and returns pandas, using wide indexed form for timeseries resources.
The converter family also includes `as_long_df()`, `as_scmrun()`, `as_polars()`, and `as_arrow()`.
They all take `year_min`, `year_max` and `column=value` filters, applied locally after the download.
An unknown filter column raises `KeyError`.

`query()` filters and trims on the server instead.
It is quicker for a large entry, but the result is a preview:

- Book timeseries queries accept year bounds, constant dimension removal, top N selection, and row limits.
- They truncate at 10000 rows by default, and `top_n` and `limit` can drop single valued index columns.
- Their filters are bare `column=value` keywords, and an unrecognised key is ignored rather than rejected.
- Lean resource and tabular queries accept `select`, `order`, `limit`, and `offset`,
  plus the full `col.op` filter vocabulary.

Use `bs.resource(tracking_id)` for an exact machine or provenance path.
`fetch()` verifies the declared SHA256 before storing bytes in the local content cache.
`as_path()` returns the verified cached file.

The asynchronous facade has the same capabilities with awaited I/O:

```python
from bookshelf import AsyncBookshelf

async with AsyncBookshelf() as bs:
    book = await bs.book("rcmip-emissions", "v5.1.0", edition=2)
    frame = await book["magicc"].as_df()
```

## Producing and curating data

Managed resources are produced only inside an activity.
The activity derives a stable config hash, records runtime provenance, materialises the object,
and sends explicit Usage and Generation lineage to the API.
Bare strings and UUIDs in `used=` are tracking ids.
Use `Used(name=...)` to resolve an input by the name another resource in the same request was given.

```python
from bookshelf import Bookshelf, Used, models

with Bookshelf() as bs:
    source = bs.book("rcmip-emissions", "v5.1.0")["magicc"]
    with bs.activity(code_ref="github.com/example/model@abc123", config={"scenario": "ssp245"}) as activity:
        output = activity.register(
            transform(source.as_df()),
            type="timeseries",
            name="model/ssp245/output",
            used=[source, Used(name="model/constants")],
        )

    draft = bs.draft_book("model-results", version="v1.0.0")
    draft.attach(
        output,
        name_in_book="ssp245",
        data_dictionary=[
            models.DataDictionaryEntry(name="region", role="dimension"),
            models.DataDictionaryEntry(name="value", type="number", role="measure"),
        ],
    )
    draft.publish()
```

`register_external()` is available on both `Bookshelf` and an activity.
The former catalogues an existing pointer.
The latter attributes an external output to the current run.
Book drafting, attachment, and publication remain separate editorial calls.
Each tabular or timeseries entry can declare its own column descriptions through
``draft.attach(..., data_dictionary=...)``.
Omitting the argument preserves the entry's existing dictionary on re-attach,
while an empty list clears it.

Use `activity.register_many()` with `RegisterItem` values for a batch.
An atomic batch over 1000 items raises before any upload begins.
A larger non atomic batch is split into requests of at most 1000 items.
If any item fails, the facade finishes every chunk and raises `PartialRegistrationError`.
The error retains indexed successful outcomes, usable committed resource handles,
and each failed index with its typed `ItemError`.
Index `-1` identifies a batch level lineage failure reported by the server.
`RegisterItem.dedupe` defaults to true.
Byte identical items owned by one organisation therefore collapse to the first canonical resource,
even when later items supply a different name.
Returned producer handles expose `registration_status` and `registration_outcome`,
so callers can detect this `aliased` result.

A failed multipart PUT can leave an unfinished upload because the server has no abort endpoint.
Registration does not begin after that failure.
A retry reuses the content addressed upload path and safely resumes the workflow.

## Publishing a recorded bundle

`bookshelf record` runs a build file offline and writes a bundle,
and `bookshelf publish` replays that bundle to the platform.
`bookshelf.publisher.replay_bundle` and `replay_bundle_sync` do the same from Python.

A replay uploads the managed bytes,
then sends the whole bundle to `POST /v1/bundles/replay` as one transactional request.
The server registers the resources, mints the recorded activity's provenance edges,
drafts the book, attaches every entry and publishes it,
and rolls all of it back on a failure anywhere.

Every resource travels under its bundle-local name.
A resource that a book entry names takes that name inside the book,
and `used=` lineage cites the name of a resource recorded earlier in the same bundle.
An input the platform already holds is cited by its digest instead.
The server computes the seal from the request,
so replaying the same bundle again converges on the one edition
and reports `converged` rather than minting a rival.

## Storing a pull request preview

`bookshelf preview upload BUNDLE...` stores the books a feedstock pull request would publish as one preview,
so a reviewer can compare them with what is published before the pull request merges.
The feedstock CI workflow runs it once per build, passing one bundle per candidate book.

```bash
bookshelf preview upload bundle-v1.0.0 bundle-v2.0.0 \
  --repository "$GITHUB_REPOSITORY" --pr "$PR_NUMBER" --pr-url "$PR_URL" \
  --head-sha "$HEAD_SHA" --main-sha "$MAIN_SHA" --tree "$TREE_SHA" \
  --run-id "$GITHUB_RUN_ID" --json
```

- It creates the preview, uploads every book's bytes under the preview, attaches each book and seals it.
- A bundle that fails validation is still a target, so the preview is failed with the reason and the command exits 7.
- Any error once the preview exists fails it before the command exits, so the check run never waits for a timeout.
- It prints the preview id, the proposal and preview links, the state and each book.
  The platform owns the check run and the pull request comment.

The command authenticates with the job's GitHub Actions OIDC token for the `bookshelf` audience.
The usual credential chain is never consulted, so the job holds no Bookshelf credential.
The workflow therefore needs the `id-token: write` permission,
and without it the command exits 3 and names the permission.

```yaml
permissions:
  contents: read
  id-token: write
```

## Uploading a file that cannot be checked in

`bookshelf upload FILE --type TYPE` puts a file on the bookshelf as an input that belongs to no book,
and prints the `bookshelf://sha256/<hex>` URI a recipe declares it by.
`Bookshelf.register_file` does the same from Python,
and `Bookshelf.resource_by_hash` resolves the digest back into the resource.

The command leaves the file hidden, so it is readable by the uploading organisation alone.

## Generated model core

The committed files under `src/bookshelf/_generated/` are generated from the vendored `openapi.json`.
Do not edit them by hand.
The root package and private package expose the same model module and contract provenance stamp:

```python
from bookshelf import OPENAPI_VERSION, models
from bookshelf._generated import models as private_models

assert models is private_models
```

`OPENAPI_VERSION` is copied from the vendored contract's `info.version`.
It is not the distribution version and does not assert an ordered minimum server version.

The API contract is vendored at `packages/bookshelf/openapi.json`.
Refresh that snapshot explicitly when the platform contract changes,
then regenerate and review the model diff in the same change.

The exact generation command is caller-independent and locked:

```bash
uv run --project packages/bookshelf --locked --group codegen \
  python packages/bookshelf/scripts/generate_models.py
```

The driver validates a complete temporary tree before promotion.
It retains the last-known-good tree through a same-filesystem backup and recovers a sole valid backup on startup.
Ambiguous, invalid, or multiple-backup states stop without deleting evidence.

## Credential providers (unified client)

The unified client (`bookshelf._core.client.BookshelfClient`) authenticates through
credential providers, each an `httpx.Auth` whose flow is sans-io,
so one provider object serves both the sync and async surfaces:

- `StaticToken`: a fixed bearer token, no refresh.
- `RefreshTokenExchange`: a WorkOS user access/refresh pair from `bookshelf auth login`.
  The refresh token rotates on each use and an `on_rotate` callback persists the new pair.
- `ClientCredentials`: an OAuth2 `client_credentials` machine credential.
  A refresh is a plain re-POST, there is nothing to persist.
- `BsatAssertion`: an agent identity assertion re-exchanged via the `jwt-bearer` grant
  against the API's `POST /oauth2/token`.
  It is explicit-only and never resolved from the environment.

Refresh mechanics are shared: proactive refresh five minutes before expiry,
one refresh-and-replay after an unexpected 401 (a second 401 raises `AuthenticationError`),
and single-flight refresh behind per-surface locks.
There is no background refresh task.
A token handed in with no known expiry is refreshed before first use,
because it may already be dead server-side.

When `auth=` is omitted, ambient credentials resolve in this order
(explicit beats ambient, machine beats human):

1. `$BOOKSHELF_TOKEN` as a static bearer
2. `$BOOKSHELF_CLIENT_ID` + `$BOOKSHELF_CLIENT_SECRET`, minted at `$BOOKSHELF_TOKEN_URL`
3. stored `bookshelf auth login` credentials
4. unauthenticated (public reads)

`auth=` also accepts a provider instance or a bare token string,
and an explicit `auth=None` stays unauthenticated.
`base_url` resolves as argument, then `$BOOKSHELF_URL`, then its alias `$BOOKSHELF_API_URL`,
then the production URL.

### Client lifecycle in an embedded service

The client is long-lived by design: token state lives in the provider
and each surface pools connections.
Construct one client at startup, inject it as a dependency, and close it at shutdown.
In FastAPI that is a lifespan:

```python
from contextlib import asynccontextmanager

from fastapi import FastAPI, Request

from bookshelf._core.auth import ClientCredentials
from bookshelf._core.client import BookshelfClient

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.bookshelf = BookshelfClient(
        auth=ClientCredentials(client_id, client_secret, token_url=token_url),
    )
    yield
    await app.state.bookshelf.aclose()

app = FastAPI(lifespan=lifespan)

@app.get("/co2")
async def co2(request: Request):
    client: BookshelfClient = request.app.state.bookshelf
    return await client.query_resource_data_async(tracking_id)
```

Do **not** open a client per request (`async with BookshelfClient(...)` inside a handler):
that churns the connection pool and discards the cached token on every call.
Context managers are optional.
Notebooks can construct a client plainly and never close it.

## Testing

```bash
cd packages/bookshelf
uv run --locked --all-extras pytest
```

The public test suite uses local transports and fixtures.
Backend contract tests live with the private platform,
where the unpublished backend package is available.
