Metadata-Version: 2.4
Name: pennsieve-ai-utils
Version: 0.2.2
Summary: Shared blackboard contract, governor LLM client, and literature tools for the Pennsieve AI Co-Scientist stages.
License: MIT License
        
        Copyright (c) 2026 Pennsieve
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Repository, https://github.com/Pennsieve/pennsieve-ai-utils
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pennsieve-llm<0.7,>=0.6.1
Requires-Dist: httpx>=0.27
Requires-Dist: json-repair>=0.30
Requires-Dist: numpy>=1.26
Requires-Dist: rank-bm25>=0.2.2
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# pennsieve-ai-utils

Shared utilities for the Pennsieve AI Co-Scientist stages:

1. `pennsieve-ai-scout`
2. `pennsieve-ai-hypothesize`
3. `pennsieve-ai-analyze`
4. `pennsieve-ai-write`

Each stage used to carry its own copy of the blackboard schema, the governor
LLM client, the model-role resolver, the agent helpers and the literature
tools. They now import them from here. The package is published to PyPI as
`pennsieve-ai-utils` (same release flow as `pennsieve-llm`: push a `vX.Y.Z`
tag), so the Pennsieve App Store build of each stage can `pip install` it
without GitHub credentials.

See [RUNBOOK.md](RUNBOOK.md) for local stage supervision, stopping, review
bundles and the remaining Pennsieve Test deployment work.

```bash
pip install pennsieve-ai-utils
```

## What's here

| Module | Owns | Used by |
|---|---|---|
| `pennsieve_ai_utils.blackboard` | Canonical `Blackboard` JSON contract, `schema_version` migrations, unknown-field preservation | all stages |
| `pennsieve_ai_utils.llm` | `LLMClient`: governor transport via `pennsieve-llm`, role-based model selection, preflight, self-healing on 403/404, `usage.jsonl` logging | all stages |
| `pennsieve_ai_utils.models` | Role table (`reasoning`, `synthesis`, `bulk`), capability ranking, allow-list discovery, operator pins | `llm` |
| `pennsieve_ai_utils.agents` | `call_json`, `extract_json`, `first_text`, `format_datasets`, attack-tag helpers, `load_prompt` | every agent |
| `pennsieve_ai_utils.processor` | Pennsieve processor env contract (`INPUT_DIR`/`OUTPUT_DIR`/run IDs), SIGTERM handling, upstream blackboard loading, output persistence | every `main.py` |
| `pennsieve_ai_utils.tools` | CrossRef, DataCite, DOI lookup, JATS/Markdown chunkers, Bedrock embeddings + reranker, RAG store reader, public Discover file list/download | scout, hypothesize, analyze, write |
| `pennsieve_ai_utils.backend` | `VP_LLM_BACKEND` selection (governor, bedrock, anthropic) and the per-backend Agent SDK environment | all stages |
| `pennsieve_ai_utils.pricing` | Dated list-price table and `summarize_usage`, used to price `usage.jsonl` by token count at Bedrock rates | write, run report |
| `pennsieve_ai_utils.bedrock_proxy` | Local signing proxy that holds the AWS credentials so stage containers never see them (local/Bedrock runs only) | workflow supervisor |
| `pennsieve_ai_utils.run_report` | Reads a run folder and builds one self-contained `report.html`, including what each stage set aside | write, operators |
| `pennsieve_ai_utils.workflow` | Local four-stage supervisor, handoff gates, review bundles, `report` subcommand (`python -m pennsieve_ai_utils.workflow`) | operators |

Everything from `backend`, `pricing`, `bedrock_proxy`, `run_report` and the
local parts of `workflow` is for running the pipeline off-platform. On the
Pennsieve platform the default backend is `governor` and that path is
unchanged; the supervisor, the signing proxy and the MCP sidecar do not run
there.

Stage-specific science — pipelines, prompts, Discover/DANDI/OpenNeuro
inspection, notebook execution, manuscript rendering — stays in each app.

## Blackboard contract

```python
from pennsieve_ai_utils import Blackboard

bb = Blackboard.load("blackboard.json")
bb.record_model_assignment(
    stage="hypothesize",
    role="reasoning",
    assignment={"model_id": "us.anthropic.claude-sonnet-4-6"},
)
bb.save("blackboard.json")
```

- One superset schema carries fields produced by every stage.
- `schema_version` enables explicit migrations; documents without a version
  are treated as legacy version 0.
- Unknown fields are preserved at their original object level on a
  load/save round trip.
- Model provenance is cumulative by stage (`model_assignments[stage][role]`)
  rather than overwritten by the next application.

## LLM client

```python
from pennsieve_ai_utils.llm import LLMClient
from pennsieve_ai_utils.agents import call_json

client = LLMClient(stage="hypothesize", roles=("reasoning", "synthesis"))
assignments = client.preflight()          # raises ModelNotAvailable if a floor can't be met
for role, assignment in assignments.items():
    bb.record_model_assignment(stage="hypothesize", role=role, assignment=assignment)

verdict = call_json(client, agent="critic", system=..., user=..., run_id=bb.run_id,
                    model="reasoning")
```

Agents address models by **role**, never by vendor tier. The legacy names
`opus` / `sonnet` / `haiku` still resolve as aliases of
`reasoning` / `synthesis` / `bulk`, and `VP_MODEL_<ROLE>` pins keep working.

Preflight asks the governor's `GET /v1/models` for the allow-list first and
falls back to tripping a `model_not_allowed` 403 on older governors. Each
assigned model is then verified with a one-token call so an IAM block or a
retired Bedrock model fails at startup rather than twenty minutes in.

The `VP_LLM_BACKEND` setting chooses where model requests go.

| Value | Where requests go | What you must set |
|---|---|---|
| `governor` | Requests go to the Pennsieve LLM governor. This is the default, and it is what the platform uses. | The platform sets `LLM_GOVERNOR_FUNCTION_NAME`. Offline tests set `PENNSIEVE_LLM_MOCK=1` instead. |
| `bedrock` | Requests go straight to Amazon Bedrock. They use your own AWS login. | Set `AWS_REGION`. Set `AWS_PROFILE` or AWS access keys. |
| `anthropic` | Requests go straight to the Anthropic API. | Set `ANTHROPIC_API_KEY`. |

Use `bedrock` and `anthropic` only for local testing. On these two settings,
there is no list of approved models and no spending limit. Name the model for
every role yourself, with `VP_MODEL_REASONING`, `VP_MODEL_SYNTHESIS` and
`VP_MODEL_BULK`. At startup, the client sends a tiny test request to each model.
It stops with an error if your account cannot use one of them. Write model
names in the Bedrock form. The client changes them to the Anthropic API form
when needed. For more about the governor, see
[LLM access on compute nodes](https://docs.pennsieve.io/docs/llm-access-on-pennsieve-compute-nodes).

Set `VP_LOG_LLM_IO=1` to save every model request and every response, word for
word. Each stage writes them to `llm-io.<stage>.jsonl` in its output folder. A
line is written as soon as each request finishes. Failed requests are saved
too. The workflow supervisor turns this on for local runs. These files contain
the full prompts. The prompts include text from papers and datasets. Read the
files before you share them.

Each line in `usage.jsonl` records the number of tokens in and out. It also
records the tokens read from and written to the prompt cache, and the backend.
It does not record a dollar cost. A report can work out the cost later, using
current prices.

## Processor helpers

```python
from pennsieve_ai_utils.processor import (
    ProcessorEnv, install_sigterm_handler, load_upstream_blackboard, write_outputs,
)

install_sigterm_handler()
env = ProcessorEnv.from_environ()
bb = load_upstream_blackboard(env.input_dir / "blackboard.json",
                              stage="analyze", upstream="Hypothesize",
                              run_id=env.execution_run_id)
...
write_outputs(bb, env.output_dir, client.usage_log)
```

These follow the
[Pennsieve processor contract](https://docs.pennsieve.io/docs/pennsieve-processors):
read from `INPUT_DIR`, write to `OUTPUT_DIR`, exit non-zero on failure.

## Releasing

```bash
# bump version in pyproject.toml, commit, then:
git tag v0.2.0 && git push origin v0.2.0
```

`.github/workflows/publish-pypi.yml` builds and publishes via PyPI Trusted
Publishing. One-time setup on pypi.org: add this repository as a trusted
publisher for the `pennsieve-ai-utils` project (GitHub environment `pypi`).
Apps pin `pennsieve-ai-utils>=0.2,<0.3` and pick up patch releases on rebuild.

## Development

```bash
python -m pip install -e ".[dev]"
python -m ruff check src tests
python -m pytest
```
