Metadata-Version: 2.4
Name: percolate-core
Version: 0.1.2
Summary: Percolate core: the worker, Content Server and Agent Runtime for a Postgres-native stack
Project-URL: Homepage, https://github.com/percolating-sirsh/percolate-core
Project-URL: Repository, https://github.com/percolating-sirsh/percolate-core
Project-URL: Specs, https://github.com/percolating-sirsh/p8-subsystems
Author: Percolate
License-Expression: MIT
License-File: LICENSE
Keywords: agents,pgvector,postgres,rag,workflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: asyncpg>=0.30
Requires-Dist: httpx>=0.28
Requires-Dist: typer>=0.15
Provides-Extra: agent
Requires-Dist: fastapi>=0.115; extra == 'agent'
Requires-Dist: mcp>=1.2; extra == 'agent'
Requires-Dist: pydantic-ai-slim[openai]>=0.4; extra == 'agent'
Requires-Dist: pydantic>=2.9; extra == 'agent'
Requires-Dist: pyyaml>=6.0; extra == 'agent'
Requires-Dist: rich>=13.9; extra == 'agent'
Requires-Dist: uvicorn>=0.34; extra == 'agent'
Provides-Extra: all
Requires-Dist: boto3>=1.35; extra == 'all'
Requires-Dist: fastapi>=0.115; extra == 'all'
Requires-Dist: mcp>=1.2; extra == 'all'
Requires-Dist: pydantic-ai-slim[openai]>=0.4; extra == 'all'
Requires-Dist: pydantic>=2.9; extra == 'all'
Requires-Dist: pyyaml>=6.0; extra == 'all'
Requires-Dist: rich>=13.9; extra == 'all'
Requires-Dist: uvicorn>=0.34; extra == 'all'
Provides-Extra: content
Requires-Dist: boto3>=1.35; extra == 'content'
Requires-Dist: fastapi>=0.115; extra == 'content'
Requires-Dist: uvicorn>=0.34; extra == 'content'
Description-Content-Type: text/markdown

# percolate-core

The processes that sit in front of the database. The database itself, and the
specs all of this is built against, live in
[p8-subsystems](https://github.com/percolating-sirsh/p8-subsystems); nothing
here is required to use it.

```bash
pip install percolate-core                 # the worker -- asyncpg, httpx, typer
pip install 'percolate-core[content]'      # + Content Server (boto3, fastapi)
pip install 'percolate-core[agent]'        # + Agent Runtime (pydantic-ai, mcp)
pip install 'percolate-core[all]'
```

```bash
percolate worker --queue http    # claim and execute tasks
percolate content serve          # uploads, scraping, ingestion
percolate agent serve            # agents, streaming, delegation
percolate agent --help           # the runtime's own operator commands
```

[`docs-sample.md`](docs-sample.md) is the one to read next: a real REST call to
the Agent Runtime, the events it streams back, the rows it leaves behind, and
every header and option a caller can send — captured output, not illustration.

---

## Layout

| Module | Extra | What it is |
|---|---|---|
| `percolate_core.core` | — | connecting **as the caller**, configuration, credential resolution |
| `percolate_core.worker` | — | the step loop and the `@handler` registry |
| `percolate_core.content` | `content` | the Content Server |
| `percolate_core.agentic` | `agent` | the Agent Runtime |

The import package matches the distribution name exactly, so there is no mapping
to remember. (`p8` was taken on PyPI.)

**One distribution rather than three.** Three packages means a version matrix
(`percolate-content 0.3` requiring `percolate-core >=0.2,<0.3`) resolved forever, for services
that release together and are written by the same people. Splitting later is
mechanical; merging two that have drifted is not.

**The base install stays small on purpose.** The common case is someone writing
their own worker, and they should not pull boto3 and pydantic-ai to do it. The
root CLI mounts each subpackage's own command group and tolerates its absence,
so `percolate worker` runs with neither installed and `percolate content` says
which extra to install — naming the module that was actually missing, rather
than guessing.

---

## `percolate_core.core` — the one that matters

It decides **whose** RLS applies, and it exists because that logic was
previously written three times on two different database drivers.

```python
from percolate_core.core import as_caller

async with as_caller(claims) as conn:      # claims = the VERIFIED JWT payload
    rows = await conn.fetch("select * from content.resources")
```

Every service connects as a low-privilege role — set explicitly by `as_caller`,
not inherited from whatever the connection string happens to log in as — and
sets the caller's claims **per transaction**, exactly as PostgREST does. A service that queried as
*itself* would bypass every policy in the collection — not by exploiting
anything, just by never presenting an identity for the policies to filter on.

Transaction-local (`set_config(..., true)`) is not a detail: an unregistered
GUC left at session scope survives into the next transaction on a pooled
connection, so the following request would inherit the previous caller's
identity.

`as_service()` exists for work with genuinely no user behind it — a scheduled
poll, a reconciliation sweep. Deliberately a separate function rather than
`as_caller(None)`, so "this query has no user" is something someone wrote down.

**One pool per process, with explicit ownership.** A service's lifespan owns it
(`open_pool`/`close_pool`, reference-counted); every short-lived helper borrows
it (`pool()`). Several mounted services in one process share it, and the last
one out closes it — without that, one service's shutdown closes the pool another
is still streaming through.

---

## Writing your own worker

Most steps need no worker from you:

| kind | who runs it |
|---|---|
| `sql` / `p8ql` | **nobody** — executes inside Postgres |
| `http_call` | the built-in handler |
| `timer` / `signal` / `decision` / `sub_workflow` | the engine |
| `work` | **you** |

When you do need one, it is this loop with a handler registered — not a
different program:

```python
from percolate_core.worker import handler, run

@handler("transcode")
async def transcode(spec, ctx):
    return {"duration": await ffmpeg(ctx["run_input"]["file_key"])}

run(queue="media")
```

`ctx` comes from `workflow.get_task_context()`: `run_input`, the accumulated
`context` (so a later step reads an earlier step's output), `task_input`,
`step_key`, and `trace_id`/`span_id`.

**The worker holds no table grants.** Every interaction is a `SECURITY DEFINER`
function call — `claim_task`, `get_task_context`, `complete_task`, `fail_task`
— which is why "bring your own worker" is safe to offer: a compromised worker
can claim and complete tasks, and nothing else.

**Raise `TerminalError` for what will not get better.** A bad argument, a
missing credential, a 404. Anything else is retried with backoff. The worker is
the only thing that knows what a failure means, so it decides and the engine
honours the verdict.

---

## Configuration

Environment only. Credentials by **reference**, never by value:
`credential_ref: "LLM_API_KEY"` on a task names a variable this process
resolves, so `workflow.tasks` stays inspectable and replayable.

| | Used by |
|---|---|
| `P8_DSN` | all |
| `P8_JWT_SECRET` | services verifying bearer tokens (same secret PostgREST uses) |
| `P8_QUEUE`, `P8_WORKER_ID`, `P8_POLL_SECONDS` | worker |
| `P8_S3_ENDPOINT`, `P8_S3_KEY`, `P8_S3_SECRET`, `P8_BUCKET` | content |

---

## Deployment

One image, several entrypoints — they share `percolate_core.core`, so separate
images would be separate builds of the same base and separate tags to keep in
step. `ENTRYPOINT ["percolate"]`, and the command selects the service:

```yaml
content: { image: percolationlabs/percolate-core:0.1.1, command: ["content","serve"] }
agent:   { image: percolationlabs/percolate-core:0.1.1, command: ["agent","serve"] }
gateway: { image: percolationlabs/percolate-core:0.1.1, command: ["agent","gateway"] }
worker:  { image: percolationlabs/percolate-core:0.1.1, command: ["worker","--queue","http"] }
```

The agent gateway is its own entrypoint because delegation is an MCP call into
it: an agent that delegates references it as an ordinary `tool_servers` row.

For the packaging decision record see
[PACKAGING.md](https://github.com/percolating-sirsh/p8-subsystems/blob/main/PACKAGING.md);
for what packaging *requires of the design*, `specs/agentic/brief.md` §10.

---

## Status

- `core`, `worker`, `content` — built, and exercised against a live PG19
  instance with MinIO.
- `agentic` — built and verified under the new namespace: `tests/smoke.py` (6
  checks, no database or credentials) and `dev/verify.py` (38 assertions against
  a live model, covering streaming, what is and is not persisted, the delegation
  and span trees, the reload round trip, mounting, session resume, tenant
  isolation, and a scheduled run completing its own workflow task).
- Still open: no service-surface entries in the specs repo's `surface.sql`,
  which by `meta/skills/spec-driven-development` §7 should exist **before** the
  endpoints they describe; and no startup check that the deployed schema is the
  one this version expects (`specs/agentic/brief.md` §10.4).

## Development

```bash
. dev/env.sh          # DSN, test user, JWT, and the LLM key
./dev/db.sh up        # a PG19 cluster of its own, loaded from the specs repo
./dev/stack.sh up     # PostgREST, the retrieval service, the agent gateway
uv run --all-extras python dev/verify.py
```

`dev/db.sh` loads `specs/*/schema.sql` from a p8-subsystems checkout directly —
never a copy — so a schema that only works because a harness did something extra
fails here. Set `P8_SPECS` if your checkout is elsewhere.
