Metadata-Version: 2.3
Name: safesink
Version: 0.4.2
Summary: MCP tool side-effect conformance verifier (formerly EffectFence)
Author: Viraj Sabhaya
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Keywords: mcp,model-context-protocol,conformance,side-effects,idempotency,safesink
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Project-URL: Homepage, https://github.com/virajsabhaya23/effectfence
Project-URL: Repository, https://github.com/virajsabhaya23/effectfence
Project-URL: Issues, https://github.com/virajsabhaya23/effectfence/issues
Provides-Extra: live-kafka-postgres
Requires-Dist: confluent-kafka>=2.5,<3; extra == "live-kafka-postgres"
Requires-Dist: psycopg[binary]>=3.2,<4; extra == "live-kafka-postgres"

# SafeSink

**SafeSink checks whether an MCP server's tools do what their annotations claim.**

MCP tool annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`,
`openWorldHint`) are untrusted hints: nothing stops a server from declaring a
tool read-only while it writes files, or idempotent while every retry charges the
customer twice. SafeSink calls the tool, observes the state it actually changed
inside an explicit filesystem, SQLite, or HTTP JSON boundary, and reports where
the declaration and the observed effect disagree.

One command, no manifest, no server changes:

```bash
safesink mcp-check -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"
```

## Install

```bash
python -m pip install safesink
```

Python 3.10+; no runtime dependencies. To work on it, from a clone:

```bash
git clone https://github.com/virajsabhaya23/effectfence.git
cd effectfence
python -m pip install -e .
```

The repository ships a dependency-free PEP 517 build backend, so the build works
offline.

## Quick start

Point it at any stdio MCP server:

```bash
mkdir -p sandbox
safesink mcp-check \
  --sandbox sandbox \
  --out out/mcp/report.json \
  --junit out/mcp/junit.xml \
  --sarif out/mcp/report.sarif \
  -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"
```

`mcp-check` starts the server, reads `tools/list`, synthesizes minimal arguments
from each tool's `inputSchema`, seeds the sandbox with probe fixtures those
arguments can refer to, skips tools declaring `destructiveHint: true`, and
verifies what changed. The command goes after `--` so its own flags are not
parsed by `safesink`.

For a server that needs a curated argument per tool, write a manifest and use
`mcp-verify` instead:

```bash
safesink mcp-verify my-manifest.json --out out/mcp/report.json
```

The honest fixture and the deliberately dishonest one show both outcomes:

```bash
safesink mcp-verify examples/mcp-conformance/passing.json   # exit 0
safesink mcp-verify examples/mcp-conformance/failing.json   # exit 2
```

From Python:

```python
from safesink import verify_manifest

report = verify_manifest("manifest.json")
print(report["verdict"], report["certificateSha256"])
```

## Verdicts and exit codes

| Verdict | Exit | Meaning |
| --- | --- | --- |
| `pass` | 0 | every case was exercised and no declared effect was contradicted |
| `inconclusive` | 3 | nothing was contradicted, but at least one case could not be verified |
| `fail` | 2 | a declared effect was contradicted, or the run could not be trusted |

`inconclusive` is a first-class outcome, not a soft pass. It covers rejected
synthesized arguments, observers whose state changes on its own, and ambiguous
results the observation boundary cannot classify. A case is only `pass` when it
has no violations **and** no unresolved evidence, so CI never reports an
unverified case as verified.

## What it checks

| Finding | Meaning |
| --- | --- |
| `READ_ONLY_HINT_MISMATCH` | a tool declared `readOnlyHint: true` changed observed state |
| `NON_DESTRUCTIVE_HINT_MISMATCH` | a tool declared `destructiveHint: false` removed state |
| `CLOSED_WORLD_HINT_MISMATCH` | a tool declared `openWorldHint: false` changed an open-world observer |
| `IDEMPOTENT_HINT_MISMATCH` | repeating the identical call changed observed state again |
| `TOOL_ERROR_EXPECTATION_MISMATCH` | `tools/call` returned `isError` contrary to the case |
| `TOOL_NOT_LISTED` | a case targets a tool that `tools/list` never advertised |
| `TOOL_COVERAGE_BELOW_MINIMUM` | too few advertised tools are covered by cases |

Two behaviours keep false findings out of reports:

- **Observer control rounds.** Every case (and every ambiguous-result trial) runs
  matched no-tool observation rounds first. A delta that reproduces without the
  tool is reported as `OBSERVER_OR_BACKGROUND_INTERFERENCE` and is not blamed on
  the tool.
- **Argument-synthesis limits are inconclusive.** When `mcp-scan` generated the
  arguments and the server rejects them, the result is
  `TOOL_ERROR_ON_GENERATED_ARGUMENTS`, not a conformance finding. Hand-written
  cases that are rejected still fail.

Ambiguous-result schedules (`ambiguousResultFault`) repeat a call under
`drop-result-after-response` or `timeout-before-send` and classify the outcome as
committed-result-lost, no-effect-timeout, duplicate-on-retry, or explicitly
unclassified.

## Validation against real servers

Measured on 2026-11-21 with the released upstream servers, `safesink mcp-check`,
default settings (destructive tools skipped):

| Server | Version | Tools | Cases | Findings | Inconclusive | Verdict |
| --- | --- | --- | --- | --- | --- | --- |
| `@modelcontextprotocol/server-filesystem` | 2.4.0 | 16 | 13 | 0 | 1 | inconclusive (exit 3) |
| `@modelcontextprotocol/server-memory` | 2.2.1 | 9 | 6 | 0 | 0 | pass (exit 0) |
| `@modelcontextprotocol/server-sequential-thinking` | 2.2.0 | 1 | 1 | 0 | 0 | pass (exit 0) |

The single inconclusive case is `copy_file`, whose second identical call is
rejected because the first call created the destination; the tool's
`idempotentHint: true` therefore remains unverified rather than violated.

These are trusted servers, so zero findings is the expected result: the table
shows that the scanner does not manufacture findings, not that it can catch a
liar. Detection is covered by the deliberately dishonest fixture, where the
suite asserts `READ_ONLY_HINT_MISMATCH`, `IDEMPOTENT_HINT_MISMATCH`, and
`NON_DESTRUCTIVE_HINT_MISMATCH`.

An earlier revision reported 5 of these 13 cases as conformance failures because
synthesized arguments pointed at directories and files that did not exist. That
false-positive rate is why generated arguments are now seeded from probe fixtures
and why a rejected generated call is inconclusive.

## CI

```yaml
- uses: actions/checkout@v7
- uses: virajsabhaya23/effectfence@main
  with:
    manifest: examples/mcp-conformance/passing.json
    out: out/mcp/report.json
    junit: out/mcp/junit.xml
    sarif: out/mcp/report.sarif
```

Pin a full commit SHA in high-assurance environments. The action verifies a
manifest; use `mcp-check` in the job directly if you have no manifest yet. JUnit
marks unverifiable cases as `skipped`, not failed, matching the verdicts above.

## Security model

- Commands are executed without a shell, as arrays, with a restricted
  environment (`PATH` plus explicitly inherited variables).
- HTTP observers are host-allowlisted and follow no redirects.
- Argument and output values are redacted by default, and sensitive observer
  digests are excluded from reports.
- `mcp-check` only ever mutates the sandbox directory you give it, and skips
  tools that declare themselves destructive unless you pass
  `--include-destructive`.
- Only configured observers define the observable boundary. Snapshot comparison
  cannot detect unobserved side effects, and it cannot prove that an external
  read occurred.
- Annotations come from the server under test, so a dishonest server can lie to
  the checker itself; SafeSink reports what it observed, within that boundary.

See [the MCP conformance guide](docs/MCP_CONFORMANCE.md) for the versioned
manifest, the observer boundary, and the full security model.

## Frozen crash/retry simulator

Earlier versions also shipped a deterministic crash/retry schedule simulator with
a SQLite effect ledger, reference sinks, a 30-case benchmark, and a live
Kafka/PostgreSQL killpoint adapter. It is **frozen**: kept for reproducibility of
existing artifacts, with no new features, adapters, or benchmark cases.

```bash
safesink-legacy verify examples/crash_after_effect.json --out out/report.json
safesink-legacy explore examples/crash_after_effect.json --out out/exploration.json
safesink-legacy benchmark benchmark/corpus.json --out benchmark/results.json
```

The implementation lives in `safesink/legacy/`; `python -m safesink.legacy.cli`
is equivalent. See [docs/FROZEN_CRASH_RETRY.md](docs/FROZEN_CRASH_RETRY.md) for
its status, scope, and known limitations (notably: its benchmark is graded by
the same model that generates the schedules, so its precision/recall figures
measure corpus authoring, not detection of real systems).

The deprecated `effectfence` import path and console script still work and
forward to `safesink`.

