Metadata-Version: 2.4
Name: fourgate
Version: 0.3.0
Summary: Independent outcome verification for consequential AI-agent actions
Author: Hifza Zafar
License: MIT License
        
        Copyright (c) 2026 Hifza
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/hifzabuildsai/fourgate
Project-URL: Source, https://github.com/hifzabuildsai/fourgate
Project-URL: Issues, https://github.com/hifzabuildsai/fourgate/issues
Keywords: mcp,ai-agents,verification
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: mcp==1.9.4; extra == "dev"
Dynamic: license-file

# Fourgate

Fourgate catches MCP tools that say 'success' when nothing actually happened.

**Status: early, pre-alpha.**

## Results so far (2026-10-01)

Tested against 4 official vendor MCP servers from 4 companies (run locally over
stdio against the vendors' live APIs), using disposable test accounts or
deliberately invalid credentials so no real data could be changed.

| Check | Result |
|---|---|
| Servers where a write tool returned an API failure as a successful tool result (no `isError`) | **3 of 4**, each reproduced at least twice, reported upstream |
| Healthy write verified by independent read-back | 1 server: PASS, with FAIL and UNKNOWN controls behaving as specified |
| Runtime shadow calls from a real agent client | 2 of 2 PASS |
| Read-back-proven silent success (reported done, record missing or wrong) | 0 observed so far |

An agent reading a result without `isError` has no protocol-level signal that
the call failed. Fourgate reports these as `UNKNOWN / success_without_record_id`
rather than PASS. Vendors are not named here; details are under
[Field evidence](#field-evidence).

Operators running scans against real MCP servers: see [`OPERATOR.md`](OPERATOR.md) for the step-by-step Windows guide and contract templates.

Evaluating a pilot: [`PILOT.md`](PILOT.md) explains what runs where, what is stored, and where your data goes.

## Scan a disposable MCP test account (under five minutes)

Python 3.10–3.13, Linux or Windows. Install the v0.3.0 release:

```bash
python -m pip install "git+https://github.com/hifzabuildsai/fourgate@v0.3.0"
```

The built wheel is also attached to the
[v0.3.0 GitHub Release](https://github.com/hifzabuildsai/fourgate/releases/tag/v0.3.0)
(`python -m pip install fourgate-0.3.0-py3-none-any.whl`). Fourgate is not on
PyPI yet. Check the install with `fourgate --version`.

Or, from a clean checkout of this repository (the bundled fixture below needs
the checkout):

```bash
python -m pip install .
```

The product flow on one screen:

```text
fourgate demo      # see it work: local, simulated, no keys, no network
fourgate init      # write scan, runtime and read-back configs for one write tool
fourgate doctor    # check the setup without side effects
fourgate scan      # contracted writes against a disposable test account
fourgate guard     # wrap the server at runtime (shadow first, then enforce)
fourgate summary   # turn outcomes.jsonl into a local HTML page
```

Each step is described below.

For the bundled **local fixture only**, set a disposable store location and
scan its deliberately broken issue creation. Bash:

```bash
export FOURGATE_DEMO_STORE="$(mktemp -d)/issues.json"
export FOURGATE_DEMO_MODE=broken
fourgate scan fixtures/contracts/scan_demo.json \
  --confirm-test-account disposable-demo --report-dir ./fourgate-output
```

PowerShell:

```powershell
$env:FOURGATE_DEMO_STORE = "$env:TEMP\fourgate-demo-issues.json"
$env:FOURGATE_DEMO_MODE = "broken"
fourgate scan fixtures/contracts/scan_demo.json --confirm-test-account disposable-demo --report-dir ./fourgate-output
```

The expected result is `FAIL / record_missing` (exit 1), with local
`fourgate-report.json` and self-contained `fourgate-report.html` in the chosen
directory. Set `FOURGATE_DEMO_MODE=healthy` and rerun with a fresh store path to
see PASS (exit 0). UNKNOWN is never counted as PASS (exit 1). Invalid scan
configuration exits 2. Reports include the exact contracted request, response,
read-back evidence and reproduction steps for a confirmed failure, after
best-effort secret redaction. Review locally before sharing.

Only explicitly named write tools and arguments in the contract are called.
The scanner supports stdio MCP servers and local command, generic HTTPS, or
GitHub Issue read-back. HTTP/GitHub verification makes GET requests to the
configured endpoint; no scan telemetry is sent to Fourgate. See
[`specs/scan-contract-v1.md`](specs/scan-contract-v1.md) and
[`SECURITY.md`](SECURITY.md). The GitHub contract is a template that still
requires a disposable repository, tokens, and real end-to-end acceptance.

## Run in CI

[`examples/ci/fourgate-scan.yml`](examples/ci/fourgate-scan.yml) is a
copy-paste GitHub Actions workflow for your own repository. It installs
Fourgate from a release tag, runs `fourgate scan` on manual dispatch, uploads
the JSON/HTML reports as a workflow artifact, and fails the job on any FAIL or
UNKNOWN (non-zero exit). Edit the marked values: contract path, the
`--confirm-test-account` label (must equal `server.test_account`), and the
environment variable names. The MCP server's write credential and the
read-back token come from two separate secrets, `FOURGATE_MCP_WRITE_TOKEN` and
`FOURGATE_READBACK_TOKEN`, mapped to separate environment variables.

Every run performs real contracted writes: target a disposable test account
only. Workflow artifacts are readable by anyone with read access to the
repository, so use a private repository and review reports before sharing.
Fourgate's own CI runs the template's steps against the bundled demo fixture
(broken exits 1, healthy exits 0); the template has not yet been run against a
hosted connector.

## Outcome Guard MVP

The current pitch-ready slice protects explicitly contracted state-changing MCP tools. After a connector reports success, Fourgate extracts only the minimum contract-approved fields and runs a deterministic authoritative verifier. No LLM makes the runtime PASS/FAIL decision.

Three outcomes are possible:

- **PASS** — connector response is unchanged.
- **CONFIRMED FAIL** — in enforce mode, Fourgate prepends an attributed `outcome_failed` verdict before the connector's original content.
- **UNKNOWN** — verifier crash, timeout, malformed output, missing selector, or Fourgate internal fault fails open; connector response is unchanged.

Shadow mode is the default. It records structural verdict metadata without changing what the model receives.

### Run the proof

```bash
fourgate demo
```

One command, no API keys, no network: five real MCP sessions against a demo connector, the guarded ones through a real `fourgate guard` process, then a self-check and a local summary page (`./fourgate-demo/fourgate-demo-summary.html`). `--pace 2` slows it down for screen recording.

1. **Without Fourgate:** the broken connector returns `Created ISSUE-001`, but the system of record has 0 issues.
2. **Enforce, broken connector:** the agent receives `[FOURGATE] {"kind":"outcome_failed", ...}` (`record_missing`) first, with the original `Created ISSUE-001` preserved behind it.
3. **Enforce, healthy connector:** the issue exists, the response passes through unchanged, and the log records PASS.
4. **Shadow, broken connector:** the agent receives exactly what it got without Fourgate; the log still records FAIL `record_missing`.
5. **Enforce, system of record unreachable:** Fourgate cannot confirm either way, records UNKNOWN (never PASS), and does not block the call.

Simulated connector; a local file stands in for the system of record.

`python demo_outcome.py` remains for development from a source checkout.

### Scaffold a setup with fourgate init

```bash
fourgate init --tool create_issue --test-account disposable-demo   --id-path result.structuredContent.id   --readback-url 'https://api.example.com/issues/{record_id}'   --expect title=title --arg title=FOURGATE-SCAN-TEST   --token-env READBACK_TOKEN --missing-status 404   -- python your_server.py
```

`init` writes three files to `./fourgate-config` (`--dir` to change; it refuses to overwrite without `--force`): `readback.json` (the HTTP read-back), `runtime.json` (guard contracts using `fourgate.verify_http`) and `scan.json` (one scan case with the same read-back). It validates all three with the same loaders scan, guard and doctor use, and writes nothing if any check fails. Then it prints the next steps with your paths filled in: `fourgate doctor`, `fourgate scan`, `fourgate guard --mode shadow`, `fourgate summary`.

No flag takes a secret: credentials are named by environment variable (`--token-env`, `--username-env`), and the files contain only those names. `init` never reads their values, does not start the server, and does not contact any endpoint.

### Preflight with fourgate doctor

```bash
fourgate doctor --contracts path/to/contracts.json --log outcomes.jsonl -- python your_server.py
```

`doctor` takes the same arguments as `guard` and checks the setup before you wrap a real server: the contracts, each verifier (for the bundled `fourgate.verify_http`, its read-back config, URL rules, budget and failure reasons), that credential variables are set and withheld from the server, the outcome log location, and, when a server command follows `--`, that the server answers `initialize` and `tools/list` and advertises every protected tool. It exits 0 when the setup is ready for shadow mode, 1 when it is not, and 2 on a usage error.

It never sends `tools/call`, never runs a verifier, never makes a network request, and never creates or modifies a file. It therefore cannot confirm that the read credential is valid or read-only, or that the read-back endpoint is reachable. When a server command is given, the server is started only for `initialize` and `tools/list`; anything the server does on its own at start-up is outside Fourgate's control.

### Wrap a local stdio server

```bash
fourgate guard \
  --contracts path/to/contracts.json \
  --mode shadow \
  --server my-server \
  --log outcomes.jsonl \
  -- python your_server.py
```

Switch to `--mode enforce` only for human-approved contracts after shadow traffic is clean. `guard` validates the contracts at startup and exits 2 without starting the server if any are invalid; per-call verifier faults are still UNKNOWN and fail open. The startup summary goes to stderr, so stdout carries only MCP traffic. `python wrap/wrap.py` with its existing flags remains supported for existing setups.

For a hosted connector, the verifier can be the scan's own HTTP read-back:
`python -m fourgate.verify_http readback.json` takes the extracted fields on
stdin and runs the same bounded GET as `fourgate scan`. Any uncertainty exits
nonzero and is recorded as UNKNOWN. `verifier.secret_env` names the read
credential's environment variables; `fourgate guard` removes them from the wrapped
server's environment while the verifier still receives them:

```json
{
  "contract_version": 1,
  "tools": {
    "create_issue": {
      "extract": {
        "issue_id": {"source": "result", "path": "result.structuredContent.issue_id"},
        "title": {"source": "arguments", "path": "title"}
      },
      "verifier": {
        "command": ["{python}", "-m", "fourgate.verify_http", "readback.json"],
        "cwd": ".",
        "timeout_ms": 2000,
        "secret_env": ["READBACK_TOKEN"]
      },
      "allowed_failure_reasons": ["field_mismatch"],
      "recovery": "stop"
    }
  }
}
```

### See the results

```bash
fourgate summary outcomes.jsonl --out fourgate-summary.html
```

Turns one or more `outcomes.jsonl` files into a single local HTML page: total
protected calls, PASS / FAIL / UNKNOWN by tool and by day, what each reason
means, and the latency Fourgate added (median and p95). The page has no
JavaScript and makes no network requests. Like the log, it contains structure
only: no arguments, record values, response bodies or credentials. It also
prints a one-line text summary (`--json` for machine-readable output).

`readback.json` is a scan contract `readback` object with `timeout_ms` of at
most 1500. The read token is sent as `Authorization: Bearer` unless
`readback.auth` selects a custom header (`{"scheme": "header", "header":
"X-Api-Key"}`) or Basic auth; `readback.headers` adds fixed, non-secret headers
such as an API version. List every read credential variable (`token_env`, and
`auth.username_env` if used) in `verifier.secret_env`. Details:
[`OPERATOR.md` → 3c](OPERATOR.md#3c-find-the-read-back-get-endpoint). Step-by-step setup and the verifier dry run are in
[`HANDOFF.md` → Shadow mode with a real connector](HANDOFF.md#shadow-mode-with-a-real-connector).

See [`specs/outcome-guard-mvp.md`](specs/outcome-guard-mvp.md) for the contract and failure semantics.

## Existing components

### Preflight CLI

`checker/preflight.py` starts a Python MCP server over stdio, speaks raw JSON-RPC, detects stdout pollution, and exercises a limited set of schema/argument edge cases. Keep it free/low-cost infrastructure; it is not a replacement for MCP Inspector or conformance tooling.

### Runtime wrap

`wrap/wrap.py` is the on-path stdio proxy. It already correlates `tools/call` requests/results by JSON-RPC id, supports byte-identical pass-through, a bounded fail-open legacy classifier path, an attributed in-band verdict, and shape-only observation.

The existing `silent_empty` behavior remains deliberately narrow. Fourgate does **not** flag `{"issues":[]}`, `No matches found`, or explicit empty structured data merely because it is empty. Empty data is wrong only when an approved contract/postcondition proves it.

## Field evidence

The original field run observed 24 real `tools/call` results and zero `silent_empty` events. Four observed failures happened before `tools/call` at spawn/connect/discovery/configuration. That weakens the original empty-payload wedge; it does not validate Outcome Guard on its own.

### Hosted scan validation (2026-09-30)

Operator-reported validation on a disposable account: a real hosted MCP write
returned an ID, and an independent HTTPS read-back with a separate credential
produced `PASS / postcondition_satisfied`. An intentionally mismatched
postcondition produced `FAIL / field_mismatch` with authoritative read-back
evidence. A manual search of the generated reports found neither tested API
credential. No naturally occurring false-success incident was observed. This
single integration does not establish production reliability.

### Runtime shadow run (2026-10-01)

Operator-reported: a real agent client made two protected send calls through
`wrap.py` in shadow mode against the same hosted email-sending MCP server on a
disposable account, verified by `fourgate.verify_http` with `secret_env`
withholding the read-back key from the connector. `outcomes.jsonl` recorded
`pass / postcondition_satisfied` for both. Two calls is a smoke test, not
production traffic.

### Errors returned as success

Operator scans of three official company MCP servers (hosting, payments, work
management), using deliberately invalid credentials so nothing could be
created, found write tools returning API failures as successful tool results
with no `isError`. Fourgate reports these as `UNKNOWN /
success_without_record_id`. Each was reproduced at least twice and reported
upstream. These are error-reporting bugs: no read-back-proven silent success
(a write reported as done whose record is missing or wrong) has been observed
yet.

## Current limitations

- Outcome contracts are hand-authored and human-approved; generation is not implemented.
- Runtime verifiers are local commands. The bundled one, `fourgate.verify_http`, is a generic HTTP GET read-back; the read credential can be sent as a Bearer token, a custom header, or Basic auth; there are no database adapters.
- Outcome verification adds read-back latency on the protected call. The runtime caps each verifier at 2000 ms and fails open (UNKNOWN) on uncertainty. `verify_http` measured 1.2–1.6 s per call on a Windows laptop, and a first cold call exceeded its budget, so headroom is small on slow networks.
- Only command-configured local stdio MCP servers are covered (scanner and runtime); no remote MCP transport.
- No dashboard, alerting product, gateway, retry engine, or automatic compensation.
- Preflight's test handshake uses an older MCP protocol version, and its tests need the `mcp==1.9.4` dev dependency.

## Tests

```bash
python -m pip install -e ".[dev]"
python -m pytest -q
python wrap/wrap.py --selfcheck
```

The `dev` extra installs `pytest` and the `mcp==1.9.4` package that the
preflight tests need.

## License

MIT — see [LICENSE](LICENSE).
