Metadata-Version: 2.4
Name: skeptic-cli
Version: 0.6.1
Summary: Independent engineering quality gate for AI-generated and human-written Python code.
Author-email: Hamza Shaikh <hamzashaikhm123@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Hamza Shaikh
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/HamzaShaikh17/Skeptic
Project-URL: Repository, https://github.com/HamzaShaikh17/Skeptic
Project-URL: Issues, https://github.com/HamzaShaikh17/Skeptic/issues
Project-URL: Documentation, https://github.com/HamzaShaikh17/Skeptic/blob/master/docs/USAGE.md
Keywords: code-quality,static-analysis,security,ci-cd,ai-agents,mcp,linting,skeptic
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: pydantic>=2.6
Requires-Dist: rich>=13.7
Requires-Dist: ruff>=0.4
Requires-Dist: bandit>=1.7
Requires-Dist: pyright>=1.1
Requires-Dist: pytest>=8.0
Requires-Dist: pytest-cov>=5.0
Requires-Dist: pytest-json-report>=1.5
Requires-Dist: pip-audit>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: mcp>=1.2
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: anyio>=4.0; extra == "dev"
Provides-Extra: narration
Requires-Dist: google-genai>=1.0; extra == "narration"
Requires-Dist: python-dotenv>=1.0; extra == "narration"
Provides-Extra: verify
Requires-Dist: google-genai>=1.0; extra == "verify"
Requires-Dist: python-dotenv>=1.0; extra == "verify"
Requires-Dist: httpx>=0.27; extra == "verify"
Dynamic: license-file

# Skeptic

[![PyPI version](https://img.shields.io/pypi/v/skeptic-cli.svg)](https://pypi.org/project/skeptic-cli/)
[![Python versions](https://img.shields.io/pypi/pyversions/skeptic-cli.svg)](https://pypi.org/project/skeptic-cli/)
[![License: MIT](https://img.shields.io/pypi/l/skeptic-cli.svg)](LICENSE)
[![Golden Cases](https://github.com/HamzaShaikh17/Skeptic/actions/workflows/golden-cases.yml/badge.svg)](https://github.com/HamzaShaikh17/Skeptic/actions/workflows/golden-cases.yml)

**An independent engineering quality gate for Python code — AI-generated or
human-written.**

Skeptic doesn't take a change's word for it. It orchestrates a battery of
established, best-in-class tools — Ruff, Pyright, Bandit, pytest/coverage,
and pip-audit — into a single, evidence-backed pass/fail verdict, and
attributes every finding against a baseline so a change is judged only on
what it actually introduced. The result is a gate an AI coding agent, a CI
pipeline, or a human reviewer can be required to satisfy before a change
counts as done — with every failure traceable to the exact file, line, and
rule behind it.

---

## Table of contents

- [Why Skeptic](#why-skeptic)
- [Installation](#installation)
- [Prerequisites](#prerequisites)
- [Quick start](#quick-start)
- [CLI reference](#cli-reference)
- [`skeptic check` — the quality gate](#skeptic-check--the-quality-gate)
- [`skeptic diff` — change-aware evaluation](#skeptic-diff--change-aware-evaluation)
- [Configuration](#configuration)
- [Architecture findings and the Change Risk Score](#architecture-findings-and-the-change-risk-score)
- [AI provenance](#ai-provenance-skeptic-provenance)
- [Plain-language narration](#plain-language-narration-optional)
- [Adversarial verifier (`skeptic verify`)](#adversarial-verifier-skeptic-verify)
- [MCP server](#mcp-server-claude-code--cursor)
- [Golden Cases (`skeptic eval`)](#golden-cases-skeptic-eval)
- [Architecture of this repository](#architecture-of-this-repository)
- [Development](#development)
- [License](#license)

## Why Skeptic

Static analysis tools already exist. What's missing is a single, opinionated
verdict that ties them together, treats a pre-existing problem differently
from one a change just introduced, and hands back evidence rather than a
summary you have to trust.

- **One verdict, five tools.** Ruff, Pyright, Bandit, pytest/coverage, and
  pip-audit run together and roll up into a single pass/fail gate, each rule
  independently reported with the exact findings behind it.
- **Change-aware, not just repo-aware.** `skeptic diff` classifies every
  finding as `INTRODUCED`, `PRE_EXISTING`, or `RESOLVED` relative to a
  baseline, so a change is never blocked by a problem it didn't cause.
- **Structural checks, not just lint.** SOLID-principle violations (SRP,
  ISP, DIP) and cyclomatic complexity are detected on top of the standard
  tool findings.
- **A documented, transparent risk score.** Every contributor to the
  Change Risk Score is listed individually — never a black-box number.
- **AI-agent-ready.** A first-class MCP server exposes the same engine to
  Claude Code, Cursor, and other MCP-compatible agents, plus commit-level
  provenance tagging and an optional LLM-backed adversarial verifier.
- **CI-native.** Deterministic exit codes and a stable `--json` schema for
  every command.

## Installation

```bash
pip install skeptic-cli
```

Installs the `skeptic` console command.

Working from a local clone instead (for example, to pick up unreleased
changes)? `pip install -e .` does the same thing from source.

## Prerequisites

**Install and activate the target repository's own dependencies before
running `skeptic check`.** Skeptic shells out to `pyright` and `pytest`
using whichever Python environment is currently active — it does not
install the target repository's dependencies for you. If they aren't
installed, Pyright cannot resolve most imports and reports a flood of
`reportMissingImports` errors unrelated to real type safety, and `pytest`
fails to collect tests at all. Because the default gate is zero-tolerance
(`types.max_errors: 0`), this alone is enough to fail almost any real
repository.

The fix: `cd` into the target repository, activate its environment (or
otherwise ensure its dependencies are installed in the active environment —
`pip install -r requirements.txt`, `uv sync`, `poetry install`, etc.), then
run `skeptic check`. Ruff, Bandit, and pip-audit don't need this — they
analyze source and dependency manifests directly rather than resolving
imports.

## Quick start

```bash
# Generate a starter config in your repo
skeptic init /path/to/your/repo

# Run the gate
skeptic check /path/to/your/repo

# Machine-readable output, for CI or an agent to parse
skeptic check /path/to/your/repo --json
```

## CLI reference

| Command | Purpose |
| --- | --- |
| [`skeptic check`](#skeptic-check--the-quality-gate) | Run the full quality gate against a repository. |
| [`skeptic diff`](#skeptic-diff--change-aware-evaluation) | Evaluate a change against a baseline — INTRODUCED / PRE_EXISTING / RESOLVED attribution and a change-specific risk verdict. |
| [`skeptic eval`](#golden-cases-skeptic-eval) | Run Skeptic's own benchmark suite against a set of known-outcome fixtures. |
| [`skeptic verify`](#adversarial-verifier-skeptic-verify) | Generate and run adversarial attack cases against a running instance you control. |
| [`skeptic provenance`](#ai-provenance-skeptic-provenance) | Estimate what share of recent commits landed while an AI agent was using `skeptic-mcp`. |
| `skeptic init` | Write a starter `skeptic.yaml` into a target repository. |

Every command supports `--json` for machine-readable output with a stable
schema, alongside a formatted table/text view for interactive use.

## `skeptic check` — the quality gate

### Example

Given a repository with an unused import, a real type error, and a
vulnerable pinned dependency:

```
$ skeptic check .

Engineering Gate: FAIL

+----------------------------------------------------------------------------+
| Rule              | Status | Detail                                        |
|-------------------+--------+-----------------------------------------------|
| lint              | FAIL   | 1 lint findings (max allowed: 0)              |
| types             | FAIL   | 1 type errors (max allowed: 0)                |
| security_critical | PASS   | 0 critical security findings (max allowed: 0) |
| security_high     | PASS   | 0 high security findings (max allowed: 0)     |
| coverage          | PASS   | 88.9% coverage (min required: 0.0%)           |
| dependencies      | FAIL   | 12 critical/high CVEs (max allowed: 0)        |
+----------------------------------------------------------------------------+

lint findings:
  app.py:1  `os` imported but unused (F401)

types findings:
  app.py:14  Argument of type "Literal['not a number']" cannot be assigned to
  parameter "a" of type "int" in function "add" (reportArgumentType)
...
```

Exit code `1`. Resolve the findings (or explicitly relax `skeptic.yaml`) and
the same command exits `0` with `Engineering Gate: PASS` — suitable for
direct use in a CI job or a shell `&&` chain.

See [`docs/USAGE.md`](docs/USAGE.md) for the full walkthrough: reading
output, `--json`/CI integration, what each check does, and troubleshooting.

## `skeptic diff` — change-aware evaluation

`skeptic check` answers "is this repository healthy?" `skeptic diff`
answers a different, often more useful question: **what did this specific
change introduce, fix, or leave untouched?**

```bash
skeptic diff                  # working tree vs HEAD
skeptic diff --base main      # working tree vs the merge-base of HEAD and main
skeptic diff HEAD~1 HEAD      # two explicit refs, no working tree involved
```

Every finding from every adapter is attributed against a baseline:

| Status | Meaning |
| --- | --- |
| `INTRODUCED` | New in this change, in a file (and, where resolvable, a specific function/class) this change actually touched. |
| `PRE_EXISTING` | Present before this change and untouched by it — never blocks or warns, regardless of severity. |
| `RESOLVED` | Present in the baseline and gone in the current state. |

Attribution is identity-based (rule + file, refined to the enclosing
symbol when available) and survives line drift and file renames, so a
finding that merely moved doesn't read as a new one, and a pre-existing
critical issue never blocks a change that didn't cause it.

```
Comparing HEAD~1 -> HEAD

INTRODUCED (1):
  app.py:12 [high] Possible SQL injection vector through string-based query construction (B608)

Change Risk Score: MEDIUM (52.0/100)
  +32.0 introduced HIGH security finding
  +12.0 no related test file detected

Change Verdict: BLOCK
```

**Verdict policy:**
- `BLOCK` — an introduced critical/high-severity finding, or newly failing
  tests. Exit code `1`.
- `WARN` — an introduced medium-severity finding, or a significant test
  coverage gap. Exit code `0`.
- `PASS` — otherwise.

The change is also evaluated for related-test coverage (by naming
convention, honestly reported as "no related tests detected" rather than
"this change is untested" — the two are not the same claim) and dependency
manifest changes (added/removed/version-changed packages). The baseline
(and, for a two-ref comparison, the target) is always checked out into an
isolated `git worktree` — your working tree is never mutated.

`--json` emits the same underlying attribution, risk score, and verdict as
the text output — never two independently computed answers.

## Configuration

Edit `skeptic.yaml` in your repository root:

```yaml
lint:
  max_errors: 0
types:
  max_errors: 0
security:
  max_critical: 0
  max_high: 0
tests:
  max_failures: 0
coverage:
  min_percent: 80
dependencies:
  max_critical_cves: 0

# Optional, unset by default — see "Architecture findings" below. Neither
# blocks the gate until you uncomment it.
# architecture:
#   max_findings: 0
# ai_review:
#   max_risk_label: MEDIUM   # LOW | MEDIUM | HIGH
```

With no `skeptic.yaml`, every threshold defaults to zero-tolerance for
lint/types/security/tests/dependencies. `architecture` and `ai_review` are
the two exceptions — they stay off until explicitly configured (see below).

## Architecture findings and the Change Risk Score

`skeptic check` also runs four deterministic, tool-free structural checks —
SRP, ISP, DIP (`tool="solid"`), and McCabe cyclomatic complexity
(`tool="complexity"`, functions over 10 flagged by default). They appear in
the table output and `--json` (`solid_findings`/`complexity_findings`)
regardless of gating:

```
architecture findings (informational — not gated by default):
  app/god_service.py:11  class 'GodService' touches 3 unrelated external
  systems (database, email, http) via: ... (SRP)
```

Unlike lint/types/security/dependencies, these are **not gated by
default** — they're newer and haven't been broadly triaged the way an
established linter has, so upgrading shouldn't silently flip an existing
repository's gate from PASS to FAIL. Opt in explicitly:

```yaml
architecture:
  max_findings: 0   # gates both solid and complexity findings together
```

### Change Risk Score

Every `skeptic check` run (CLI table, `--json`, and MCP output) includes a
composite LOW/MEDIUM/HIGH label — security, regression (test health),
architecture, complexity, and test confidence, each scored 0–100 and always
shown alongside the label, not hidden behind it:

```
Change Risk Score: MEDIUM (38.2/100 - security=40.0, regression=0.0, architecture=45.0, complexity=10.0, test_confidence=100.0)
```

The weights and thresholds are a documented, first-pass heuristic (see
[`src/skeptic/core/risk_score.py`](src/skeptic/core/risk_score.py)), not an
empirically calibrated model — treat the label as a prioritization signal.
Gate on it explicitly if you want it to block:

```yaml
ai_review:
  max_risk_label: MEDIUM   # fails the gate if the label exceeds this
```

## AI provenance (`skeptic provenance`)

```bash
skeptic provenance /path/to/your/repo --since-ref HEAD~20
```

Estimates what share of recent commits landed while an AI agent was
actively using `skeptic-mcp` against a repository, correlating each
commit's timestamp against the local [MCP call log](#call-logging).

This is an approximation, not a precise record: `skeptic-mcp`'s tools are
read-only analysis, so the call log records when an agent called them, not
which lines it edited. A commit landing within `--window-minutes` (default
15) of a logged call is labeled `ai_generated` (pure addition) or
`ai_modified` (touched existing lines); everything else is `human`.
Requires no setup — with no MCP history for a repository, everything is
reported as `human`.

## Plain-language narration (optional)

```bash
skeptic check /path/to/your/repo --narrate
```

Sends each SOLID finding to an LLM for a short "why this matters and how to
fix it" explanation, printed under the finding. The model never originates
a finding or changes the verdict — it only narrates a finding a
deterministic check already produced. If narration fails (missing key,
network error, rate limit), `skeptic check` still runs and reports
normally, minus the narration text.

Requires `GEMINI_API_KEY`:

```bash
pip install -e ".[narration]"   # installs google-genai + python-dotenv
cp .env.example .env            # then fill in GEMINI_API_KEY
```

`.env` is loaded automatically and is gitignored — never commit it. The
model is configurable via `SKEPTIC_GEMINI_MODEL` if the default is
deprecated.

## Adversarial verifier (`skeptic verify`)

Generates and runs attack test cases against a **running instance you
control** — boundary values, invalid input, injection, auth bypass, IDOR,
concurrency (race conditions), and failure-mode (timeout) probes — and
reports pass/fail per category with the exact request that triggered each
result.

```bash
pip install -e ".[verify]"   # installs google-genai + python-dotenv + httpx
# start your own app locally first, e.g.: uvicorn app.main:app --port 8000

skeptic verify /path/to/your/repo --target http://localhost:8000
```

**Safety is a design constraint, not an afterthought:**
- **Read-only against your code.** It never writes to the repository path —
  only generates requests and sends them to `--target`.
- **The model never executes anything.** Attack cases are returned as
  structured data (method/path/headers/body) via a JSON schema, never as
  code — the only thing that ever runs is an HTTP request Skeptic's own
  code sends.
- **Refuses non-local targets by default.** `--target` must resolve to
  localhost or a private address (`10.x`, `172.16–31.x`, `192.168.x`,
  link-local), or the command exits immediately before generating
  anything — it sends real injection/auth-bypass/IDOR payloads, so it
  shouldn't be pointable at a service you don't own by accident. Pass
  `--allow-external` only against a target you're certain is yours.

`--diff-ref` (default `HEAD`) focuses attack-case generation on your
uncommitted changes when `path` is a git repository; it falls back to
general-purpose REST API cases otherwise. `passed: null` on a result means
the heuristic genuinely can't tell (for example, every concurrent request
to a mutating endpoint succeeding identically could indicate a race
condition or a correctly idempotent endpoint) — a human should look at
`detail`; it's never silently coerced to a pass.

Uses the same `GEMINI_API_KEY`/`.env` as `--narrate` above.

## MCP server (Claude Code / Cursor)

Skeptic's engine is also exposed as an MCP server, so an AI coding agent
can call it mid-task instead of a human running `skeptic check` by hand.
Same engine, same adapters, same gate — the CLI and the MCP server are both
thin clients of `skeptic.core`.

### Install

`pip install -e .` (see [Installation](#installation)) also installs the
`skeptic-mcp` console command, which starts the server over stdio.

### Configure your project

Add this to your project's `.mcp.json` (Claude Code) or `.cursor/mcp.json`
(Cursor) — the repository you want the agent to check, not Skeptic's own:

```json
{
  "mcpServers": {
    "skeptic": {
      "command": "skeptic-mcp",
      "args": []
    }
  }
}
```

`skeptic-mcp` must resolve on `PATH` in whatever environment your editor
launches subprocesses from, the same as any other locally installed MCP
server.

### Tools exposed

| Tool | Signature | Returns |
| --- | --- | --- |
| `skeptic_check` | `(repo_path: str)` | Full pass/fail verdict, evidence for every failing rule, and tool statuses. Equivalent to `skeptic check --json`. |
| `skeptic_get_findings` | `(repo_path: str, severity: str \| None)` | Every raw finding across all adapters, optionally filtered to one severity (`critical`/`high`/`medium`/`low`), not limited to findings tied to a failing gate rule. |
| `skeptic_gate_status` | `(repo_path: str)` | Same gate evaluation as `skeptic_check`, without the findings payload — a cheap pass/fail poll. |

The same [Prerequisites](#prerequisites) requirement applies: the target
repository's own dependencies need to be installed and active in the
environment the MCP server runs in, or `types` findings will mostly be
import-resolution noise.

### Call logging

Every call to any of the three tools is appended to a local, per-repository,
append-only JSONL log at `~/.skeptic/mcp_logs/<repo-name>-<hash>.jsonl` —
timestamp, session ID, tool name, arguments, and the full result. This log
is the basis for [AI provenance](#ai-provenance-skeptic-provenance)
tagging.

## Golden Cases (`skeptic eval`)

Skeptic's own benchmark suite: given a repository with a known engineering
failure, does Skeptic detect it? See
[`golden_cases/manifest.yaml`](golden_cases/manifest.yaml) and each case's
`README.md` for the fixture format and the current case set (security,
architecture, complexity, dependency, testing, regression, adversarial, and
change-level cases under `golden_cases/changes/`).

```bash
skeptic eval                          # runs golden_cases/, deterministic profile
skeptic eval --json                   # machine-readable, stable schema
skeptic eval --category security      # filter to one category
skeptic eval --compare benchmarks/v0.5.2.json   # regression check against a saved baseline
skeptic eval --changes                # also run the skeptic diff regression suite (slower, opt-in)
```

Adversarial cases (LLM-verifier-based, e.g. IDOR/auth-bypass/injection
against a live endpoint) are authored and schema-valid but not yet executed
by `eval` — `eval` reports how many are defined without spending an LLM
call on them. [`.github/workflows/golden-cases.yml`](.github/workflows/golden-cases.yml)
runs the deterministic profile (plus `--changes`) in CI and fails the build
only on a regression against the latest saved baseline in
[`benchmarks/`](benchmarks/), not on an absolute threshold.

## Architecture of this repository

See [`src/skeptic/core/models.py`](src/skeptic/core/models.py) for the
language-agnostic finding schema. Python-specific tool wrappers live in
[`src/skeptic/adapters/python/`](src/skeptic/adapters/python/) — adding
support for another language means adding a new adapter directory, not
rewriting the core engine.

## Development

```bash
pip install -e ".[dev]"
pytest tests/
```

Contributions are welcome. Please open an issue to discuss substantial
changes before submitting a pull request.

## License

[MIT](LICENSE) © Hamza Shaikh
