Metadata-Version: 2.4
Name: depverify
Version: 0.1.1
Summary: Verify Python/PyPI dependencies referenced in LLM-generated code against reality. Advisory only, never blocking.
Author: depverify
License: MIT
Keywords: llm,hallucination,slopsquatting,pypi,supply-chain,dependencies
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Intended Audience :: Developers
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Provides-Extra: flask
Requires-Dist: flask>=2.0; extra == "flask"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pandas; extra == "dev"
Dynamic: license-file

# depverify

Verifies Python/PyPI dependencies referenced in LLM-generated code against
reality. Advisory only: it produces a JSON report. It never installs,
deletes, halts, or quarantines anything.

## Why

LLM coding models hallucinate package names. USENIX Security 2025
(Spracklen et al.) measured ~19.7% of recommended packages as hallucinated
across 16 models (~5% commercial, ~21% open-source); a May 2026
replication found frontier models compressed to ~4.6-6.1%, but quantized
local models remain far worse. Attackers register hallucinated names on
PyPI with malware ("slopsquatting"). depverify checks whether a name an
LLM told you to `pip install` or `import` actually exists, and, if it
does, whether it looks reputable.

## Install

```
pip install depverify              # core
pip install depverify[flask]       # + flask itself, for the HTTP integration below
```

`depverify[flask]` installs the `flask` dependency only. The HTTP
integration itself (`integrations/cortexfeed/flask_blueprint.py`) is
source-only, deliberately not part of the installed `depverify` package
-- see [Flask HTTP integration](#flask-http-integration) below.

Dev: `pip install -e ".[flask,dev]"` (pytest, pandas, flask).

## Usage

```
depverify check path/to/answer.txt
cat answer.txt | depverify check -
depverify check answer.txt --json
depverify check answer.txt --symbols
```

A successful check's exit code is always `0`, regardless of what it finds
(`NOT_FOUND`, `suspicious`, etc.) -- this is an advisory tool, not a
linter that fails your build over the *contents* of a report. That
guarantee covers verdicts, not invocation: a CLI usage error (bad
arguments, a missing/unreadable input file, non-UTF-8 input) is reported
as a clean one-line message on stderr and exits non-zero, the same way
`depverify check` with a missing required argument already does -- it is
not itself a verdict, so it is not covered by the "always 0" guarantee.

`--symbols` additionally checks statically-resolvable attribute chains
(e.g. `pandas.DataFrame.flatten`) against packages already installed in
the local environment. **Security note:** this is the one depverify
feature that imports locally installed code, which can execute that
code's import-time side effects -- see
[Security note: `--symbols` and local code execution](#security-note---symbols-and-local-code-execution)
before enabling it. Off by default; normal `depverify check` never
imports or executes anything locally installed.

### Library

```python
from depverify import verify_text

report = verify_text(llm_answer_text)
print(report.to_json())
```

## Verdict model

Per detected dependency:

```
verdict ∈ EXISTS | NOT_FOUND | STDLIB_SKIPPED | LOCAL_OR_UNKNOWN | CANT_VERIFY
```

`EXISTS` packages additionally carry a `risk` object:
`level ∈ ok | suspicious`, with
`reasons ⊆ {young_package, low_downloads, near_name:<popular-package>, version_missing}`.

**Existence and reputation are orthogonal.** A package that EXISTS can
still be `suspicious`. depverify never folds "sketchy" into "missing" --
those are different questions with different implications.

### The core asymmetry

A name that 404s on PyPI is interpreted differently depending on how it
was detected:

- Detected via an explicit `pip install <name>` line -> **NOT_FOUND**.
  The LLM told you to install this; it doesn't exist; that's the signal
  slopsquatting exploits.
- Detected via `import <name>` -> **LOCAL_OR_UNKNOWN**. It might be your
  own module, a relative import target, or a name that isn't on PyPI at
  all for a legitimate reason. Flagging every unresolvable import as
  "not found" would bury real warnings in false positives from ordinary
  project code (`import myapp.models`, etc.).

## Report shape

```json
{
  "packages": [
    {"name_raw": "cv2", "resolved": "opencv-python", "source": "import",
     "verdict": "EXISTS", "risk": {"level": "ok", "reasons": []}},
    {"name_raw": "pdfreader-pro", "resolved": "pdfreader-pro", "source": "pip_install",
     "verdict": "NOT_FOUND", "risk": null}
  ],
  "summary": {"checked": 2, "not_found": 1, "suspicious": 0, "cant_verify": 0}
}
```

`summary.cant_verify` counts only package-level `CANT_VERIFY` verdicts. It
does not include symbol-level `CANT_VERIFY` results (e.g. a `--symbols`
chain that couldn't be checked because the package isn't installed
locally) -- those live under each package's own `symbols` list (see
`--symbols` below) and are a separate count from the top-level summary.
This is deliberate: `summary()` predates the symbol-checking feature and
was kept unchanged so existing consumers of `Report.summary()`/`to_dict()`
see byte-identical output when `check_symbols` is left at its default.

## Reputation signals (EXISTS packages only)

- **Age**: days since the earliest PyPI release.
- **Downloads**: last-30-day count from pypistats.org. If that API is
  unreachable or rate-limited, depverify silently omits download-based
  reasons -- a missing download count is never treated as CANT_VERIFY.
- **Near-name**: `difflib.SequenceMatcher.ratio() >= 0.88` (tunable, see
  `reputation.py`) against a vendored list of popular package names
  (`depverify/top_packages.json`), flagged as `near_name:<popular-package>`
  when the candidate itself isn't already a popular package.

Default suspicion rule (a tunable default, not ground truth):
`suspicious` if `(age < 60 days AND downloads < 1000/month) OR any near_name hit`.
Tune the constants in `depverify/reputation.py` for your risk tolerance.

## What this does NOT do

- **No package-content scanning.** depverify checks whether a name exists
  and looks reputable by metadata -- it does not download, sandbox, or
  static-analyze package code. For that, see dedicated tools like Socket
  or Snyk.
- **No blocking.** Nothing here gates a pip install, a CI job, or an LLM
  response. It's a report.
- **No npm/yarn.** Python/PyPI only, by design.
- **No LLM-based judging.** Every verdict is deterministic: extraction is
  regex/AST, existence is a PyPI lookup, reputation is arithmetic over
  metadata. No model calls anywhere in this codebase.

## Security note: `--symbols` and local code execution

**Normal `depverify check` (no `--symbols`) never imports or executes any
locally installed package code.** Existence checks are PyPI metadata
lookups over HTTP; reputation checks are pypistats.org lookups and
arithmetic. Nothing in that path touches your local Python environment's
installed packages, and nothing ever executes the LLM-generated code
being scanned.

`--symbols` is different, and deliberately scoped narrowly because of it:

- To check whether an attribute chain like `pandas.DataFrame.flatten`
  really exists, `--symbols` uses `importlib.import_module()` to import
  the already-installed package and `getattr()` to walk the chain.
  Importing a Python module runs that module's top-level code -- this is
  ordinary Python behavior, not something depverify adds, but it means
  `--symbols` is the one code path in this tool that executes local code
  as a side effect of scanning.
- **This is not the same as executing the LLM's code.** `--symbols` never
  runs the snippet being checked; it only imports packages by name using
  `importlib`, and only ever calls `getattr`/`hasattr` on the resulting
  module or class objects -- never instantiates a class, never calls a
  function or method. (Confirmed in `tests/test_symbols.py`: walking a
  chain that reaches a property or method never triggers the property
  getter, the method body, or the class's `__init__`.)
- **`--symbols` never installs anything.** If a package exists on PyPI
  but is not already installed locally, the result is `CANT_VERIFY`, not
  an install-then-import. depverify's "never installs, deletes, halts, or
  quarantines anything" guarantee holds for `--symbols` too.
- **The residual risk:** if a malicious or already-compromised package
  happens to be installed in the same environment running depverify,
  `--symbols` will trigger that package's import-time code the same way
  a plain `import thatpackage` anywhere else in that environment would.
  depverify does not sandbox, isolate, or vet locally installed packages
  before importing them for a symbol check.
- **Consequently: only enable `--symbols` in an environment where the
  installed package set is already trusted** -- the same trust you'd
  already extend to running `python -c "import <installed package>"` in
  that environment. Do not run `--symbols` as a way to safely inspect an
  environment whose installed packages you don't already trust, and do
  not present or rely on `--symbols` as a sandbox or security boundary
  around untrusted local installs -- it is not one.

## Limitations

- **Dynamic imports are not detected.** `importlib.import_module("name")`
  string literals are invisible to the AST/regex extraction in
  `extract.py`. Only literal `import x` / `from x import y` statements
  and `pip install` lines are found.
- **Only pip-style dependency declarations are parsed.** `setup.py`,
  `pyproject.toml` (`[project.dependencies]`), Conda environment files,
  and Poetry's `pyproject.toml` dependency tables are not parsed.
- **The import-name -> distribution-name mapping table is incomplete by
  nature.** `depverify/mapping_table.py` is hand-curated (57 entries as
  of this writing) and will always miss some real-world aliases. Unmapped
  import names fall back to querying PyPI with the import name as-is,
  which works for the common case (import name == distribution name) but
  not for every alias.
- **Verification is point-in-time.** A name that 404s right now can be
  registered on PyPI minutes later -- including by an attacker watching
  for exactly this kind of hallucinated name (slopsquatting). Nothing
  here caches a "safe" verdict indefinitely; the cache TTLs (24h for
  EXISTS, 6h for NOT_FOUND) reflect that names that don't exist yet are
  the more time-sensitive case.
- **EXISTS ≠ safe: compromised legitimate packages are invisible to this
  tool.** A package that exists, is old, and has millions of downloads
  can still ship malware in a compromised release. depverify's existence
  and reputation checks say nothing about supply-chain compromise of an
  otherwise-legitimate package.
- **`depverify/top_packages.json` was hand-vendored, not fetched live,**
  because the environment this project was built in could not reach
  `raw.githubusercontent.com`, `api.github.com`, or any CDN mirror (only
  `pypi.org` / `files.pythonhosted.org` / bare `github.com` were
  reachable). It's a ~450-name curated list of well-known packages
  written from training knowledge, not the real top-5000
  `hugovk/top-pypi-packages` dataset. Run `scripts/fetch_top_packages.py`
  from a network with GitHub raw-content access to regenerate the real
  list before relying on near-name detection for anything beyond obvious
  cases.
- **Attribute-chain verification (`--symbols`) only covers simple,
  statically-resolvable chains rooted directly in an imported name**
  (e.g. `pandas.DataFrame.flatten`). It does not do instance attribute
  inference, signature/arity checking, type inference, `.pyi` stub
  parsing, or follow chains past a function call
  (`requests.get(url).json` is not checked past `requests.get`) or past a
  reassigned import. See `depverify/symbols.py`'s module docstring for
  the full scope, and
  [Security note: `--symbols` and local code execution](#security-note---symbols-and-local-code-execution)
  before enabling it -- unlike the rest of depverify, it imports locally
  installed code.

## Flask HTTP integration

`integrations/cortexfeed/flask_blueprint.py` is a small, source-only Flask
Blueprint exposing `verify_text` over HTTP (`POST /verify`). It is
**not** part of the installed `depverify` package -- `pip install
depverify[flask]` only pulls in the `flask` dependency it needs, not this
file itself. That's intentional: this blueprint is meant to be copied
into your own Flask app (it imports only from `depverify`, nothing from
`cortexfeed`, so it has no hidden dependency on the project it's named
after), not imported from a `pip`-installed depverify. To use it, copy
`integrations/cortexfeed/flask_blueprint.py` from a source checkout into
your own project and register the blueprint it defines:

```python
from flask_blueprint import depverify_bp   # after copying the file in
app.register_blueprint(depverify_bp)
```

See the file's own module docstring for the request/response shape.

## Project layout

```
depverify/
├── depverify/           # the library + CLI -- this is what `pip install depverify` ships
├── integrations/cortexfeed/   # source-only Flask blueprint; copy into your app, not pip-installed
├── tests/                # unit tests (mocked, zero network) + eval/ (live network)
└── scripts/               # fetch_top_packages.py: regenerate the vendored top-package list
```

## Development

```
pip install -e ".[flask,dev]"
pytest tests/ --ignore=tests/eval     # unit tests, zero network
python tests/eval/run_eval.py          # eval, live network against PyPI/pypistats
```
