Metadata-Version: 2.5
Name: slopaudit
Version: 1.0.0
Summary: Verify AI-generated code: LLM-app security, hallucinated packages, stubs and spec drift.
Project-URL: Homepage, https://github.com/patsypppe/slopaudit
Project-URL: Repository, https://github.com/patsypppe/slopaudit
Project-URL: Issues, https://github.com/patsypppe/slopaudit/issues
Project-URL: Changelog, https://github.com/patsypppe/slopaudit/blob/main/CHANGELOG.md
Author: slopaudit contributors
License: MIT
License-File: LICENSE
Keywords: ai-generated-code,llm,sarif,security,static-analysis
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: rich>=13
Requires-Dist: tomli>=2; python_version < '3.11'
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# slopaudit

[![PyPI](https://img.shields.io/pypi/v/slopaudit.svg)](https://pypi.org/project/slopaudit/)
[![Python versions](https://img.shields.io/pypi/pyversions/slopaudit.svg)](https://pypi.org/project/slopaudit/)
[![CI](https://github.com/patsypppe/slopaudit/actions/workflows/ci.yml/badge.svg)](https://github.com/patsypppe/slopaudit/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/patsypppe/slopaudit/blob/main/LICENSE)
[![Good first issues](https://img.shields.io/github/issues/patsypppe/slopaudit/good%20first%20issue.svg?label=good%20first%20issues)](https://github.com/patsypppe/slopaudit/labels/good%20first%20issue)

**Check AI-written code before you trust it.** slopaudit is an open-source command-line tool for Python
that finds the problems AI assistants and LLM-powered apps tend to introduce, and that general scanners
were not built to look for.

- **Is it safe?** Does model output reach `eval`, a shell or a SQL query? Does untrusted input end up
  in a system prompt? Does an agent tool hand the model a shell?
- **Is it real?** Does that imported package exist, or is it invented (or one typo from a popular one)?
- **Is it finished?** Stubs, "rest of code here" comments, swallowed errors, truncated files.
- **Is it what you asked for?** Does the code meet each requirement in your spec, and did the change
  delete or weaken tests to get there?

It runs on your machine, never executes the code it scans, needs no account or API key, and has no telemetry.

## Install

You need Python 3.10 or newer. Pick one:

```bash
pipx install slopaudit             # recommended: an isolated command-line tool
uv tool install slopaudit          # same idea, if you use uv
python3 -m pip install slopaudit   # inside a virtual environment
```

Then check it works:

```bash
slopaudit --version
```

**Just try it, without installing anything:**

```bash
pipx run slopaudit scan .          # or: uvx slopaudit scan .
```

<details>
<summary>Don't have pipx, upgrading, uninstalling, or installing from source</summary>

| I want to | Run |
| --- | --- |
| Get pipx on macOS | `brew install pipx && pipx ensurepath` |
| Get pipx elsewhere | `python3 -m pip install --user pipx && python3 -m pipx ensurepath` |
| Upgrade | `pipx upgrade slopaudit`, or `python3 -m pip install --upgrade slopaudit` |
| Uninstall | `pipx uninstall slopaudit`, or `python3 -m pip uninstall slopaudit` |
| Install a specific version | `pipx install slopaudit==1.0.0` |
| Install the latest unreleased code | `pipx install git+https://github.com/patsypppe/slopaudit` |
| Hack on it | `git clone https://github.com/patsypppe/slopaudit && cd slopaudit && make install` (or open it in [Codespaces](https://codespaces.new/patsypppe/slopaudit)) |

On macOS and most Linux systems the command is `python3`; `python: command not found` only means that.
On Windows use `py` instead of `python3`.

</details>

**In CI**, no install step is needed. Use the GitHub Action:

```yaml
# .github/workflows/slopaudit.yml
name: slopaudit
on: [pull_request]
permissions: { contents: read, security-events: write }
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: patsypppe/slopaudit@v1.0.0
```

**Before every commit**, with [pre-commit](https://pre-commit.com):

```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/patsypppe/slopaudit
    rev: v1.0.0
    hooks:
      - id: slopaudit
```

## See it work

```console
$ slopaudit scan examples/vulnerable_agent
vulnerable_agent/app.py
  16   HIGH   LLM-08  Tool `run_command` passes a model-chosen argument to subprocess.run() (shell)
        ↳ line 14: tool argument `command` is chosen by the model
        ↳ line 16: passed to subprocess.run()
  21   HIGH   LLM-01  Untrusted input reaches the system prompt of client.messages.create()
        ↳ line 20: untrusted input request.args
        ↳ line 20: stored in `topic`
        ↳ line 21: placed in the system prompt of client.messages.create()
  27   CRITICAL   LLM-02  Model output reaches eval() without validation
        ↳ line 21: model output from client.messages.create()
        ↳ line 21: stored in `reply`
        ↳ line 27: passed to eval()
  30   MEDIUM   INC-02  `parse_invoice` has no implementation (pass)
  34   CRITICAL   SEC-01  Hardcoded credential assignment (q9X$…(21 chars))

1 files scanned: 2 critical, 2 high, 1 medium
```

Every flow finding shows the path the data took, so you can judge it in seconds. Each finding also has a
`fix:` line and a longer explanation: `slopaudit explain LLM-02`. No model is involved: slopaudit reads the
code's structure and follows values from where they come from to where they are used
([how it works](https://github.com/patsypppe/slopaudit/blob/main/docs/how-it-works.md)).

## Quickstart

```bash
slopaudit scan .                              # scan the current folder
slopaudit scan . --no-network                 # skip the PyPI package check
slopaudit scan --staged                       # only what you are about to commit
slopaudit scan . --format sarif -o out.sarif  # for GitHub code scanning
slopaudit scan src --spec SPEC.md             # check the code against a requirements file
slopaudit explain LLM-02                      # why a rule exists, and how to fix it
```

**From Python**, for example to check an assistant's answer before saving it:

```python
import slopaudit

blockers = [f for f in slopaudit.scan_source(answer) if f.severity >= slopaudit.Severity.HIGH]
```

New to it? The [getting-started guide](https://github.com/patsypppe/slopaudit/blob/main/docs/getting-started.md) takes five minutes, and has a
ten-line test file that shows every kind of finding. Adopting on a big
codebase? Start with a [baseline](https://github.com/patsypppe/slopaudit/blob/main/docs/suppressing-findings.md#3-a-baseline-for-existing-code) so only new
problems fail.

## What it checks

| Family | Rules | Catches |
| --- | --- | --- |
| **LLM-app security** | LLM-01 to LLM-06, LLM-08, LLM-09, LLM-12 | Prompt-injection paths, model output reaching `eval`, shell, SQL, file paths and URLs, over-permissive agent tools, destructive tools with no confirmation, unbounded model calls |
| **Secrets** | SEC-01 | Hardcoded API keys, tokens and private keys (masked in all output) |
| **Hallucinated packages** | HAL-01 to HAL-04 | Imports that do not exist on PyPI, brand-new packages, typosquats, undeclared dependencies |
| **Unfinished code** | INC-01, INC-02, INC-04, INC-05, INC-06, INC-08 | Placeholder comments, stubs, swallowed exceptions, chat text pasted into files, many TODOs, truncated files |
| **Intent and consistency** | SPC-02, DRF-01 to DRF-03, REG-01, REG-02 | Unmet requirements, style drift from the rest of the repo, deleted or skipped tests |

The full list, with severity and fix advice for each rule, is the [rules reference](https://github.com/patsypppe/slopaudit/blob/main/docs/rules.md).
Beyond scanning, there are optional model-assisted commands: [`gen-tests`](https://github.com/patsypppe/slopaudit/blob/main/docs/beyond-scanning.md#generated-tests)
and [`fix`](https://github.com/patsypppe/slopaudit/blob/main/docs/beyond-scanning.md#repair-loop) (a dry run unless you pass `--apply`).

## How well does it work?

Three checks, each reproducible from this repository. None is a guarantee; the caveats matter.

**1. Labelled corpus** (`slopaudit bench`, 34 files). Every rule scores 1.00 precision; recall is 0.91,
with two labelled misses (a value that crosses a function boundary, or goes through `self`). The corpus is
small and was used to tune the rules, so treat it as a regression gate.

**2. Other scanners on that corpus** ([full report](https://github.com/patsypppe/slopaudit/blob/main/docs/benchmarks.md)): files flagged at medium
severity or higher, run as external programs.

| Tool | LLM-app security (15) | Secrets (1) | Incomplete code (4) | Hallucinated packages (1) | False alarms on 13 safe files |
| --- | ---: | ---: | ---: | ---: | ---: |
| slopaudit | 12 | 1 | 3 | 1 | 0 |
| Bandit | 6 | 0 | 0 | 0 | 1 |
| Semgrep, public python rules | 8 | 0 | 2 | 0 | 2 |
| Semgrep, public ai rules | 12 | 0 | 0 | 0 | 6 |
| vibechck | 0 | 0 | 2 | 0 | 1 |

The corpus targets what slopaudit was built for, so it flatters slopaudit. Bandit and Semgrep are broader
tools not designed for these cases, and Semgrep's `ai` rules are SDK best-practice lints. Semgrep also
catches the two cross-function cases slopaudit misses, because it flags any dynamic `eval`.

**3. Mature packages as a false-positive control** ([report](https://github.com/patsypppe/slopaudit/blob/main/docs/real-world-check.md)): 1.5 million lines
across twelve well-reviewed packages, including the OpenAI, Anthropic, LangChain and LiteLLM SDKs.
**0 critical or high findings** after tuning. The first run had 122 critical findings and about 900 medium
ones; fixing them is in the [changelog](https://github.com/patsypppe/slopaudit/blob/main/CHANGELOG.md). About 25 medium and a few hundred low findings remain.
This measures noise only: those packages are not AI-generated.

## Honest limits

- **A clean scan is not proof of safety.** The flow rules follow values inside one function. They miss a
  value returned from a helper and `eval`-ed elsewhere, and values stored on `self`
  ([how it works](https://github.com/patsypppe/slopaudit/blob/main/docs/how-it-works.md#what-it-cannot-see)).
- **Without a model, the spec check compares names and words.** It can show code exists, not that it behaves
  correctly. In the demo it reports a requirement as met while the code does the opposite.
- **Experimental parts.** `scan --spec --llm`, `gen-tests`, `fix` and the Docker sandbox are tested against
  local stand-ins, not live services, so they are outside the 1.0 stability promise
  ([what 1.0 promises](https://github.com/patsypppe/slopaudit/blob/main/docs/decisions/0002-what-1-0-promises.md)).
- **Python only.** JavaScript and TypeScript files are skipped, and the scan says so when it finds nothing to read.
- **Linux CI only.** Installing and running on macOS has been checked by hand; Windows is untested
  ([#6](https://github.com/patsypppe/slopaudit/issues/6)).

## Documentation

| | |
| --- | --- |
| [Getting started](https://github.com/patsypppe/slopaudit/blob/main/docs/getting-started.md) | Install, first scan, reading output |
| [Configuration](https://github.com/patsypppe/slopaudit/blob/main/docs/configuration.md) | Every `slopaudit.toml` setting |
| [CI integration](https://github.com/patsypppe/slopaudit/blob/main/docs/ci-integration.md) | GitHub Action, pre-commit, GitLab |
| [Suppressing findings](https://github.com/patsypppe/slopaudit/blob/main/docs/suppressing-findings.md) | Comments, baselines, ignores |
| [Beyond scanning](https://github.com/patsypppe/slopaudit/blob/main/docs/beyond-scanning.md) | Spec checks, generated tests, drift, regression, repair |
| [Python API](https://github.com/patsypppe/slopaudit/blob/main/docs/python-api.md) | Use it from your own code or agent |
| [Privacy and safety](https://github.com/patsypppe/slopaudit/blob/main/docs/privacy-and-safety.md) | What runs, what is sent, the sandbox |
| [How it works](https://github.com/patsypppe/slopaudit/blob/main/docs/how-it-works.md) | Pipeline and taint tracking |
| [FAQ and troubleshooting](https://github.com/patsypppe/slopaudit/blob/main/docs/faq.md) | Common questions and fixes |
| [All docs](https://github.com/patsypppe/slopaudit/blob/main/docs/README.md) | The full index |

## Contributing

Contributions are welcome: rules, false-positive reports, docs and tests. The most valuable thing you can
do is **report a false positive or a miss with the smallest code that shows it**; every confirmed report
becomes a regression test.

**Looking for something to work on?** The [contributor roadmap](https://github.com/patsypppe/slopaudit/issues/85)
lists every open issue by area and difficulty. Each one says why it matters, which files to change, and when
it is done:

| Start here | |
| --- | --- |
| [Good first issues](https://github.com/patsypppe/slopaudit/labels/good%20first%20issue) | Small and well-defined: aliases, output formats, docs, test samples |
| [Help wanted](https://github.com/patsypppe/slopaudit/labels/help%20wanted) | New rules and integrations |
| [Advanced](https://github.com/patsypppe/slopaudit/labels/advanced) | The flow tracker and bigger design questions; discuss before coding |

Comment on an issue before you start so nobody duplicates work. Setup takes one click in
[Codespaces](https://codespaces.new/patsypppe/slopaudit), or `make install` locally. Start with [CONTRIBUTING.md](https://github.com/patsypppe/slopaudit/blob/main/CONTRIBUTING.md) and
[writing a rule](https://github.com/patsypppe/slopaudit/blob/main/docs/writing-rules.md), and please follow the [code of conduct](https://github.com/patsypppe/slopaudit/blob/main/CODE_OF_CONDUCT.md).
Questions go in [Discussions or issues](https://github.com/patsypppe/slopaudit/blob/main/SUPPORT.md); security problems in
[SECURITY.md](https://github.com/patsypppe/slopaudit/blob/main/SECURITY.md).

## License

[MIT](https://github.com/patsypppe/slopaudit/blob/main/LICENSE). slopaudit is independent work: it takes no code, rules, messages or structure from other
tools. The [clean-room record](https://github.com/patsypppe/slopaudit/blob/main/docs/clean-room.md) explains how, and why other scanners are only ever *run*,
never copied. Thanks to the authors of Bandit, Semgrep and vibechck for the prior art that made the
comparison possible.
