Metadata-Version: 2.4
Name: tokenwatch-cli
Version: 0.1.0
Summary: Scan a repository for accidentally committed secrets
Author: tokenwatch contributors
License-Expression: GPL-2.0-only
Project-URL: Homepage, https://github.com/TheVoidThatConsumes/tokenwatch
Project-URL: Source, https://github.com/TheVoidThatConsumes/tokenwatch
Project-URL: Bug Tracker, https://github.com/TheVoidThatConsumes/tokenwatch/issues
Keywords: security,secrets,credentials,git,scanning,devsecops
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# tokenwatch

A local CLI that scans a repository for accidentally committed secrets — API keys, tokens, passwords, private keys, database connection strings — before they travel further than they should.

Runs against the current working tree, the full git history, or both. Exits non-zero on any finding or if the CI workflow has been tampered with, so it drops straight into a pipeline or pre-commit hook as a hard gate.

## Why tokenwatch?

Secrets leak into repositories constantly, usually by accident: a developer hardcodes an API key while debugging, then commits and pushes it. Deleting it from the latest commit doesn't remove it from git's history — the object database keeps every version of every file, and anyone with clone access can walk that history and find it. tokenwatch checks both the present and the past.

## How it works

Two independent detection layers, run together on every file:

- **Pattern matching** — regex signatures for known secret formats: AWS access keys, GitHub tokens (`ghp_`, `ghs_`, `gho_`), GitLab runner registration tokens (`GR1348`), JWT structure, PEM private key headers, database connection strings, Slack tokens, generic bearer tokens and API key assignments.
- **Entropy scoring** — a secondary signal for high-randomness strings that don't match any known pattern (threshold: 4.5 bits/char). Real secrets tend to look statistically random; human-written config values do not. File and module paths are filtered out before scoring. Catches secrets tokenwatch doesn't have a signature for yet.

Findings are redacted before they're ever stored or printed — only the first and last 4 characters of a matched secret are shown, the rest is masked.

## Install

**pip (recommended):**

```
pip install tokenwatch
```

**zip (no install, copy into project):**

Download the zip and extract the `tokenwatch/` folder anywhere inside your project. No dependencies beyond the Python standard library and `git` on PATH.

```
your-project/
  tokenwatch/
    scanner_core.py
    file_walker.py
    history_walker.py
    main.py
```

Any folder name works. The CLI figures out its own location automatically.

## Getting started

Verify the tool is working correctly in your environment:

**pip:**

```
tokenwatch verify
```

**zip:**

```
python tokenwatch/main.py verify
```

This runs both detection layers against synthetic inputs and confirms false-positive suppression is working. If everything passes, you're ready to scan.

## Usage

The examples below use the pip form. For the zip layout, replace `tokenwatch` with `python tokenwatch/main.py`.

**Verify the installation:**

```
tokenwatch verify
```

**Scan the working tree:**

```
tokenwatch scan .
```

**Scan working tree and full git history** — catches secrets that were committed and later deleted:

```
tokenwatch scan . --history
```

**Save a report** — same as scan but writes a timestamped JSON to `reports/` inside the scanned project:

```
tokenwatch report . --history
```

**Check the exit code when scripting:**

```
tokenwatch scan . --history
echo $?
```

0 means clean. 1 means findings present or workflow tamper detected.

## GitHub Actions integration

The first time you run `scan`, tokenwatch automatically generates `.github/workflows/tokenwatch.yml` if it doesn't already exist — no separate setup step needed.

You'll see:

```
tokenwatch: no workflow found — generated .github/workflows/tokenwatch.yml
  run: git add .github/workflows/tokenwatch.yml
       git commit -m 'add tokenwatch CI workflow'
       git push
```

Commit and push the generated file to activate the CI gate. Once active, the workflow runs `scan . --history` on every push and pull request.

The generated workflow differs by install method. The pip layout installs tokenwatch via pip in CI and calls the console script directly. The zip layout calls the script by path. Both are generated automatically — no manual editing needed.

`fetch-depth: 0` is set in the generated workflow and is required — without it, Actions performs a shallow clone and `--history` silently finds nothing.

To regenerate or restore a workflow manually:

```
tokenwatch init
tokenwatch init --force
```

`--force` overwrites an existing file and shows a diff first.

## Tamper detection

tokenwatch hashes its own workflow file when it generates it and stores that hash in `.tokenwatch_state` at your repo root. On every subsequent scan, it:

1. Recomputes the hash and compares — any content change is flagged.
2. Checks that a `run:` line still calls `scan . --history` — catches a weakened command even if the hash was manually updated to cover it.

A tamper warning is printed to stderr before the scan runs and causes the process to exit 1, the same as a real finding. To restore a tampered workflow:

```
tokenwatch init --force
```

Commit both `.tokenwatch_state` and the restored workflow file.

## Suppressing false positives

**Inline suppression** — append `# tokenwatch: ignore` to any line to exclude it from working-tree findings:

```
key = "AKIAABCDEFGHIJKLMNOP"  # tokenwatch: ignore
```

Applies to working-tree scans only. History findings cannot be suppressed inline — they exist in committed blobs that are immutable.

**Ignore file** — add a `.tokenwatchignore` at your project root, one glob pattern per line, `#` comments allowed:

```
tests/*
*.example
fixtures/*
```

Patterns match both the full relative path and the bare filename. Note: `**` recursive matching is not supported — `tests/*` only matches files directly inside `tests/`, not nested subdirectories.

## What gets skipped

tokenwatch automatically skips files and directories that produce noise rather than signal:

**Directories:** `.git`, `node_modules`, `__pycache__`, `.venv`, `venv`, `dist`, `build`, `.tox`, `.mypy_cache`, `.pytest_cache`, `vendor`, `target`

**Binary extensions:** image, audio, video, archive, compiled, and font formats, and database files (`.db`, `.sqlite`, `.sqlite3`)

**Minified and bundled assets:** any filename containing `.min.`, `.bundle.`, `.chunk.`, `.dev.`, or `.prod.` — these are machine-generated and produce large volumes of entropy false positives from webpack chunk paths and sourcemap hashes

**Binary content:** extensionless files are sniffed for null bytes and skipped if found

**Oversized files:** anything over 5 MB

## Exit codes

| Code | Meaning |
|------|---------|
| 0 | Clean — no findings, workflow intact |
| 1 | Findings present, or workflow tamper detected |

## Scope

tokenwatch deliberately does not:

- Watch a repository live or continuously
- Rotate or revoke leaked secrets automatically
- Make any network calls — all analysis is local, reading only from the filesystem and the local `.git` directory

## License

GPLv2 — see LICENSE.
