Metadata-Version: 2.4
Name: open-source-secret-watcher
Version: 0.1.0
Summary: Defensive-security / responsible-disclosure platform that scans public GitHub repos for exposed secrets.
Author: Secret Watcher contributors
License: MIT
Keywords: security,secrets,trufflehog,responsible-disclosure,defensive-security
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.7
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: pydantic>=2.6
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: jinja2>=3.1
Requires-Dist: httpx>=0.27
Requires-Dist: keyring>=24
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"

# Open Source Secret Watcher

A defensive-security and **responsible-disclosure** platform that discovers **public**
GitHub repositories, scans them for potentially exposed secrets using the official
[TruffleHog](https://github.com/trufflesecurity/trufflehog) CLI, classifies and scores
findings (reducing false positives with regex + entropy analysis), and helps repository
maintainers remediate.

Built for **education, portfolio use, defensive security, and responsible disclosure.**

---

## ⚠️ Safety principles (enforced in code, not just documented)

This tool is deliberately constrained so it cannot be misused as an attack tool:

- **Public repositories only.**
- It **never uses, authenticates with, sends requests using, or tests** a discovered
  credential, and **never** tries to determine whether one is active.
- The regex/entropy layers only decide whether a string *resembles* a known credential
  format. They make **no network calls**.
- **Raw secrets are never stored.** Only a masked preview (`prefix…last4`) and a one-way
  SHA-256 fingerprint are persisted. There is no config flag and no database column that
  holds the raw value — the capability does not exist in this build. The raw string lives
  in memory only long enough to compute the preview/fingerprint and run format checks.
- Raw secrets are **never logged**.
- Disclosure reports are **written to disk for a human to review and send** — nothing is
  emailed or posted automatically.

> The intent is to help owners fix leaks, not to collect or exploit credentials.

---

## Features

| Area | What it does |
|------|--------------|
| **Discovery** | GitHub Search API across configurable keywords, recent-creation window, pagination, rate-limit backoff, concurrent queries. Dry-run by default. |
| **Scanning** | Programmatic TruffleHog execution (async), JSON parsing, concurrent scans, retries, scan history, already-scanned tracking. |
| **Classification** | OpenAI, Anthropic, AWS, GitHub, Google, Stripe, Slack, Azure, generic, unknown. |
| **Regex validation** | Format/length/prefix checks per provider; rejects obvious placeholders; assigns confidence. |
| **Entropy analysis** | Shannon entropy with hex/base64-aware thresholds; combined entropy + regex confidence score. |
| **Risk scoring** | Critical / High / Medium / Low from category, confidence, exposure location, repo popularity, commit age, and exposure history. |
| **Storage** | SQLite + SQLAlchemy. Repo metadata, scan records, findings (masked). |
| **Notifications** | Email + Markdown responsible-disclosure templates with remediation guidance. Files only. |
| **Dashboard** | Live Rich terminal dashboard. |
| **Export** | CSV export of findings (masked). |

---

## Architecture

```
discover → queue(DB) → scan(TruffleHog) → parse(JSON) → validate(regex+entropy)
        → score(confidence + risk) → store(DB) → notify(files) / dashboard / export
```

```
secret_watcher/
├── config/        # Pydantic settings (TOML + env)
├── discovery/     # GitHub Search API client (+ offline mock) and orchestrator
├── scanner/       # TruffleHog runner (+ offline mock) and scan manager
├── parser/        # TruffleHog JSON → normalised findings
├── validators/    # regex_validator.py, entropy_validator.py, scoring.py
├── database/      # SQLAlchemy models, session, data-access helpers
├── notifications/ # remediation guidance, Jinja2 templates, reporter
├── dashboard/     # Rich terminal dashboard
├── cli/           # Typer CLI
├── security.py    # masking + fingerprint (the only code that touches a raw secret)
└── export.py      # CSV export
tests/             # pytest suite (pure modules + mocked adapters)
```

Stages communicate through the database, so they are decoupled and resumable. Pure logic
(validators, scoring, parser) does no I/O and is unit-tested. Network/process stages sit
behind interfaces with offline mocks, so the whole pipeline runs without a token or
TruffleHog via `--mock`.

---

## Installation

See [INSTALL.md](INSTALL.md) for full details. Quick version:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
# Install TruffleHog v3 (for real scans): https://github.com/trufflesecurity/trufflehog
cp config.example.toml config.toml         # edit as needed
export GITHUB_TOKEN=ghp_your_token         # for the GitHub Search API
```

Requires **Python 3.12+**.

---

## Usage

Every command accepts `--config PATH` and `--mock` (offline mocks, no token/TruffleHog).

```bash
# 1. Discover public repos (dry-run; add --queue to enqueue them)
secretwatch discover --queue

# 2. Scan the queue with TruffleHog
secretwatch scan

# 3. Generate responsible-disclosure reports (email + markdown) into reports/
secretwatch report

# 4. Live dashboard
secretwatch dashboard            # --once to render a single frame

# 5. Export findings to CSV (masked previews only)
secretwatch export findings.csv

# 6. Aggregate statistics
secretwatch stats
```

### Try it offline (no token, no TruffleHog)

```bash
secretwatch discover --mock --queue
secretwatch scan --mock
secretwatch dashboard --mock --once
```

---

## Sample disclosure report

See [`reports/samples/`](reports/samples/) for a full email + Markdown example. Excerpt:

```
  Repository:   https://github.com/example/env-sample
  File:         .env (line 3)
  Category:     AWS credential
  Confidence:   0.34 (CRITICAL risk)
  Reference:    AKIAIOS…MPLE   (masked — the full value is not stored)
```

## Sample dashboard

A captured render is in [`reports/samples/dashboard.txt`](reports/samples/dashboard.txt):

```
╭───── Overview ──────╮ ╭─────── By Severity ───────╮ ╭───── By Type ─────╮
│ Repositories  2     │ │ CRITICAL    1             │ │ AWS credential  2 │
│      Scanned  2     │ │ HIGH        3             │ │ Stripe key      2 │
│        Queue  0     │ │ MEDIUM      0             │ ╰───────────────────╯
│     Findings  4     │ │ LOW         0             │
╰─────────────────────╯ ╰───────────────────────────╯
Recent Detections: example/env-sample | Stripe key | HIGH | sk_live…LE12
```

---

## Testing

```bash
pytest -q
```

The suite covers regex/entropy/scoring/parser logic, database helpers, and a full
mock-driven pipeline that asserts **no raw secret is ever persisted** to the database,
reports, or CSV.

---

## Configuration

All behaviour is driven by `config.toml` (see `config.example.toml`). Notable knobs:

- `discovery.keywords`, `discovery.created_within_days`, `discovery.max_results`
- `scanner.trufflehog_path`, `scanner.max_concurrent_scans`, `scanner.max_retries`
- `entropy.*` thresholds, `scoring.*` weights and `scoring.min_confidence`
- `reports.reporter_name` / `reports.reporter_contact` for the disclosure signature

The GitHub token is read from `GITHUB_TOKEN` in the environment, never from the file.

---

## License

MIT. Use responsibly. You are responsible for complying with GitHub's Terms of Service
and all applicable laws when using this tool.
