Metadata-Version: 2.4
Name: spyglass-pro
Version: 1.1.0
Summary: Spyglass-Pro — unified OSINT superpower wiring Spyglass-OSINT, InstaCloak, and Lens-Extender into one platform
Author-email: TeamLeader <teamleadertrue@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Ahmad170412/Spyglass-PRO
Project-URL: Source, https://github.com/Ahmad170412/Spyglass-PRO
Project-URL: Issues, https://github.com/Ahmad170412/Spyglass-PRO/issues
Keywords: osint,recon,unified,instagram,ip,investigation
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: Topic :: Security
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: flask>=3.0
Requires-Dist: rich>=13.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: httpx>=0.27
Requires-Dist: phonenumbers>=8.13
Requires-Dist: waitress>=2.0
Provides-Extra: dev
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Dynamic: license-file

<div align="center" markdown="1">

<img src="https://raw.githubusercontent.com/Ahmad170412/Spyglass-PRO/main/assets/logo.svg" alt="Spyglass-Pro logo" width="320"/>

# Spyglass-Pro

### One dashboard. Three recon engines. **Zero packets sent to your target.**

[![CI](https://github.com/Ahmad170412/Spyglass-PRO/actions/workflows/ci.yml/badge.svg)](https://github.com/Ahmad170412/Spyglass-PRO/actions)
[![Release](https://img.shields.io/github/v/release/Ahmad170412/Spyglass-PRO?color=6366f1&label=release)](https://github.com/Ahmad170412/Spyglass-PRO/releases)
[![PyPI version](https://img.shields.io/pypi/v/spyglass-pro.svg)](https://pypi.org/project/spyglass-pro/)
[![License: MIT](https://img.shields.io/badge/license-MIT-06b6d4.svg)](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-3776ab?logo=python&logoColor=white)](https://www.python.org/)
[![Passive only](https://img.shields.io/badge/recon-passive--only-6366f1?logo=shield&logoColor=white)](#passive-only)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CONTRIBUTING.md)

[**Quick Start**](#quick-start) · [**Features**](#features) · [**Architecture**](#architecture) · [**Configuration**](#configuration) · [**API**](#api) · [**Changes**](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CHANGELOG.md) · [**Contributing**](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CONTRIBUTING.md) · [**Security**](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/.github/SECURITY.md) · [**Audit**](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/AUDIT-2026-10.md)

</div>

---

![Spyglass-Pro overview — stat cards and the per-module data-source grid](https://raw.githubusercontent.com/Ahmad170412/Spyglass-PRO/main/docs/screenshots/dashboard-overview.png)

## Why Spyglass-Pro?

Most OSINT tooling is a pile of scripts that assume you already know what to
run, in what order, and how to read the output. Spyglass-Pro is a single
dashboard that answers a different question: **what is actually knowable about
a target, and how do I know the answer is real?**

- **Three engines, one interface.** Spyglass-OSINT, a stealth social harness,
  and Lens-Extender are merged behind a single job pool that runs them in
  parallel and correlates the entities they each find.
- **Passive by construction, not by policy.** Every probe that touched a
  target was *deleted*, not disabled — no flag, env var or config path brings
  it back. The [audit](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/AUDIT-2026-10.md#active-scanning-removed) lists every
  one so it can't quietly reappear.
- **It refuses to lie to you.** A follower list behind a login is reported as
  *walled with a reason*, never as an empty list. A recon leg that can't run
  is named in `/api/status`. A partial result never renders like a complete one.
- **Honest degradation is a feature, not a bug report.** Run it bare and it
  tells you exactly which panels are breach-only instead of quietly showing
  you nothing.

> **Built for people who need the truth.** OSINT tools are dual-use. Only
> investigate systems and accounts you own or have explicit permission to
> test. See [Responsible use](#responsible-use).

---

## Table of Contents

- [What's new in 1.1](#whats-new-in-11)
- [Quick Start](#quick-start)
- [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Optional CLI engines](#optional-cli-engines)
- [Passive-only](#passive-only)
- [Configuration](#configuration)
- [API](#api)
- [Cases](#cases)
- [Testing](#testing)
- [Platforms](#platforms)
- [Project structure](#project-structure)
- [Contributing](#contributing)
- [Changelog](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CHANGELOG.md)
- [Responsible use](#responsible-use)
- [License](#license)

---

## What's new in 1.1

1.1 is a hardening release: three new **keyless** social sources, a security
pass over the seams between versions, and honest answers where 1.0 was thin.
The full list is in the [changelog](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CHANGELOG.md).

- **Keybase** joins the social panel — keyless, and the strongest correlation
  source in the tree: it returns the account's cryptographically-**proven**
  identities (Twitter, GitHub, Reddit, Hacker News, DNS, web), claimed by the
  user's own key. That is a fact, not a guess from a shared handle.
- **GitHub and GitLab** are keyless too. Both report the budget tier their
  *response* actually came back with (GitHub 60 req/h, GitLab 500 req/min
  unauthenticated); set `GITHUB_TOKEN` / `GITLAB_TOKEN` to raise it, and the
  panel says which tier it got instead of assuming.
- **The case store keeps its promise.** With `SPYGLASS_STORE_KEY` set, the
  entity index is now encrypted alongside the result snapshot — before, the
  index held the same domains, IPs and emails in cleartext, leaking exactly
  what the encryption existed to protect.
- **The default install is closed off.** With no token, the dashboard answers
  only loopback `Host`s (DNS-rebinding defense), refuses cross-site /
  foreign-`Origin` browser requests, masks proxy credentials in every
  response, and sets a `Content-Security-Policy` plus the usual framing and
  sniffing headers.
- **IPv6 works.** The `ip` module accepted an IPv6 literal and then always
  failed to resolve it; it resolves now, and a malformed address is refused
  instead of accepted.
- **Real phone formats pass, flag-shaped input does not** — `+1 415 555 0100`
  and `(415) 555-0100` are accepted; a leading `-` is not.
- **Windows and CI.** Broader Windows tool fallbacks (winget/choco), a Windows
  test job, and a `mypy` job that actually runs in CI.

---

## Quick Start

Copy-paste ready. No API keys, no accounts, no configuration.

```bash
pip install spyglass-pro
spyglass-pro
```

Or from source:

```bash
git clone https://github.com/Ahmad170412/Spyglass-PRO.git && cd Spyglass-PRO
python3 -m venv .venv && source .venv/bin/activate
pip install -e .

python run.py
```

Open <http://127.0.0.1:5000>.

That's it — the keyless panels (**Website**, **IP Address**, **Full
Investigation**, and the social adapters **GitLab**, **GitHub**, **Keybase**,
**Mastodon**, **Bluesky**) work immediately. The **email**, **username** and
**phone** panels need optional CLI engines. The one-line installer ships with a
**source checkout**, not the wheel, so use it from the clone above:

```bash
python scripts/install_tools.py --check    # see what's missing
python scripts/install_tools.py            # install the missing ones
```

If you installed from PyPI (no checkout), install the engines directly — the
installer's docstring lists the exact command per tool and platform. The Python
ones go through `pipx` (isolated venvs); `blackbird` is a git clone and
`phoneinfoga` a release binary, so the script is the reliable path when you
have the source.

`run.py` adds the venv, pipx and blackbird locations to `PATH` itself, so the
server finds your engines even when launched from cron, a container or a
launch agent instead of a login shell.

---

## Features

| Feature | What it actually does |
| --- | --- |
| **Website** | DNS records, SPF/DMARC/CAA, passive subdomains from crt.sh + CertSpotter + HackerTarget + Wayback, WHOIS with IANA referral-following, history |
| **Email** | holehe + user-scanner + blackbird sweeps, cross-tool agreement, LeakCheck + Scylla breach data |
| **Username** | sherlock + maigret + user-scanner + blackbird, with four-way overlap categories |
| **Phone** | phoneinfoga + ignorant, plus breach correlation |
| **IP Address** | InternetDB, RDAP, RIPEstat, reverse DNS, DNSBL, Shodan |
| **Dark web** | Ahmia onion-index search, LeakCheck + Scylla breach data, keyed IntelX + HIBP |
| **Investigation** | Runs every engine in parallel and correlates shared emails, IPs, domains and usernames |
| **OPSEC** | Local self-check: your real IP, proxy status, DNS-leak exposure, timezone/geo fingerprint risk |
| **Metadata** | ExifTool + PDF/OOXML/image fallbacks, MD5/SHA-256, GPS, with file-type reconciliation |
| **Social** | Instagram (stealth browser) + GitLab, GitHub, Keybase, Mastodon and Bluesky (keyless, instant), one registry-driven panel |
| **Cases** | Encrypted-at-rest run history with entity diffing, timelines and dossier export |

---

## Architecture

```mermaid
graph LR
    UI[Dashboard<br/>Flask + vanilla JS] --> AUTH[[Auth gate<br/>Bearer token]]
    AUTH --> POOL[Job pool<br/>queue cap · 429 · timeouts]
    POOL --> SG[spyglass_osint<br/>email · username · phone<br/>website · ip · darkweb<br/>opsec · metadata]
    POOL --> LENS[lens_extender<br/>IP / domain recon]
    POOL --> SOC[social harness<br/>registry-driven adapters]
    SG --> STORE[(Case store<br/>Fernet at rest)]
    LENS --> STORE
    SOC --> STORE
    SG -. shells out to .-> CLI[holehe · sherlock<br/>maigret · ignorant · blackbird]
    UI --> EXT[(JSON / CSV / MD<br/>export artifacts)]

    style POOL fill:#1e293b,stroke:#6366f1,color:#e2e8f0
    style STORE fill:#1e293b,stroke:#06b6d4,color:#e2e8f0
    style UI fill:#1e293b,stroke:#8b5cf6,color:#e2e8f0
```

Every box under `POOL` only ever talks to third-party services. There is no
edge from any of them to the target.

---

## Installation

**Requirements:** Python 3.10+ (tested on 3.10–3.13 in CI), macOS, Linux or
Windows.

```bash
git clone https://github.com/Ahmad170412/Spyglass-PRO.git && cd Spyglass-PRO
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
```

Then:

```bash
python run.py                 # serves on 127.0.0.1:5000 (waitress; Flask dev server only with --debug)
python -m spyglass_pro        # equivalent
spyglass-pro                  # console entry point
```

<details>
<summary><b>Social recon needs one more package — for Instagram only</b></summary>

The Instagram panel drives a real browser via `cloakbrowser`, which is
deliberately **not** a core dependency — it pulls a ~200 MB browser binary and
wants a license key:

```bash
pip install cloakbrowser
```

GitLab, GitHub, Keybase, Mastodon and Bluesky are keyless JSON sources: they
need no browser, no account and no key, and work on a bare install. The two
code forges fill the same panel shape from very different APIs — with GitLab's
unauthenticated profile being the honest *short* form (name, handle, avatar,
state, published email) — so the panel's capability header never promises what
the keyless API cannot return. Keybase is the odd one out and the most useful:
it returns the account's cryptographically-**proven** identities (the Twitter,
GitHub, Reddit and Hacker News accounts the user has signed for), which is
correlation you can trust rather than infer.

</details>

---

## Optional CLI engines

The **email**, **username** and **phone** modules are orchestrators: they shell
out to external CLI engines rather than collecting themselves.

```bash
python scripts/install_tools.py --check    # report which are missing
python scripts/install_tools.py            # install the missing ones
```

This covers sherlock, maigret, holehe, ignorant, blackbird, user-scanner,
shodan, phoneinfoga, exiftool, dig/whois/curl and torsocks — every tool the
dashboard can detect. It ships with a **source checkout** (`git clone` /
`pip install -e .`), not the published wheel; a PyPI install has no
`scripts/` directory, so install the engines directly there. The script is
idempotent — re-running it only installs what is still missing. Python CLIs
install via pipx (isolated venvs, no dependency conflicts); system tools via
brew or apt.

### What you get without them

Degrading without an error is not the same as being useful, so here is exactly
what a bare install can and cannot do:

| Module | With engines | Without engines |
| --- | --- | --- |
| Email | holehe + user-scanner + blackbird sweeps, plus breach | **breach data only** |
| Username | sherlock + maigret + user-scanner + blackbird sweeps, plus breach | **breach data only** |
| Phone | phoneinfoga + ignorant, plus breach | **breach data only** |

Breach checking (LeakCheck + Scylla) needs no account and no CLI tool, so it
works on a bare install. **Website** and **IP Address** run without any *recon*
engine, but their richer legs depend on tools that may be absent — Website uses
`dig`, `whois` and `curl`; IP uses `shodan` for host/port data. **Full
Investigation** merges the modules above, so its coverage is theirs. `GET
/api/status` reports exactly which tools are missing on your machine.

To confirm what the running server can see, `GET /api/status` returns a
per-tool `true`/`false` map, a `missing_tools` list and a per-panel `degraded`
map.

> `holehe` and `ignorant` enumerate which platforms an email or phone number is
> registered on. That is active account enumeration against third-party
> services and it sits against Spyglass-Pro's passive-only posture. Only use
> them on identifiers you own or have permission to test.

---

## Passive-only

**Spyglass-Pro never opens a connection to a target.** Every query goes to a
third party: DNS resolvers, Certificate Transparency logs, WHOIS registries,
InternetDB, RDAP, RIPEstat, Wayback, crt.sh, CertSpotter, HackerTarget and
breach databases. Shodan, GreyNoise, Censys, AbuseIPDB, VirusTotal and NVD
enrichment are wired but need an API key.

There is no port scan, no directory brute force, no HTTP probing of discovered
hosts, no zone-transfer attempt and not even a fetch of the target's homepage.
That code was **removed, not disabled** — the
[audit inventory](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/AUDIT-2026-10.md#active-scanning-removed) lists every
removed function, so none of it can be switched back on by configuration or a
stray flag.

**Where the line sits:** ordinary DNS record lookups (A, MX, NS, TXT, SPF,
DMARC, CNAME) are still made, because no useful host recon exists without them
and every OSINT tool treats them as passive.

**What that costs you, plainly:** no HTTP headers, no TLS certificate or cipher
details, no technology fingerprinting, no robots.txt or sitemap, no page
content, no favicon hash and no exposed-file findings. Active collection lives
in a separate project.

---

## Configuration

All settings are environment variables (CLI flags exist for the basics — see
`spyglass-pro --help`):

| Variable | Default | Purpose |
| --- | --- | --- |
| `SPYGLASS_PRO_HOST` | `127.0.0.1` | Bind address |
| `SPYGLASS_PRO_PORT` | `5000` | Port |
| `SPYGLASS_PRO_TOKEN` | — | Require this Bearer token on every `/api/*` route |
| `SPYGLASS_PRO_ALLOWED_HOSTS` | loopback + bind | Extra `Host` values to accept when no token is set (comma-separated; `*` disables the check) |
| `SPYGLASS_PRO_SECRET` | random per process | Flask `SECRET_KEY` |
| `SPYGLASS_PRO_DEBUG` | `0` | Werkzeug debug mode (**RCE console — never enable on a reachable host**) |
| `SPYGLASS_PRO_MAX_QUEUE` | `16` | Max pending scan jobs before a 429 |
| `SPYGLASS_PRO_JOB_TIMEOUT` | `900` | Seconds a request waits for a job |
| `SPYGLASS_PROXY` | — | Proxy for every outgoing query |
| `SPYGLASS_HOME` | `~/.spyglass` | Case store location |
| `SPYGLASS_STORE_KEY` | — | Base64 32-byte Fernet key for data at rest |
| `SPYGLASS_STORE_PASS` | — | Derive the Fernet key from a password instead |

**Non-loopback deployments** must set `SPYGLASS_PRO_TOKEN`; the server warns at
startup if you don't. While no token is set the dashboard answers only requests
whose `Host` is loopback (or listed in `SPYGLASS_PRO_ALLOWED_HOSTS`) — that is
what stops a DNS-rebinding page from reaching it. Serving it under a real
hostname without a token therefore needs `SPYGLASS_PRO_ALLOWED_HOSTS`, e.g.:

```bash
SPYGLASS_PRO_TOKEN=$(openssl rand -hex 24) python3 run.py
```

Independently of the token, browser requests tagged cross-site
(`Sec-Fetch-Site: cross-site`) or carrying a foreign `Origin` are refused, and
every response sets `Content-Security-Policy`, `X-Frame-Options`,
`X-Content-Type-Options` and `Referrer-Policy`.

---

## API

Every route is under `/api/*` and requires the Bearer token when
`SPYGLASS_PRO_TOKEN` is set.

```bash
# What's installed, what's missing, what degraded
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:5000/api/status

# Run a module
curl "http://127.0.0.1:5000/api/spyglass/run?module=website&target=example.com" \
     -H "Authorization: Bearer $TOKEN"

# Correlated multi-input investigation
curl -X POST http://127.0.0.1:5000/api/investigate/run \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"website":"example.com","email":"user@example.com"}'

# Stored-run queries
curl "http://127.0.0.1:5000/api/cases/diff?target=example.com&qtype=website" \
     -H "Authorization: Bearer $TOKEN"
```

| Endpoint | Purpose |
| --- | --- |
| `GET /api/status` | Engines, platforms, per-tool availability, `missing_tools`, `degraded` |
| `GET /api/spyglass/run` | Run one spyglass module (`module`, `target`, + flags) |
| `GET /api/lens/run` | Lens-Extender passive recon |
| `POST /api/investigate/run` | Parallel multi-input investigation |
| `GET /api/social/platforms` | Registered adapters and their capabilities |
| `POST /api/social/run` | Collect one social profile |
| `GET /api/cases` · `/diff` · `/timeline` · `/profile` | Stored-run queries |
| `DELETE /api/cases/run` | Delete one stored run (and its entity rows) |
| `GET /api/jobs` · `/api/job/<id>` · `/api/stream/<id>` | Async job registry, polling, live SSE log |
| `GET /api/settings` · `POST /api/settings` | API-key vault (masked read) and proxy config |
| `GET /api/healthz` | Liveness plus pool saturation |
| `GET/POST /api/proxy` | Read or set the outbound proxy (scheme-validated) |

A busy server returns **HTTP 429** for a full job queue, distinct from a scan
that ran and failed — so a client can tell back-pressure from a broken payload.

---

## Cases

Every scan can be stored. Stored runs keep a full result snapshot plus an
entity index (domains, IPs, emails, phones, orgs, onions, breaches), which is
what the **Stored Cases** panel reads:

- **Diff last two runs** — entities that appeared or disappeared since the
  previous run of the same target and query type.
- **Timeline** — every stored run of a target, oldest first.
- **Export dossier** — the latest run per query type merged into one JSON
  profile.

Data at rest is encrypted with Fernet when `SPYGLASS_STORE_KEY` or
`SPYGLASS_STORE_PASS` is set; otherwise it is stored in plaintext. This covers
both the result snapshot and the entity index (domains, IPs, emails, orgs) —
the index rows hold the same targets, so encrypting one without the other would
leak them. JSON/CSV/report artifacts from a scan land in the working directory
of the run.

---

## Testing

```bash
.venv/bin/python tests/run_tests.py
```

**545 offline checks** — the three engines' own suites plus the merge layer
(API surface, auth gate, request-origin gate, proxy validation and masking,
job pool, target validation incl. IPv6, entity encryption, case queries,
cross-platform filename and decoding behaviour) and the node-checked dashboard
JS. No network, no browser, finishes in a few seconds.

CI runs the suite on Python 3.10–3.13 (plus macOS and Windows), type-checks
with `mypy`, lints with `ruff`, `node --check`s the dashboard JS, and asserts
the built wheel actually carries `LICENSE`, `NOTICE` and all three vendored
engines.

---

## Platforms

macOS, Linux and Windows are supported. CI runs the suite on Ubuntu
(Python 3.10–3.13) and macOS; Windows ships documented fallbacks for the
tools it lacks. Two platform differences are worth knowing before you file
a bug:

- **Windows has no `dig` or `whois`.** The DNS legs fall back to Python's
  resolver and the HTTP legs to `urllib`, so the modules still run — but DNS
  record tables, SPF/DMARC and WHOIS come back thinner until you install the
  tools. `GET /api/status` lists exactly which are missing on your machine.
- **Windows reserved filenames.** `CON`, `PRN`, `AUX`, `NUL`, `COM1`–`COM9` and
  `LPT1`–`LPT9` cannot be used as filenames in any case, so Spyglass-Pro
  prefixes them rather than producing artifacts that fail to open there.

---

## Project structure

```
run.py                     dev entry point
scripts/install_tools.py   idempotent CLI-engine installer
tests/                     545 offline checks (run_tests.py)
assets/logo.svg            README logo
spyglass_pro/
  app.py                   Flask routes + auth gate + proxy validation
  runner.py                job pool (queue cap, timeouts), engine dispatch
  social/                  multi-platform social harness (schema, registry,
                           adapters) — merge layer, not vendored
  cli.py                   `spyglass-pro` console entry point
  templates/ static/       dashboard UI
  modules/                 vendored engines (see NOTICE)
```

The three engines live behind a directory boundary on purpose, so the next sync
from upstream is mechanical. `NOTICE` records exactly where each copy differs
from its origin — including the bug fixes that exist only here.

---

## Contributing

Read [CONTRIBUTING.md](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/CONTRIBUTING.md). The short version: **the passive-only
rule overrides everything else**, new behaviour needs a test that fails without
it, and a partial result must never look like a complete one.

---

## Responsible use

OSINT tools are dual-use. Only investigate systems and accounts you own or
have explicit permission to test.

Instagram collection may violate those platforms' terms of service. Because
Spyglass-Pro is passive-only it does not touch a target's network — but the
social module still drives a real browser against a real account, so use burner
credentials you are willing to lose. Passive queries also disclose your
*investigation target* to third-party services and your source IP to all of
them; use a proxy if that matters to you.

You are responsible for compliance with applicable laws.

---

## License

MIT — see [LICENSE](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/LICENSE). One copyright holder covers this repository in
full, including the engines under `spyglass_pro/modules/`; [NOTICE](https://github.com/Ahmad170412/Spyglass-PRO/blob/main/NOTICE)
records where those came from and why they are kept behind a directory
boundary.
