Metadata-Version: 2.4
Name: darknet-crawler
Version: 0.1.1
Summary: DarkNet Crawler (DNC) — standalone dark-web search aggregator with optional WorldBase integration. Defensive OSINT / threat-intelligence tooling.
Author: Piano Operator
License: MIT
Keywords: osint,darkweb,tor,threat-intel,search,fastapi
Classifier: Programming Language :: Python :: 3
Classifier: Framework :: FastAPI
Classifier: Topic :: Security
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pydantic>=2.0
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.30
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rank-bm25>=0.2
Provides-Extra: tor
Requires-Dist: stem>=1.8; extra == "tor"
Requires-Dist: httpx-socks>=0.10; extra == "tor"
Provides-Extra: playwright
Requires-Dist: playwright>=1.40; extra == "playwright"
Provides-Extra: pg
Requires-Dist: psycopg2-binary>=2.9; extra == "pg"
Requires-Dist: asyncpg>=0.29; extra == "pg"
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == "redis"
Provides-Extra: ui
Requires-Dist: streamlit>=1.30; extra == "ui"
Provides-Extra: mcp
Requires-Dist: mcp<2.0,>=1.0; extra == "mcp"
Provides-Extra: ssl
Requires-Dist: cryptography>=42.0; extra == "ssl"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: respx>=0.20; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Provides-Extra: llm
Dynamic: license-file

# Piano

A dark-web search aggregator for defensive OSINT and threat-intelligence research.

Piano queries existing .onion search indexes (Torch, TorDex, Ahmia, and others)
through Tor, extracts identifiers (cryptocurrency wallets, PGP keys, emails,
CVE references, onion addresses), and presents deduplicated, BM25-ranked results
through a CLI, a REST API, and an MCP server. An optional web UI provides
browser-based access over HTTPS for LAN use.

The project is inspired by patterns proven in the [IntelShed WorldBase](https://github.com/intelshed/worldbase)
`darkweb_*` modules and reimplements them as a standalone codebase without
a hard dependency on WorldBase.

## Status

Early-stage research tool. The architecture is designed to fail gracefully:
a down Tor proxy, an unreachable engine, or a missing LLM API key degrades
to reduced functionality rather than crashing. However, the tool has not been
audited for production use, and the engine addresses may rotate without notice.

## Scope and limitations

- **Search-first**: The primary surface queries existing search indexes.
  Broad spidering of arbitrary .onion domains is opt-in and depth-limited.
- **Text-only**: Binary MIME types are dropped at the HTTP layer.
- **No PII resale**: Extracted identifiers are used for local correlation only.
- **Single operator**: No multi-tenancy, no authentication beyond an optional
  API key. The HTTPS UI is intended for LAN use by a single operator.
- **Engine volatility**: .onion addresses change frequently. Engine addresses
  were verified against curated sources (edoardottt/awesome-hacker-search-engines,
  ahmia.fi) at the time of writing; unverified or dead engines are marked
  deprecated and skipped by the circuit breaker.

## Architecture

```
Surface:  CLI (dnc_cli.py)  ·  REST API (dnc/api.py)  ·  MCP server (mcp_dnc/)  ·  Web UI (frontend/)
Intelligence:  entity_extractor  ·  ranker (BM25)  ·  correlator  ·  llm_enricher  ·  actor_profiler  ·  forensics
Acquisition:  engine_registry  ·  search_orchestrator  ·  tor_proxy  ·  tor_rotator  ·  html_parsers  ·  fetcher  ·  cache
```

The MCP server uses a Tool Bus pattern: 25 internal functions are compressed
into 6 semantic meta-tools, reducing the `tools/list` payload by approximately
58% and avoiding client-side timeouts on cold start.

## Quick start

### Option A — Docker (recommended, full standalone)

```bash
git clone https://github.com/sookoothaii/DarkNetCrawler.git
cd DarkNetCrawler

cp .env.example .env
# Edit .env: set DNC_ENABLED=1
# For LLM enrichment: DNC_LLM_ENRICH=1, DNC_NVIDIA_API_KEY=nvapi-...
# (free key at https://build.nvidia.com/)

# Start Tor + API + Web UI
docker compose up -d --build

# Open http://localhost:8010
# CLI:   docker compose exec api python -m dnc_cli search "leaked credentials" --engines tordex --mode tor
# Logs:  docker compose logs -f api
# Stop:  docker compose down
```

Optional services (profiles):

```bash
# MCP server (for Devin, Claude Desktop, Windsurf, Cursor)
docker compose --profile mcp up -d

# Redis (future cache backend, v2)
docker compose --profile redis up -d
```

### Option B — Local Python (no Docker)

```bash
git clone https://github.com/sookoothaii/DarkNetCrawler.git
cd DarkNetCrawler

cp .env.example .env
# Edit .env: set DNC_ENABLED=1, DNC_TOR_PROXY=socks5://127.0.0.1:9050
# Requires a local Tor service or Tor Browser running on port 9050

pip install -e ".[dev,tor,mcp]"

# CLI
python -m dnc_cli check
python -m dnc_cli search "leaked credentials" --engines tordex --mode tor

# REST API + Web UI
python server.py --host 0.0.0.0 --port 8010
# Open http://localhost:8010

# HTTPS (for LAN access from mobile devices)
pip install -e ".[ssl]"
python scripts/generate_cert.py
python server.py --ssl --host 0.0.0.0 --port 8010
# Open https://<lan-ip>:8010

# MCP server (for Devin, Claude Desktop, Windsurf, Cursor)
python -m mcp_dnc.server
```

## Tor

Piano requires a Tor SOCKS5 proxy. The included `docker-compose.yml` provides
a dedicated Tor container built from `tor/Dockerfile` (Debian + apt install tor,
no external image dependency). The container exposes SOCKS on `127.0.0.1:9150`
and the Tor control port on `127.0.0.1:9051`.

Alternatively, any local Tor service (e.g. Tor Browser on port 9050) or an
existing Tor container on a shared Docker network may be used — set
`DNC_TOR_PROXY` accordingly in `.env`.

The fetcher uses `httpx-socks` with remote DNS (`rdns=True`) to resolve `.onion`
addresses through the Tor proxy. Standard `httpx` SOCKS5 transport performs
local DNS, which cannot resolve `.onion` hostnames.

## LLM enrichment

LLM-assisted query refinement, IOC classification, and page summarization are
optional and gated behind `DNC_LLM_ENRICH=1`. The implementation uses a
standalone NVIDIA NIM cascade (nemotron-3-super-120b → nemotron-3-ultra-550b
→ mistral-nemotron) with Ollama qwen3:8b as an offline fallback. When the LLM
is unavailable, all features degrade to regex-only extraction.

## Testing

```bash
ruff check dnc mcp_dnc tests dnc_cli.py server.py
python -m pytest -x
python scripts/smoke_test.py
```

74 unit tests cover the engine registry, entity extraction, HTML parsers,
circuit breaker state machine, TTL cache, envelope contract, BM25 ranker,
Tor rotator, forensics, and configuration parsing.

## License

MIT. See [LICENSE](LICENSE).

## Disclaimer

This software is a defensive research tool. The operator assumes all legal
responsibility for its use. Read [DISCLAIMER.md](DISCLAIMER.md) before first run.
