Metadata-Version: 2.4
Name: evilspider
Version: 2.6.0
Summary: Fast Async Attack Surface Recon Crawler & Hunter
Author: Baba01hacker666
License: MIT
Project-URL: Homepage, https://github.com/Baba01hacker666/evilspider
Project-URL: Repository, https://github.com/Baba01hacker666/evilspider
Project-URL: Bug Tracker, https://github.com/Baba01hacker666/evilspider/issues
Keywords: web-crawler,reconnaissance,bug-bounty,security-tools,attack-surface,secret-scanner,api-discovery
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Topic :: Security
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: jinja2>=3.0.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: bandit>=1.7.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"

# 🕷️ EvilSpider

[![CI](https://github.com/Baba01hacker666/evilspider/actions/workflows/ci.yml/badge.svg)](https://github.com/Baba01hacker666/evilspider/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/evilspider.svg)](https://pypi.org/project/evilspider/)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![AsyncIO](https://img.shields.io/badge/AsyncIO-High%20Performance-green.svg)](https://docs.python.org/3/library/asyncio.html)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**EvilSpider** is a fast, asynchronous attack-surface crawler and reconnaissance engine designed for bug bounty hunters, penetration testers, and security researchers.

Unlike traditional generic crawlers (e.g., Scrapy or Crawlee), EvilSpider focuses strictly on **offensive security use cases**: uncovering hidden endpoints, hunting leaked secrets and API keys, extracting parameterized URLs, fingerprinting technologies, mapping forms, and generating actionable reconnaissance dashboards.

---

## 📥 Installation

### 📦 Via `pip`
```bash
pip install evilspider
```

### ⚡ Via `pipx` (Recommended for isolated CLI)
```bash
pipx install evilspider
```

### 🛠️ From Source (Development)
```bash
git clone https://github.com/Baba01hacker666/evilspider.git
cd evilspider
pip install -e .
```

---

## 🚀 Quick Start & Smart Defaults

EvilSpider uses **smart defaults** — you don't even need to type `crawl`! Simply run EvilSpider directly with your target URL or flags:

### 1. Instant Recon Crawl (Smart Default)
```bash
evilspider https://example.com
# or with flags:
evilspider -u https://example.com -d 3
```

### 2. Multi-Target Crawl from File (Save findings to JSON)
```bash
evilspider -l targets.txt -t 20 -o results.json
```

### 3. Pipeline URLs to Nuclei / FFUF / HTTPX
```bash
evilspider https://example.com --urls-only | nuclei -t http/cves/
```

### 4. Generate Interactive Dark-Mode HTML Dashboard
```bash
evilspider https://example.com --report-html report.html --report-md report.md
```

> 💡 **Note on Output Saving:** EvilSpider keeps your filesystem clean. It **only saves files to disk when you explicitly tell it to** (using `-o`, `--output`, `--csv`, `--report-html`, `--report-md`, etc.). Otherwise, findings are streamed cleanly in the terminal.

---

## ⚡ Key Features

- **🧠 Smart Defaults**: No boilerplate required. Run `evilspider https://example.com` directly without typing `crawl`.
- **🛡️ WAF & Anti-Bot Hunter (25+ Protections)**: Automatic signature detection for Cloudflare, AWS WAF, Akamai Kona / Bot Manager, Imperva Incapsula, Fastly Signal Sciences, F5 BIG-IP ASM, ModSecurity / OWASP CRS, Sucuri CloudProxy, Barracuda, Fortinet FortiWeb, Datadome, PerimeterX, and more. Features **adaptive evasion & backoff** to prevent IP bans.
- **⚡ Smart SPA & JS-Heavy App Engine**: Automatically detects Single Page Applications (React, Vue, Angular, Next.js, Nuxt, Svelte, Vite) and performs **multi-layer JS decoding & deobfuscation**:
  - Decodes **Base64-encoded** hidden API endpoints and secret tokens
  - Unescapes **Hex (`\x..`)**, **Unicode (`\u....`, `\u{...}`)**, and **URL-encoded** strings
  - Deobfuscates **JS Obfuscator string array tables** (e.g. `_0x1234 = ['/api/...', ...]`)
  - Mines **Client-side Routes** (React Router `<Route path="...">`, Vue Router, Angular Router, Next.js dynamic routes)
  - Mines **Webpack & Vite dynamic chunk bundles** and fetches/analyzes them recursively
  - Extracts embedded **GraphQL queries & mutations**
- **💾 Clean Output Saving**: Only writes to disk when explicitly asked via `-o` / `--output` or report flags.
- **🚀 Async Concurrency Engine**: Ultra-fast asynchronous crawler built on `asyncio` and `aiohttp` with connection pooling, rate limiting, and configurable worker threads.
- **🚨 Secret & Sensitive Data Hunter**: Built-in regex rule engine scanning for 30+ types of credentials across responses (AWS keys, Google Cloud API keys, JWT tokens, Stripe keys, GitHub PATs, Slack webhooks, Private keys, Database URIs, and leaked authorization headers).
- **📝 HTML Form & Input Extractor**: Detects and categorizes HTML forms (Login, Registration, File Uploads, Password Reset, Search) along with hidden CSRF tokens and field names.
- **🛠️ Tech Stack Fingerprinter**: Detects CMS (WordPress, Drupal, Joomla, Shopify), Backend Frameworks (Django, Laravel, Spring Boot, Express, Rails, FastAPI), Web Servers (Nginx, Apache, Caddy, Cloudflare, IIS), and Frontend Frameworks.
- **🛡️ Security Headers & Reflection Auditor**: Identifies missing security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options), dangerous CORS misconfigurations (`*` with credentials), and reflected URL parameters (potential XSS sinks).
- **📖 API Documentation & Schema Parsing**: Automatically parses Swagger 2.0 / OpenAPI 3.x specifications and GraphQL schemas to extract all declared endpoints, parameters, and routes.
- **🎯 Active Sensitive File Prober (`probe`)**: Fast active probing module hunting for VCS leaks (`.git/HEAD`, `.svn`), exposed configs (`.env`, `web.config`), backup archives (`dump.sql`, `backup.zip`), and debug endpoints (`actuator/health`, `pprof`) with smart soft-404 detection.
- **🌐 Scope & Filter Controls**: Scope by domain, subdomains, regex whitelist/blacklist, extension filters, path exclusions, status codes, and loop prevention algorithms.
- **🎭 Browser Impersonation Profiles**: Modern User-Agent and Client Hints / header profiles for Chrome, Firefox, Edge, Safari, Chrome Android, and Safari iOS.
- **📊 Multi-Format Reporting**: Export findings to JSON, JSON Lines (NDJSON), CSV, Plain text URLs, Parameter wordlists, Markdown reports, and a standalone responsive **Interactive Dark-Mode HTML Dashboard**.

---

## 🔥 Recon Workflows

### 🕵️ 1) Full Hunter Mode (Secrets + Uploads + Sensitive Probing)
```bash
evilspider crawl \
  -u https://target.tld \
  --robots --sitemaps \
  --detect-uploads \
  --probe-sensitive \
  -d 4 -t 15 \
  --report-html report.html
```

### 🔑 2) Sensitive Endpoint & Backup Prober (`probe`)
```bash
evilspider probe \
  -u https://target.tld \
  --categories vcs,env_config,backups,api_docs,actuators_debug \
  -o probe_findings.json
```

### 📁 3) Offline JavaScript & File Analysis (`extract`)
```bash
evilspider extract -f app.bundle.js -o extracted_assets.json
```

### 🔐 4) Authenticated Crawl with Session Cookies & Bearer Token
```bash
evilspider crawl \
  -u https://target.tld \
  -C 'session=abc12345; role=admin' \
  --bearer-token 'eyJhbGciOiJIUzI1Ni...' \
  -H 'X-Forwarded-For: 127.0.0.1'
```

### 🌐 5) Proxied Crawl through Burp Suite / OWASP ZAP
```bash
evilspider crawl \
  -u https://target.tld \
  -x http://127.0.0.1:8080 \
  --insecure \
  --report-redirects
```

### 🎯 6) Parameter Wordlist Harvesting for Fuzzing
```bash
evilspider crawl \
  -u https://target.tld \
  --params-file params_wordlist.txt \
  --urls-only > discovered_urls.txt

# Pipe discovered parameters into ffuf
ffuf -u "https://target.tld/endpoint?FUZZ=1" -w params_wordlist.txt
```

---

## 🛠️ CLI Options Reference

### `evilspider crawl` (Main Crawler)

| Flag | Description |
|---|---|
| `-u, --url` | Target URL to crawl (e.g. `https://example.com`) |
| `-l, --list` | File containing target URLs (or `-` for stdin) |
| `-c, --config` | Path to JSON configuration file |
| `-t, --threads` | Number of concurrent workers (default: `10`) |
| `-d, --max-depth` | Maximum crawl depth (default: `3`) |
| `-m, --max-links` | Maximum total links to visit (default: `5000`) |
| `--max-time` | Maximum crawl duration in seconds |
| `--rate-limit` | Maximum requests per second |
| `--delay` | Delay between requests in seconds |
| `-s, --status` | Interesting status codes (default: `200`) |
| `--exclude-status` | Status codes to ignore (e.g. `404,500`) |
| `-e, --exts` | Extension focus list (e.g. `php,bak,env,json`) |
| `-k, --keywords` | Keyword filter for response bodies |
| `-p, --params-only` | Only report URLs with query parameters |
| `--include-subdomains` / `--no-include-subdomains` | Toggle subdomain in-scope crawling (default: `true`) |
| `--scope-regex` | Whitelist regex pattern for allowed URLs |
| `--exclude-regex` | Blacklist regex pattern for ignored URLs |
| `--exclude-paths` | Comma-separated paths to ignore (`/logout,/signout`) |
| `--robots` / `--sitemaps` | Parse `robots.txt` and `sitemap.xml` |
| `--detect-uploads` | Detect forms with file uploads |
| `--probe-sensitive` | Actively probe target for sensitive files (.git, .env, backups, etc.) |
| `--probe-categories` | Prober categories (`vcs,env_config,backups,api_docs,actuators_debug,graphql,admin_auth`) |
| `--no-secrets` | Disable secrets and API keys scanner |
| `--no-forms` | Disable HTML forms extractor |
| `--no-tech` | Disable technology stack fingerprinter |
| `--no-security-audit`| Disable security headers auditor |
| `-A, --user-agent` | Custom User-Agent string |
| `-i, --impersonate` | Browser profile (`chrome`, `firefox`, `edge`, `safari`, `chrome-android`, `safari-ios`) |
| `-C, --cookies` | Cookie string or path to cookie file |
| `-H, --headers` | Custom header (repeatable, e.g. `-H 'X-Header: value'`) |
| `-x, --proxy` | Proxy URL (HTTP/SOCKS5 e.g. `http://127.0.0.1:8080`) |
| `--bearer-token` | Authorization Bearer token |
| `--auth-basic` | HTTP Basic Auth in format `user:pass` |
| `-T, --timeout` | Request timeout in seconds (default: `5`) |
| `--retries` | Number of request retries on failure (default: `2`) |
| `--follow-redirects` | Follow HTTP redirects (default: `true`) |
| `--report-redirects`| Include redirect chains in results |
| `--insecure` | Disable SSL certificate verification |
| `-o, --output` | Primary output JSON file path |
| `-j, --json` | Stream findings as JSON Lines to stdout |
| `--csv` | Export findings as CSV |
| `--report-html` | Generate interactive standalone HTML dashboard |
| `--report-md` | Generate Markdown summary report |
| `--urls-only` | Clean stdout output containing only discovered URLs |
| `--params-file` | Save discovered query parameter wordlist |
| `--secrets-file` | Save discovered secrets to separate JSON |
| `--forms-file` | Save extracted forms to separate JSON |
| `-q, --quiet` | Suppress banner and info logs |
| `--silent` | Completely silent mode for UNIX piping |
| `-v, --verbose` | Enable debug logging |
| `--no-color` | Disable ANSI terminal colors |

---

## 📊 Interactive HTML Dashboard

EvilSpider generates a standalone, self-contained HTML report with:
- Summary metric cards (Endpoints, Secrets, Forms, Subdomains, Technologies, Security Issues)
- Dynamic table filtering and instant keyword search
- Dedicated tabs for Secrets, Forms, Technologies, Security Headers, API Docs, and Subdomains
- Zero external CDN dependencies (works completely offline)

---

## 🏗️ Architecture

```
evilspider/
├── analyzer/
│   ├── waf.py            # 25+ WAF & Anti-Bot signature detector & adaptive evader
│   ├── js_analyzer.py    # SPA detector, Base64/Hex/Unicode decoder & client route miner
│   ├── secrets.py        # 30+ regex rules for API keys, tokens & credentials
│   ├── tech.py           # Web server, framework & CMS fingerprinter
│   ├── security.py       # Security headers, CORS & reflected param auditor
│   └── api_parser.py     # Swagger 2.0 / OpenAPI 3.x & GraphQL parser
├── extractors/
│   ├── html_extractor.py # Fast HTML parser for links, scripts, frames & comments
│   ├── js_extractor.py   # JavaScript REST endpoint & webpack chunk extractor
│   ├── form_extractor.py # HTML form, input & CSRF token parser
│   └── sourcemap.py      # Source map (.js.map) unminified route extractor
├── prober/
│   ├── probe.py          # Active sensitive file prober with soft-404 detection
│   └── wordlists.py      # Curated recon path lists (VCS, configs, backups, APIs)
├── output/
│   ├── formatter.py      # ANSI terminal formatting, logo & summary tables
│   ├── reporter.py       # Multi-format reporter (JSON, CSV, MD, HTML, TXT)
│   └── templates/
│       └── report_template.html # Responsive dark-mode dashboard
├── tests/                # Comprehensive unit and integration test suite
├── _version.py           # Single source of truth for version and author metadata
├── config.py             # Config parser and CLI argument validator
├── crawler.py            # Async crawl orchestrator and queue worker pool
└── main.py               # CLI entrypoint (crawl, probe, extract, report)
```

---

## ⚖️ Legal Disclaimer

EvilSpider is developed for authorized security assessments, penetration testing, bug bounty reconnaissance, and educational purposes only. Do not use this tool against targets without explicit authorization. The authors assume no liability for misuse.
