Metadata-Version: 2.4
Name: robofinder
Version: 0.1.0
Summary: Extract historical robots.txt paths via Wayback Machine – a recon tool for bug hunters
Project-URL: Homepage, https://github.com/Spix0r/robofinder
Project-URL: Bug Tracker, https://github.com/Spix0r/robofinder/issues
Project-URL: Changelog, https://github.com/Spix0r/robofinder/blob/main/CHANGELOG.md
Author: gnomegl
Author-email: Spix0r <spix0r@tuta.io>
License: MIT License
        
        Copyright (c) 2024 Spix0r
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bugbounty,osint,recon,robots,security,wayback
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Requires-Python: >=3.8
Requires-Dist: requests>=2.28.0
Requires-Dist: validators
Description-Content-Type: text/markdown

<h1 align="center">RoboFinder</h1>

<p align="center">
  Uncover hidden endpoints by mining every historical <code>robots.txt</code> snapshot from the Wayback Machine.
</p>

<p align="center">
  <a href="https://pypi.org/project/robofinder/"><img src="https://img.shields.io/pypi/v/robofinder?color=blue" alt="PyPI"></a>
  <a href="https://pypi.org/project/robofinder/"><img src="https://img.shields.io/pypi/dm/robofinder" alt="Downloads"></a>
  <img src="https://img.shields.io/badge/python-3.8%2B-blue" alt="Python 3.8+">
  <a href="LICENSE"><img src="https://img.shields.io/github/license/Spix0r/robofinder" alt="License"></a>
  <a href="https://github.com/Spix0r/robofinder/stargazers"><img src="https://img.shields.io/github/stars/Spix0r/robofinder?style=social" alt="Stars"></a>
</p>

---

## Why RoboFinder?

Sites regularly scrub sensitive paths from `robots.txt` — but the **Wayback Machine keeps every version ever crawled**.

RoboFinder queries Archive.org's CDX API to pull all historical `robots.txt` snapshots for a target, deduplicates every `Allow`, `Disallow`, and `Sitemap` directive, and prints the full list. Paths that were quietly removed from production may still be alive and reachable.

**Built for:** bug bounty recon · OSINT · attack-surface mapping · forgotten endpoint discovery

---

## Install

```bash
pip install robofinder
```

<details>
<summary>Install from source</summary>

```bash
git clone https://github.com/Spix0r/robofinder
cd robofinder
pip install .
```
</details>

---

## Quick start

```bash
# Scan a single target
robofinder -u https://example.com

# Full URLs ready to probe
robofinder -u https://example.com -c

# Extract forgotten URL parameters
robofinder -u https://example.com -p

# Save results to a file
robofinder -u https://example.com -o results.txt

# Pipe into other tools
robofinder -u https://example.com -c | httpx -silent -mc 200
robofinder -u https://example.com -c | nuclei -t exposures/
```

---

## Usage

```
robofinder -u <URL> [options]
```

| Flag | Long form | Default | Description |
|------|-----------|---------|-------------|
| `-u` | `--url` | — | Target URL *(required)* |
| `-o` | `--output` | — | Save results to a file |
| `-t` | `--threads` | `10` | Number of fetch threads |
| `-c` | | — | Prefix each path with the target URL |
| `-p` | | — | Extract URL parameters from historical paths |
| `-s` | `--silent` | — | Suppress the banner |
| | `--debug` | — | Verbose debug output |

### Concatenate paths with the target URL (`-c`)

Useful for piping directly into scanners:

```bash
robofinder -u https://example.com -c
# https://example.com/admin
# https://example.com/api/v1
# https://example.com/internal
```

### Extract forgotten URL parameters (`-p`)

Finds parameter names that appeared in historical paths:

```bash
robofinder -u https://example.com -p
# id
# token
# redirect_url
```

---

## Changelog

See [CHANGELOG.md](https://github.com/Spix0r/robofinder/blob/main/CHANGELOG.md).

---

## License

[MIT](LICENSE) © [Spix0r](https://github.com/Spix0r)
