Metadata-Version: 2.4
Name: sitewalker
Version: 0.4.0
Summary: Crawl a website and create a structured map of its pages
License: MIT
License-File: LICENSE
Author: Neil Johnson
Author-email: neil@cadent.net
Requires-Python: >=3.11,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: beautifulsoup4 (>=4.12.3,<5.0.0)
Requires-Dist: requests (>=2.32.3,<3.0.0)
Requires-Dist: urllib3 (>=2.2.3,<3.0.0)
Description-Content-Type: text/markdown

# sitewalker

Crawl a website and create a structured map of its pages.

## Requirements

- Python 3.11 or later

## Installation

```bash
pipx install sitewalker
```

## Usage

```bash
# Map all pages on a site (single-level crawl)
sitewalker example.com

# Recursive crawl of all internal pages
sitewalker example.com -r

# Collect external links
sitewalker example.com -e

# Collect external links and check their HTTP status
sitewalker example.com -e --check-external

# Recursive crawl with external link collection
sitewalker example.com -r -e

# Include images, stylesheets, and scripts in the inventory
sitewalker example.com -r -a

# Hunt broken links: non-200s summarized on stdout, with the page to fix
sitewalker example.com -r --broken-only

# Only crawl web pages (skip images, PDFs, etc.)
sitewalker example.com -r -p

# Crawl an HTTP-only site (e.g., LAN staging server)
sitewalker http://staging.lan --allow-private

# Verbose output for debugging
sitewalker example.com -r -v
```

The target accepts a bare domain (`example.com`) or a full URL (`http://example.com`). Bare domains default to HTTPS — if the connection fails, sitewalker exits with a message to provide the full URL.

## Options

| Flag | Description | Default |
|------|-------------|---------|
| **Crawl scope** | | |
| `-r`, `--recursive` | Recursively crawl internal links | Off |
| `-a`, `--assets` | Also discover `img`/`script`/`link`/`source` assets (recorded via HEAD, never parsed) | Off |
| `-p`, `--pages` | Only crawl web pages (HTML, PHP, etc.) | Off |
| `--max-pages` | Maximum number of pages to crawl | 1000 |
| `--max-depth` | Maximum link distance from start URL (BFS) | 10 |
| **External links** | | |
| `-e`, `--external-links` | Collect external links | Off |
| `--check-external` | Check HTTP status of external links (requires `-e`) | Off |
| `--max-external-links` | Maximum external links to check with `--check-external` | 500 |
| `--domain-delay` | Minimum seconds between requests to the same external domain | 5.0 |
| **Output** | | |
| `--output-dir DIR` | Directory for CSV output (created if missing) | Current directory |
| `--output-filename NAME` | Base name for output files (bare name, no path) | `{domain}_{timestamp}` |
| `--broken-only` | Print a summary of non-200 URLs to stdout after the crawl | Off |
| **Requests** | | |
| `-t`, `--timeout` | Request timeout in seconds | 30 |
| `--delay` | Delay between requests in seconds (use 0 for local) | 1.0 |
| **Configuration** | | |
| `--config PATH` | Config file to load | `~/.config/sitewalker/config.toml` |
| `--no-config` | Ignore any config file | Off |
| **Safety overrides** | | |
| `--allow-private` | Allow crawling domains that resolve to private IPs | Off |
| `--ignore-robots` | Ignore robots.txt rules | Off |
| **Other** | | |
| `-v`, `--verbose` | Enable verbose/debug output | Off |

## Configuration

Settings that rarely change can live in a TOML config file at `~/.config/sitewalker/config.toml` (or `$XDG_CONFIG_HOME/sitewalker/config.toml`; override the location with `--config`, skip it entirely with `--no-config`). CLI flags always win over config values.

```toml
# Where CSVs are written when --output-dir isn't given
output_dir = "~/crawls"

# Replaces the built-in extension set used by -p/--pages.
# Include "" to keep treating extension-less URLs and directories as pages.
page_extensions = ["", "html", "htm", "php", "story"]
```

Recognized keys: `output_dir`, `page_extensions`. Unknown keys are an error, so typos fail loudly instead of being ignored.

## Output

Results are saved to a CSV file named `{domain}_{timestamp}.csv` in the current directory. Use `--output-dir` to write elsewhere (the directory is created if needed) and `--output-filename` to replace the generated name:

```bash
sitewalker example.com -r -e --output-dir ~/crawls --output-filename acme-audit
# → ~/crawls/acme-audit.csv and ~/crawls/acme-audit_external_links.csv
```

`--output-filename` must be a bare name — the directory always comes from `--output-dir`. Columns:

- **URL** — the page URL
- **Title** — the page's `<title>` tag content
- **Status Code** — HTTP response status
- **Found On** — the page where this URL was first discovered (empty for the start URL). For a broken link, this is the page to fix.
- **Kind** — `page` (fetched and parsed for links) or `asset` (recorded only, when `-a` is used).

URLs discovered beyond `--max-depth` are included as rows with title `skipped: max_depth` and an empty status code, so nothing the crawler saw is invisible in the output. Use `--broken-only` to print a summary of every non-200 URL (with its Found On page) to stdout at the end of the run.

When using `-e`, external links are additionally saved to `{domain}_{timestamp}_external_links.csv`. The internal pages CSV is always generated. With `--check-external`, the external links CSV includes a Status Code column.

## Security

- **SSRF protection**: Domains that resolve to private/reserved IP addresses are blocked by default. Use `--allow-private` to override for legitimate internal use.
- **robots.txt**: Respected by default. Use `--ignore-robots` to override.
- **CSV injection**: Output values are sanitized to prevent spreadsheet formula injection.
- **Crawl limits**: Recursive crawls are bounded by `--max-pages` and `--max-depth` to prevent resource exhaustion.

## Roadmap

- `--format json` — JSON output format
- `--check-alt` — alt text auditing for the image inventory (`-a` covers the inventory itself)

## License

MIT

