Metadata-Version: 2.4
Name: ioc-hostchecker
Version: 0.1.2
Summary: Query IPs, domains, URLs and file hashes against multiple threat intel sources in one shot.
Project-URL: Homepage, https://kukitzz.github.io/host-checker/
Project-URL: Repository, https://github.com/kukitzz/host-checker
Project-URL: Documentation, https://github.com/kukitzz/host-checker#readme
Project-URL: Issues, https://github.com/kukitzz/host-checker/issues
Project-URL: Changelog, https://github.com/kukitzz/host-checker/blob/main/CHANGELOG.md
Author: host-checker contributors
License: MIT
License-File: LICENSE
Keywords: dfir,ioc,osint,pentest,security,soc,threat-intel
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: Log Analysis
Classifier: Topic :: Security
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn[standard]>=0.29
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-httpx>=0.30; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/kukitzz/host-checker/main/docs/assets/banner.png" alt="host-checker — multi-source threat intel for IPs, domains, URLs and hashes" width="100%">
</p>

<p align="center">
  <a href="https://github.com/kukitzz/host-checker/actions/workflows/ci.yml"><img src="https://github.com/kukitzz/host-checker/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/ioc-hostchecker/"><img src="https://img.shields.io/pypi/v/ioc-hostchecker?color=22d3ee" alt="PyPI"></a>
  <a href="./LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License: MIT"></a>
  <a href="./pyproject.toml"><img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python 3.11+"></a>
  <img src="https://img.shields.io/badge/providers-13-22d3ee" alt="13 providers">
  <img src="https://img.shields.io/badge/IOCs-IP%20%C2%B7%20domain%20%C2%B7%20URL%20%C2%B7%20hash-22d3ee" alt="IP / domain / URL / hash">
  <a href="https://kukitzz.github.io/host-checker/"><img src="https://img.shields.io/badge/website-online-22d3ee" alt="Project website"></a>
</p>

<p align="center">
  <b>Query IPs, domains, URLs and file hashes against multiple threat-intel sources in one shot — from a CLI, a self-hosted HTTP API or an interactive web UI.</b>
</p>

<p align="center">
  <a href="https://kukitzz.github.io/host-checker/#demo"><img src="https://raw.githubusercontent.com/kukitzz/host-checker/main/docs/assets/demo.png" alt="CLI demo" width="760"></a>
</p>

<p align="center"><sub><a href="https://kukitzz.github.io/host-checker/#demo">▶ See it animated on the project site</a></sub></p>

`host-checker` is an open-source, self-hostable IOC checker that exposes more IOC types than typical hosted tools, accepts bulk input, runs locally so your targets never leave your machine, ships as a single Python package with a **CLI**, a **FastAPI server** and a **web UI**, exports to **MISP** and **STIX 2.1**, and uses an extensible provider architecture so adding a new threat-intel source is a single file.

### The web UI

<p align="center">
  <img src="https://raw.githubusercontent.com/kukitzz/host-checker/main/docs/assets/webui.png" alt="host-checker web UI" width="100%">
</p>

Filter by verdict, sort by score, expand any IOC to see the per-provider breakdown, copy a single provider's JSON, or download the whole result set as MISP or STIX — all without a build step.

---

## Features

- **Multiple IOC types**: IPv4, IPv6, CIDR, domain, URL, MD5/SHA1/SHA256.
- **Automatic refanging**: paste `evil[.]com`, `hxxp://`, `[dot]`, `(.)` — all normalised.
- **BYOK with sensible defaults**: keyless providers (Tor exit list, crt.sh, RDAP, IPinfo) work out of the box; the rest unlock when you add a key.
- **Auto-pivot**: a domain check automatically fans out to its resolved IPs, linked back via `pivoted_from`.
- **Concurrent**: async fan-out across providers and IOCs with a configurable concurrency cap.
- **Three surfaces**: CLI for scripts and CI, JSON HTTP API for automation, and an interactive web UI for live triage. All powered by the same engine.
- **Aggregated, explainable score**: weighted-mean malice rolled into a 0–100 number plus a clear per-provider breakdown.
- **On-disk cache** with configurable TTL.
- **Local allowlist** so internal targets never leave your machine.
- **JSON output**: pipe into `jq`, MISP, TheHive, or your SIEM.
- **No vendor lock-in**: every result includes the raw provider payload.

## Supported providers

| Provider        | IOCs                                         | Requires key | Free tier        |
| --------------- | -------------------------------------------- | ------------ | ---------------- |
| `tor_exit`      | IPv4, IPv6                                   | no           | unlimited        |
| `crtsh`         | domain                                       | no           | unlimited        |
| `rdap`          | domain, IPv4, IPv6                           | no           | unlimited        |
| `urlhaus`       | IPv4, domain, URL, MD5, SHA256               | yes (abuse.ch) | unlimited      |
| `ipinfo`        | IPv4, IPv6                                   | no (key bumps limits) | 50k/month   |
| `virustotal`    | IP, domain, URL, MD5, SHA1, SHA256           | yes          | 500/day          |
| `abuseipdb`     | IPv4, IPv6                                   | yes          | 1 000/day        |
| `greynoise`     | IPv4                                         | yes          | 50/day community |
| `otx`           | IP, domain, URL, MD5, SHA1, SHA256           | yes          | unlimited        |
| `threatfox`     | IPv4, domain, URL, MD5, SHA1, SHA256         | yes (abuse.ch) | unlimited      |
| `malwarebazaar` | MD5, SHA1, SHA256                            | yes (abuse.ch) | unlimited      |
| `shodan`        | IPv4, IPv6                                   | yes          | $5 one-time      |
| `securitytrails`| domain, IPv4                                 | yes          | 50/month         |

`abuse.ch` providers (`urlhaus`, `threatfox`, `malwarebazaar`) share a single Auth-Key — register once at <https://auth.abuse.ch/> and put it in `HC_ABUSECH_AUTH_KEY`.

`rdap` is keyless and gives you **domain registration age** (newly-registered domains are flagged suspicious) plus registrar and IP network ownership — a great default-on signal.

Adding more sources is a single file — see [Adding a provider](#adding-a-provider).

---

## Install

```bash
pip install ioc-hostchecker
hostchecker check 8.8.8.8 evil[.]com
hostchecker serve   # → web UI on http://localhost:8000/
```

Or from source, for development:

```bash
git clone https://github.com/kukitzz/host-checker.git
cd host-checker
pip install -e ".[dev]"
cp .env.example .env  # then fill in the keys you have
```

Python 3.11+ required.

> The PyPI distribution is published as **`ioc-hostchecker`** (the plain `hostchecker` name was already taken). The import package and the CLI command are both still `hostchecker`.

### Docker

```bash
# Run the web UI / API:
docker run --rm -p 8000:8000 \
  --env-file .env \
  -v "$PWD/.hostchecker-cache:/var/cache/hostchecker" \
  ghcr.io/kukitzz/host-checker:latest
# → open http://localhost:8000/

# Or run the CLI inside the container:
docker run --rm --env-file .env ghcr.io/kukitzz/host-checker:latest \
  hostchecker check 8.8.8.8 evil[.]com
```

## CLI usage

```bash
# Inline IOCs (comma- or space-separated; defanged forms accepted)
hostchecker check 8.8.8.8 evil[.]com hxxps://malicious[.]example/path

# From a file (one per line or CSV)
hostchecker check --input iocs.txt

# Machine-readable JSON
hostchecker check 1.1.1.1 --format json | jq .

# Export to threat-intel exchange formats
hostchecker check 1.2.3.4 evil.com --format misp > event.json     # MISP event
hostchecker check 1.2.3.4 evil.com --format stix > bundle.json   # STIX 2.1 bundle

# Only run specific providers
hostchecker check evil.com -p virustotal -p urlhaus

# List providers and their enablement status
hostchecker providers

# Start the HTTP API
hostchecker serve --host 0.0.0.0 --port 8000
```

Exit code is non-zero when any IOC's aggregate verdict is `malicious` — useful for CI gates.

## HTTP API

```bash
hostchecker serve &

curl -s http://localhost:8000/providers | jq .

curl -s -X POST http://localhost:8000/check \
     -H 'content-type: application/json' \
     -d '{"targets": ["8.8.8.8", "evil[.]com"]}' | jq .
```

Swagger UI lives at `/docs`, ReDoc at `/redoc`.

## Web UI

Open <http://localhost:8000/> after `hostchecker serve`. The page gives you:

- A textarea (with `localStorage` persistence — your last query survives reloads).
- Toggles for auto-pivot and bypass-cache.
- A collapsible panel listing every provider and whether it's currently enabled (greyed out when no key is configured).
- Results rendered as colour-coded cards by verdict (red / yellow / green / grey), with per-IOC counts, the aggregate score, and any `pivoted from` annotation.
- Each card expands to show the per-provider breakdown with a one-click **copy json** button on every row.
- Client-side filter (`malicious / suspicious / clean / unknown / all`) and sort (`score / verdict / IOC`).

Tailwind, htmx and Alpine.js are loaded from CDN — no build step. For air-gapped deployments you can vendor the three files into a `static/` directory and edit `templates/base.html` to point at the local copies; the rest of the app needs no changes.

## Configuration

All settings live in `.env` (see `.env.example`). Keys are read once at startup. Beyond credentials:

| Variable               | Default                | Meaning                                  |
| ---------------------- | ---------------------- | ---------------------------------------- |
| `HC_REQUEST_TIMEOUT`   | `15.0`                 | Per-request HTTP timeout (seconds).      |
| `HC_MAX_CONCURRENCY`   | `10`                   | Max in-flight provider queries.          |
| `HC_MAX_RETRIES`       | `3`                    | Retries on 429/502/503/504 & net errors. |
| `HC_RETRY_BACKOFF_BASE`| `0.5`                  | Base seconds for backoff (full jitter).  |
| `HC_CACHE_DIR`         | `.hostchecker-cache`   | Local cache directory.                   |
| `HC_CACHE_TTL`         | `3600`                 | Cache TTL (seconds, `0` disables).       |
| `HC_CACHE_BACKEND`     | `file`                 | `file` or `sqlite`.                      |
| `HC_AUTO_PIVOT`        | `true`                 | Resolve domain IOCs to IPs and check.    |
| `HC_PIVOT_LIMIT`       | `5`                    | Max IPs to pivot to per domain.          |
| `HC_ALLOWLIST_FILE`    | `(none)`               | Path to a plain-text allowlist file.     |

## Auto-pivot

When you check a domain, `host-checker` resolves it via DNS and **also checks each of the resulting IPs** against every supported provider, with a `pivoted_from` annotation linking them to the originating domain. This catches the common case where a domain looks clean on VirusTotal but its IPs are flagged by AbuseIPDB and Shodan reports a CVE.

```text
MALICIOUS • score 78.5 • evil.com (domain)
  ↳ vt: 12/63 engines flag malicious  …
SUSPICIOUS • score 42.0 • 1.2.3.4 (ipv4) • pivoted from evil.com
  ↳ abuseipdb: 87% confidence, 142 reports
  ↳ shodan: 8 open ports, 2 CVEs
```

Disable per-run with `--no-pivot`, globally with `HC_AUTO_PIVOT=false`. Cap the fan-out with `HC_PIVOT_LIMIT` (default `5`). Pivoted IPs respect the allowlist and the cache exactly like explicit IOCs.

## Caching

Provider results are cached on disk keyed by `(provider, ioc_type, ioc_value)`. Entries older than `HC_CACHE_TTL` are ignored, and errors / skipped / rate-limited results are never cached. Two backends:

- **`file`** (default) — one JSON file per entry. Zero setup, human-inspectable.
- **`sqlite`** — a single SQLite database. Use it once you're checking thousands of IOCs: indexed lookups and a one-statement purge instead of a directory full of files. Enable with `HC_CACHE_BACKEND=sqlite`.

Inspect and maintain the cache:

```bash
hostchecker cache status   # backend, dir, TTL, entry count
hostchecker cache purge    # delete only expired entries
hostchecker cache clear    # delete everything
```

Bypass it for a single run with `--no-cache`, or disable globally with `HC_CACHE_TTL=0`.

> Note: switching backends doesn't migrate existing entries — the cache is regenerable, so just let it repopulate (or `hostchecker cache clear` first).

## Allowlist

Create a plain text file with one IP, CIDR or domain per line. Lines starting with `#` are comments. Domain entries match the listed domain **and all its subdomains** (`example.com` allowlists `foo.bar.example.com`).

```text
# corp ranges
10.0.0.0/8
192.168.0.0/16
# our infra
example.com
mycompany.local
```

Point at it via the env var (`HC_ALLOWLIST_FILE=./allowlist.txt`) or on the command line (`--allowlist ./allowlist.txt`). Matching IOCs short-circuit as `CLEAN` and **never** reach any upstream provider — useful both for noise reduction and for keeping internal targets off third-party servers.

Hashes are intentionally never allowlist-able: a hash identifies a file, not infrastructure, and silencing a hash by accident is worse than a noisy alert.

## How the score works

Each provider returns a verdict in `{clean, suspicious, malicious, unknown, error, skipped}`. The aggregator:

1. Drops `error` and `skipped` from the calculation.
2. Maps verdicts to weights (`clean=0`, `suspicious=1`, `malicious=2`).
3. Multiplies by a per-provider weight (defaults in `core/aggregator.py:DEFAULT_WEIGHTS`).
4. Rescales to `[0, 100]`.

The thresholds for the final `aggregate_verdict` are intentionally simple — read `aggregate()` and tune for your environment.

## Adding a provider

Create `src/hostchecker/providers/your_source.py`:

```python
from ..config import settings
from ..core.ioc import IOC, IOCType
from ..core.models import ProviderResult, Verdict
from ..core.registry import register
from .base import Provider

@register
class YourSourceProvider(Provider):
    name = "your_source"
    supported_types = {IOCType.IPV4, IOCType.DOMAIN}
    requires_key = True

    def api_key(self) -> str | None:
        return settings.your_source_api_key

    async def query(self, ioc, client):
        resp = await client.get("https://example.com/api", params={"q": ioc.value})
        # … map response to a ProviderResult …
        return ProviderResult(provider=self.name, verdict=Verdict.CLEAN, summary="…")
```

Then:

1. Add the import to `providers/__init__.py`.
2. Add `your_source_api_key: str | None = None` to `config.py:Settings`.
3. Add the env var to `.env.example`.
4. Optionally tune its weight in `aggregator.py:DEFAULT_WEIGHTS`.

That's it — the registry picks it up automatically.

## Exports

Three machine-readable formats are produced from the same engine:

| Format  | What it is                                                              | Where                                             |
| ------- | ----------------------------------------------------------------------- | ------------------------------------------------- |
| `json`  | The native `CheckResponse` schema, raw provider payloads included.      | `--format json` · `POST /check` · UI button       |
| `misp`  | A MISP event JSON, ready for `POST /events/add` against a MISP instance.| `--format misp` · `POST /export/misp` · UI button |
| `stix`  | A STIX 2.1 bundle of `indicator` SDOs, with deterministic UUIDs.        | `--format stix` · `POST /export/stix` · UI button |

```bash
# Push a check result straight into a MISP instance:
hostchecker check 1.2.3.4 evil.com --format misp \
  | curl -s -X POST https://misp.example.org/events/add \
         -H "Authorization: $MISP_API_KEY" \
         -H "Accept: application/json" \
         -H "Content-Type: application/json" -d @-
```

In the web UI a **Download** row above the results lets you pull the same three formats with one click — useful for handing a triage to an analyst working in MISP / TheHive / OpenCTI.

## Deploying for a team

The API and web UI ship with **no authentication** — perfect locally, not for a shared server. The [`deploy/`](./deploy) directory has a Traefik-based `docker-compose` that puts HTTPS (Let's Encrypt) and HTTP basic-auth in front of the app, with the app container never exposing a port directly:

```bash
cd deploy
cp .env.example .env       # set HC_DOMAIN, ACME_EMAIL, BASIC_AUTH
cp ../.env.example app.env # set your provider API keys
docker compose up -d --build
```

There's also a `docker-compose.local.yml` to test the proxy + auth flow on `localhost:8080` without a domain or TLS. Full instructions, including credential generation, are in [`deploy/README.md`](./deploy/README.md).

## Roadmap

- AbuseIPDB v3 when available; VirusTotal relationship pivoting.
- Web UI: query history sidebar, side-by-side IOC comparison.
- Optional SSO proxy recipe (oauth2-proxy / Authelia) alongside the basic-auth deploy.

## License

MIT — see [LICENSE](./LICENSE).

## Disclaimer

This tool aggregates third-party data. Verdicts are best-effort and depend on the underlying providers. Always corroborate before acting on a single source.
