Metadata-Version: 2.4
Name: qarbaal
Version: 0.1.1
Summary: Extract, normalize, validate, and export indicators of compromise from text and files.
Author: ahur4
License-Expression: MIT
Project-URL: Documentation, https://ahur4.github.io/Qarbaal/
Project-URL: Homepage, https://github.com/ahur4/Qarbaal
Project-URL: Source, https://github.com/ahur4/Qarbaal
Project-URL: Repository, https://github.com/ahur4/Qarbaal
Project-URL: Issues, https://github.com/ahur4/Qarbaal/issues
Project-URL: Changelog, https://github.com/ahur4/Qarbaal/blob/main/CHANGELOG.md
Keywords: ioc,threat-intelligence,dfir,cti,stix,indicators,qarbaal
Classifier: Development Status :: 4 - Beta
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12; extra == "html"
Provides-Extra: pdf
Requires-Dist: pymupdf>=1.24; extra == "pdf"
Provides-Extra: docx
Requires-Dist: python-docx>=1.1; extra == "docx"
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == "yaml"
Provides-Extra: pcap
Requires-Dist: scapy>=2.5; extra == "pcap"
Provides-Extra: pydantic
Requires-Dist: pydantic>=2.4; extra == "pydantic"
Provides-Extra: all
Requires-Dist: beautifulsoup4>=4.12; extra == "all"
Requires-Dist: pymupdf>=1.24; extra == "all"
Requires-Dist: python-docx>=1.1; extra == "all"
Requires-Dist: pyyaml>=6.0; extra == "all"
Requires-Dist: scapy>=2.5; extra == "all"
Requires-Dist: pydantic>=2.4; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: mypy>=1.13; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.26; extra == "docs"
Dynamic: license-file

# qarbaal

Extract, normalize, validate, and export indicators of compromise from text and files.

[![Python 3.10–3.13](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-3776AB)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/github/license/ahur4/Qarbaal)](https://github.com/ahur4/Qarbaal/blob/main/LICENSE)
[![CI](https://img.shields.io/github/actions/workflow/status/ahur4/Qarbaal/ci.yml?branch=main&label=CI)](https://github.com/ahur4/Qarbaal/actions/workflows/ci.yml)
[![Documentation](https://img.shields.io/badge/docs-mkdocs-0A7AAA)](https://ahur4.github.io/Qarbaal/)

**Documentation:** <https://ahur4.github.io/Qarbaal/>

`qarbaal` reads reports, logs, and message files on your machine and returns classified indicators. It refangs defanged text, records where each value came from, and writes CSV, JSON, STIX 2.1, MISP, OpenIOC, and network rules. The base install uses only the Python standard library. It does not call an enrichment API.

## Installation

Python 3.10 or newer.

```console
pip install qarbaal
```

That install has no runtime dependencies.

## Installing optional extras

Text, JSON, CSV, XML, EML, and MBOX are read with the standard library. The other readers are extras, so a text-only install does not pull them in.

One extra:

```console
pip install "qarbaal[html]"
```

Several at once:

```console
pip install "qarbaal[html,pdf,docx]"
```

Every optional extra:

```console
pip install "qarbaal[all]"
```

| Input | Extra |
| --- | --- |
| Text, JSON, CSV, XML, EML, MBOX | none |
| HTML | `html` |
| PDF | `pdf` |
| DOCX | `docx` |
| YAML | `yaml` |
| PCAP | `pcap` |

A missing extra raises `OptionalDependencyError` and names the install command.

### Pydantic models

```console
pip install "qarbaal[pydantic]"
```

That extra enables `qarbaal.adapters.to_pydantic(indicator)` and `extraction_to_pydantic(result)`. Each call returns a pydantic `BaseModel` (`QarbaalIndicator`) for validation and serialization in pydantic-based apps and APIs. Core results stay plain dataclasses, so the base install keeps zero runtime dependencies. Calling either adapter without pydantic raises `OptionalDependencyError` with the install hint.

## Quickstart

```python
from qarbaal import IOCExtractor, extract

report = """
From: support[at]corp-it-desk[.]com
Lure: hxxps://login[.]corp-it-desk[.]com/session
Beacon: 185.220.101.42
MD5: d41d8cd98f00b204e9800998ecf8427e
"""

result = extract(report)
for indicator in result:
    print(indicator.type, indicator.value, indicator.tags, indicator.line)

extractor = IOCExtractor(
    extract_types=["ipv4", "domain", "url", "hash", "email"],
    normalize=True,
    validate=True,
    deduplicate=True,
)
same = extractor.extract(report, source="ticket-14")
```

`extract()` uses the default type set. File paths, mutex names, and User-Agent strings stay out of that set until you ask for them with `extract_types` or `"all"`.

A synthetic incident is included:

```python
from qarbaal import SAMPLE, extract

print(extract(SAMPLE).stats.counts)
```

## What you get back

Each indicator keeps three strings:

| Field | Meaning |
| --- | --- |
| `value` | First-seen sieve form. Hashes are lower case, CVE IDs are upper case. |
| `raw` | Exact slice of the source, which may still be defanged. |
| `normalized` | Canonical form used for the fingerprint. |

It also carries:

- Provenance on every sighting: source label, line, character offset, and page number for PDF.
- A SHA-256 fingerprint of the type and the normalized value. `raw`, `value`, and `normalized` are all kept.
- Local enrichment only. Address flags, a best-effort domain split, and URL parts are computed on the machine. Nothing is sent to a network service.
- Deduplication and cross-source correlation. `occurrences` counts sightings. `Extraction.shared()` lists indicators that appear in more than one named source.
- Custom `PatternSpec` patterns, allow and deny globs, and custom validators.
- Streaming through `extract_iter` for large text, with offsets preserved across chunks.
- Noise tags, a structural confidence heuristic, and pivot links you open yourself.

Exports are CSV, JSON, JSONL, a defanged list, STIX 2.1, MISP, OpenIOC, and Suricata and Snort rules for addresses, domains, and URLs.

## Command line

```console
qarbaal report.txt
qarbaal mail.eml notes.json -f stix -o bundle.json
qarbaal --hide-private --defang -f table alerts.log
qarbaal --types all -f suricata capture.txt
```

With no file, `qarbaal` reads stdin. Formats: `table`, `csv`, `json`, `jsonl`, `stix`, `list`, `misp`, `openioc`, `suricata`, `snort`.

## Further reading

- [Installation](https://ahur4.github.io/Qarbaal/installation/)
- [Indicator types](https://ahur4.github.io/Qarbaal/indicator-types/)
- [Defanging](https://ahur4.github.io/Qarbaal/defanging/)
- [Normalization](https://ahur4.github.io/Qarbaal/normalization/)
- [Exports](https://ahur4.github.io/Qarbaal/exports/)
- [Command line](https://ahur4.github.io/Qarbaal/cli/)
- [API reference](https://ahur4.github.io/Qarbaal/reference/)
- [Changelog](https://ahur4.github.io/Qarbaal/changelog/)
