Metadata-Version: 2.4
Name: autopyara
Version: 0.1.2
Summary: Next-Gen Automatic YARA Rule Generator
Home-page: https://github.com/Botacin-s-Lab/AutoPYaraPyPI
Author: GIA MTech
Maintainer: Mabon Ninan
Maintainer-email: ninanmm@tamu.edu
License: MIT
Project-URL: Homepage, https://github.com/Botacin-s-Lab/AutoPYaraPyPI
Project-URL: Documentation, https://botacin-s-lab.github.io/AutoPYaraPyPI/
Project-URL: Source, https://github.com/Botacin-s-Lab/AutoPYaraPyPI
Project-URL: Issue Tracker, https://github.com/Botacin-s-Lab/AutoPYaraPyPI/issues
Project-URL: Changelog, https://github.com/Botacin-s-Lab/AutoPYaraPyPI/releases
Keywords: yara,malware-analysis,malware-clustering,bloom-filter,threat-intelligence,security
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jpype1
Requires-Dist: yara-python
Requires-Dist: yaramod
Requires-Dist: scikit-learn>=1.0
Requires-Dist: numpy<2.0,>=1.20
Requires-Dist: packaging>=20.0
Requires-Dist: ppdeep
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: maintainer
Dynamic: maintainer-email
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<div align="center">

# AutoPYara

**Automated, Cluster-Driven YARA Rule Generation**

[![PyPI version](https://img.shields.io/pypi/v/autopyara.svg)](https://pypi.org/project/autopyara/)
[![Python Versions](https://img.shields.io/pypi/pyversions/autopyara.svg)](https://pypi.org/project/autopyara/)
[![Downloads](https://img.shields.io/pypi/dm/autopyara.svg)](https://pypistats.org/packages/autopyara)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Docs](https://github.com/Botacin-s-Lab/AutoPYaraPyPI/actions/workflows/docs.yml/badge.svg)](https://botacin-s-lab.github.io/AutoPYaraPyPI/)
[![Release Pipeline](https://github.com/Botacin-s-Lab/AutoPYaraPyPI/actions/workflows/release.yml/badge.svg)](https://github.com/Botacin-s-Lab/AutoPYaraPyPI/actions/workflows/release.yml)

Automatically discover malware families and generate high-quality, tightly scoped YARA rules using probabilistic clustering and Bloom-filtered n-gram analysis.

[**Documentation**](https://botacin-s-lab.github.io/AutoPYaraPyPI/) · [PyPI](https://pypi.org/project/autopyara/) · [Report a Bug](https://github.com/Botacin-s-Lab/AutoPYaraPyPI/issues) · [Changelog](CHANGELOG.md) · [Releasing](RELEASING.md)

</div>

---

## 📌 Overview

**AutoPYara** is a Python framework for automated YARA rule generation from collections of malware samples. It combines:

- Variational Bayesian Gaussian Mixture Models (VBGMM)
- Augmented DBSCAN with centroid refinement
- Malicious/benign Bloom filter isolation
- Byte-level n-gram feature extraction

The result: **cluster-aware, precision-engineered YARA signatures** with minimal manual effort.

## 🧠 How it works

```mermaid
flowchart TD
    A[Malware Samples] --> B[Byte n-gram Extraction]
    B --> C["Bloom Filter Isolation<br/>(benign removal + malicious focus)"]
    C --> D["Clustering Engine<br/>(VBGMM or Augmented DBSCAN)"]
    D --> E[Cluster-Specific Signature Construction]
    E --> F[High-Quality YARA Rules]
```

## ✨ Features

* **Automated Clustering** — group similar malware samples together automatically to create concise, targeted rules.
* **Two Core Presets** — the standard `AutoYara` (VBGMM) approach, or the enhanced `AutoPYara` (Augmented DBSCAN) pipeline.
* **Built-in Bloom Filters** — ships with pre-trained EMBER and AutoPYara filters to efficiently filter out benign n-grams.
* **Multiple Output Formats** — raw strings, compiled `yara-python` objects, or `yaramod` parsed objects.
* **Custom Training** — train your own Bloom filters on proprietary datasets.

## 🚀 Installation

### Requirements

* Python >= 3.9
* A **Java Runtime Environment (JRE 11+)** on `PATH` or pointed to by `JAVA_HOME`. AutoPYara's clustering/rule-generation backend runs inside a JVM. `pip install` itself doesn't need Java, but `AutoPYara()` will raise a clear error the first time you construct it without one — install a JRE before you actually use the tool. On Debian/Ubuntu: `sudo apt install default-jre`.

```bash
pip install autopyara
```

<details>
<summary>Install from a local build instead</summary>

```bash
python -m build
pip install dist/autopyara-*.whl
```

</details>

<details>
<summary>Note on first run: the Bloom filter data (~600MB)</summary>

To keep the initial install lightweight, the package needs about 600MB of pre-trained Bloom filter data that isn't bundled in the distribution. You don't need to fetch this manually — the first time you `import autopyara` and the data is missing, it's downloaded automatically from the [`data-branch`](https://github.com/Botacin-s-Lab/AutoPYaraPyPI/tree/data-branch) branch of this repository. To trigger it explicitly (e.g. to pre-warm a Docker image):

```bash
autopyara-download
```

</details>

## ⚡ Quick Start

Generating your first YARA rule is as simple as pointing the tool at a directory of malware samples.

```python
from autopyara import AutoPYara

# 1. Initialize the tool
tool = AutoPYara()

# 2. Generate a rule using the AutoPYara preset
results = tool.generate(
    input_files="/path/to/malware/directory",
    preset="AutoPYara",
    rule_name="my_custom_rule",
    output_format="string"
)

# 3. Print the results
print(f"Discovered {results['k_clusters']} distinct malware clusters.")
print("\nGenerated YARA Rule:")
print(results['rule_string'])
```

<details>
<summary><strong>⚙️ Core presets</strong></summary>

#### `preset="AutoYara"` (Standard)

**Algorithm:** Variational Bayesian Gaussian Mixture Model (VBGMM)

**Behavior:** Automatically infers the number of clusters ($K$) probabilistically.

**Best for:** General-purpose rule generation where the structural diversity of the input directory is completely unknown.

#### `preset="AutoPYara"` (Enhanced)

**Algorithm:** Augmented DBSCAN combined with KMeans soft clustering

**Behavior:** Uses a custom Augmented DBSCAN to calculate $K$ prior to centroid optimization.

**Best for:** Producing more tightly bound rules for closely related malware families.

</details>

<details>
<summary><strong>🛠 Advanced usage</strong></summary>

#### Defining a custom $K$

If you want to manually force the algorithm to split your samples into a specific number of clusters, you can override the presets:

```python
# Force exactly 4 clusters using the AutoPYara augmented pipeline
results = tool.generate(
    input_files="/path/to/malware",
    preset="AutoPYara",
    augmented_target_k=4  # Forces the optimizer to find 4 clusters
)
```

#### Output formats

By default, AutoPYara returns a raw string. You can integrate it directly into existing analysis pipelines by requesting Python objects instead:

```python
# Returns a compiled yara-python object ready for immediate scanning
results = tool.generate(
    input_files="/path/to/malware",
    output_format="yara-python"
)

compiled_rule = results["output"]
matches = compiled_rule.match("/path/to/suspicious/file.exe")
```

Supported formats: `'string'`, `'yara-python'`, and `'yaramod'`.

#### Custom Bloom filters

`generate()` defaults to the built-in `"ember"` Bloom filters for both benign and malicious data. You can switch to the `"autopyara"` defaults, or provide absolute paths to your own retrained filters:

```python
results = tool.generate(
    input_files="/path/to/malware",
    bloom_malicious="/absolute/path/to/custom/malicious_bloom",
    bloom_benign="/absolute/path/to/custom/benign_bloom",
)
```

#### Training new Bloom filters

Train custom Bloom filters on your own proprietary benign or malicious datasets with `train()`:

```python
tool = AutoPYara()

# Extract 8-grams from a directory of benign software
tool.train(
    input_dir="/path/to/benign/software",
    output_dir="/path/to/save/new/bloom",
    ngram_size=8
)
```

</details>

<details>
<summary><strong>📚 Full API reference: <code>generate()</code></strong></summary>

| Parameter | Type | Default | Description |
|------------|------|----------|-------------|
| `input_files` | `str` \| `list` | **Required** | Path to input directory or list of sample file paths. |
| `preset` | `str` | `None` | `'AutoYara'` or `'AutoPYara'`. Auto-configures the clustering pipeline. |
| `bloom_malicious` | `str` | `'ember'` | Built-in flag (`'ember'`, `'autopyara'`) or path to custom malicious Bloom filters. |
| `bloom_benign` | `str` | `'ember'` | Built-in flag (`'ember'`, `'autopyara'`) or path to custom benign Bloom filters. |
| `output_format` | `str` | `'string'` | `'string'`, `'yara-python'`, or `'yaramod'`. Determines output rule format. |
| `rule_name` | `str` | `'autoyara_rule'` | Base string used to name the generated rules. |
| `k_cluster` | `int` | `0` | Hardcode $K$ for VBGMM. Do **not** use with `preset="AutoPYara"`. |
| `augmented_target_k` | `int` | `None` | Hardcode target $K$ for the Augmented DBSCAN pipeline. |
| `verbose` | `bool` | `False` | Enable detailed logging during cluster generation. |

</details>

## 📖 Documentation

Full documentation lives at **[botacin-s-lab.github.io/AutoPYaraPyPI](https://botacin-s-lab.github.io/AutoPYaraPyPI/)**. It's intentionally basic for now — installation, quick start, and the API reference — with more material (including the accompanying paper, once published) landing there over time.

## 🧪 Development

```bash
pip install -e ".[test]"
pytest tests/
```

`tests/test_core_helpers.py` and `tests/test_augmented_dbscan.py` are pure-Python unit tests (no JVM/network needed). `tests/test_smoke_generate.py` runs the real pipeline end-to-end against small synthetic dummy files (not real malware) using the built-in Bloom filters.

See [RELEASING.md](RELEASING.md) for how versioning and PyPI publishing work.

## 🤝 Contributing

We're accepting contributions — if you run into an issue or have a fix, fork the repo, open a PR against `main`, and we'll take a look. PRs are automatically built and tested; once checks pass and a maintainer approves, it gets merged. `main` itself isn't open to direct pushes from anyone (including maintainers) — everything goes through review. See [CONTRIBUTING.md](CONTRIBUTING.md) for details.

## License

MIT — see [LICENSE](LICENSE).

---

<div align="center">

Maintained by Mabon Ninan, Texas A&M University — [ninanmm@tamu.edu](mailto:ninanmm@tamu.edu)

</div>
