Metadata-Version: 2.4
Name: differential_coverage
Version: 0.1.0
Summary: Compute differential coverage for fuzzer runs.
Author-email: Valentin Huber <contact@valentinhuber.me>
License-Expression: MIT
Project-URL: Repository, https://github.com/valentinhuber/differential_coverage
Project-URL: Documentation, https://github.com/valentinhuber/differential_coverage#readme
Keywords: fuzzing,coverage,differential-coverage
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: mypy>=1.19.1; extra == "dev"
Requires-Dist: pre-commit>=4.3.0; extra == "dev"
Requires-Dist: pytest>=9.0.2; extra == "dev"
Requires-Dist: ruff>=0.15.0; extra == "dev"

# Differential Coverage

Absolute coverage numbers aren't painting the whole picture: Different fuzzers may reach the same total coverage, but cover different parts of the program. Fuzzer (A) may reach more coverage than fuzzer (B), but (B) may reach coverage that (A) doesn't reach — so it's still valuable.

## Definition

Differential coverage is a measure for this. It was proposed by Leonelli et al. in their paper on TwinFuzz[^1]. This implementation relies on the formulas from the SBFT’25 Competition Report[^2]. There, Crump et al. define a fuzzers overall score compared to others as

$$
relscore(f,b,s,e) = \left|f_0\in F|e\notin \text{cov}(b,t,s)\forall t\in \text{trials}(f_0,b)\right|
\times\frac
{\left|\{t\in \text{trials}(f,b)\ |\ e \in \text{cov}(b,t,s)\}\right|}
{\left|\{t\in \text{trials}(f,b)\ |\ \text{cov}(b,t,s)\neq \emptyset\}\right|}
$$

The score of a fuzzer is then

$$
\text{score}(f,b,s)=\sum_{e\in E}\text{relscore}(f,b,s,e)
$$

## Explanation
This can be simplified to the following:

$$
\text{differential coverage}(f,e) = \text{number of fuzzers that never hit }e
\times\frac
{\text{number of trials of }f\text{ that hit }e}
{\text{number of trials of }f\text{ with non-empty cov}}
$$
$$
\text{score}(f) = \text{sum of differential coverage}(f,e)\text{ over all edges e}
$$

## Usage
Assume `<input_dir>` is a directory of directories for each fuzzer, where each fuzzer subdirectory contains coverage files for each trial. Currently, the output format of `afl-showmap` is supported (lines with `edge_id:count`, where we only care if `count > 0`).

PRs for other data formats welcome :)

### Command Line Interface
```
differential_coverage <input_dir>
```

#### Example
[`tests/sample_coverage`](./tests/sample_coverage/) provides sample coverage data to explain differential coverage. It contains 3 fuzzers and 3 edges:
- Edge 1 is always hit by all fuzzers
- Edge 2 is always hit by `fuzzer_c`, sometimes hit by `fuzzer_a`, and never hit by `fuzzer_b`
- Edge 3 is always hit by `fuzzer_b` and `fuzzer_c`, but only sometimes by `fuzzer_a`

```
differential_coverage tests/sample_coverage
```
then produces
```
fuzzer_c: 1.00
fuzzer_a: 0.50
fuzzer_b: 0.00
```
### API

```python
from pathlib import Path
from differential_coverage import (
    calculate_scores_for_campaign,
    calculate_differential_coverage_scores,
)

# From a campaign directory (one subdir per fuzzer, each with coverage files):
scores = read_campaign_and_calculate_score(Path("path/to/campaign_dir"))
# -> dict[str, float]: fuzzer name -> score

# From in-memory campaign data (fuzzer -> trial_id -> edge_id -> count):
campaign = {...}  # dict[str, dict[str, dict[int, int]]]
scores = calculate_differential_coverage_scores(campaign)
# -> dict[str, float]: fuzzer name -> score
```

## Installation
```bash
pip install .
```

## Development
Install dev dependencies and set up the pre-commit hook (runs a couple of checks before committing):
```bash
pip install -e ".[dev]"
pre-commit install
```

[^1]: TWINFUZZ: Differential Testing of Video Hardware Acceleration Stacks, https://www.ndss-symposium.org/ndss-paper/twinfuzz-differential-testing-of-video-hardware-acceleration-stacks/

[^2]: SBFT’25 Competition Report — Fuzzing Track, https://ieeexplore.ieee.org/document/11086561
