Metadata-Version: 2.5
Name: CVE-RiskEngine-RF
Version: 0.1.2
Summary: Calibrated random forest scoring of CVE exploitability against disclosure-time EPSS, with the honest limits of the reference study attached.
Project-URL: Homepage, https://pypi.org/project/CVE-RiskEngine-RF/
Project-URL: Repository, https://github.com/rezayw/CVE-RiskEngine-RF
Project-URL: Paper, https://github.com/rezayw/CVE-RiskEngine-RF/tree/main/IEEEtranBST2
Author-email: Reza Yuzron Wardana <23525008@mahasiswa.itb.ac.id>, Hari Purnama <Hari.Purnama@itb.ac.id>
License: MIT License
        
        Copyright (c) 2026 Reza Yuzron Wardana and Hari Purnama
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cisa-kev,cve,epss,exploitability,machine-learning,prioritization,random-forest,security-operations,vulnerability-management
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Requires-Dist: joblib>=1.4
Requires-Dist: numpy>=1.26
Requires-Dist: scikit-learn<1.7,>=1.4
Provides-Extra: api
Requires-Dist: fastapi>=0.111; extra == 'api'
Requires-Dist: pydantic>=2.7; extra == 'api'
Requires-Dist: uvicorn[standard]>=0.29; extra == 'api'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: twine>=5.1; extra == 'dev'
Description-Content-Type: text/markdown

# cve-risk-engine

Calibrated random forest scoring of CVE exploitability, packaged with the honest
limits of the study it comes from.

This is the reference implementation of the scoring layer of

> **Exploitability-Driven CVE Prioritization for Security Operations: A Recursive
> Feature Elimination Random Forest Approach** — Reza Yuzron Wardana, Hari Purnama.
> Manuscript in preparation. Code, data and evaluation artifacts:
> [CVE-RiskEngine-RF](https://github.com/rezayw/CVE-RiskEngine-RF)

**What this package is not.** It is not a claim that machine learning beats EPSS. On
the study's held-out temporal test partition the forest ranks better than
disclosure-time EPSS, but the difference in average precision is not statistically
established, the raw scores are not probabilities, and a CVSS-only score stays
close behind. The package ships those facts as metadata and refuses to return an
uncalibrated number as a probability.

## Install

```bash
pip install CVE-RiskEngine-RF
# optional HTTP service
pip install "CVE-RiskEngine-RF[api]"
```

## Use

```python
from cve_risk_engine import CveRiskEngine

engine = CveRiskEngine.load()
result = engine.score(
    cvss_vector="CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
    cvss_score=9.8,
    epss_score=0.00061,   # EPSS at or before the disclosure date, not today's value
    cve_id="CVE-2025-14611",
    cwe="CWE-798",
)
```

```python
{
  "cve_id": "CVE-2025-14611",
  "raw_model_score": 0.897937,
  "calibrated_probability": 0.012108,
  "cvss_score": 9.8,
  "epss_score": 0.00061,
  "final_risk_score": 0.4961,
  "risk_level": "medium",
  "priority_band": "critical",
  "priority_score": 0.012108,
  "priority_action": "Patch within 7 days",
  "recommendation": "Patch within 30 days",
  "features": {...}
}
```

`risk_level` and `priority_band` disagree on purpose. The first is the CVSS severity
scale, the second is the operational band. See "Prioritize with `priority_band`" below.

Command line:

```bash
cve-risk-engine info
cve-risk-engine score --vector "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H" \
                      --cvss 9.8 --epss 0.00061 --cwe CWE-798 --id CVE-2025-14611
cve-risk-engine batch records.csv --out scored.csv
```

HTTP:

```bash
uvicorn cve_risk_engine.api:app --port 8000
# GET /health  GET /model  POST /score  POST /score/batch
```

## Read the two numbers differently

| field | what it is | what it is not |
|---|---|---|
| `raw_model_score` | forest output, useful for **ranking** | a probability; Brier 0.0629 against 0.0019 for EPSS |
| `calibrated_probability` | Platt map fitted on the study's validation partition | a per-CVE certainty; the base rate is 0.245% |
| `final_risk_score` | `0.5 · calibrated + 0.5 · CVSS/10` | a fitted risk model; the weights are a policy choice |
| `risk_level` | the CVSS v3.1 severity scale applied to `final_risk_score` | an exploitability verdict; above `medium` it is unreachable, see below |
| `priority_band` | validation quantiles of `calibrated_probability` | a probability; it is a queue position, see below |

## What the model actually does

Corpus: 44,474 CVEs published in 2025 with a CVSS v3.1 vector. Target: CISA KEV
entry **within 90 days of disclosure**, 109 positives (0.245%); CVEs already
KEV-listed at disclosure are excluded, so no record is right-censored. Temporal
split: train 25,951 (70 positives) to 2025-07-31, validation 7,363 (18) Aug–Sep,
test 11,160 (21) Oct–Dec. Features: 7 of 13 candidates retained by RFECV on the
training partition. Imbalance: SMOTE 1:1 chosen on validation, applied inside the
training folds only.

| Metric (test) | Random forest | Disclosure-time EPSS | Paired Δ, 95% CI |
|---|---:|---:|---:|
| ROC-AUC | 0.8807 | 0.7833 | — |
| Average precision | 0.0159 | 0.0074 | +0.0085 [−0.0029, +0.0409], p=0.128 |
| MCC | 0.0589 | 0.0439 | +0.0150 [−0.0102, +0.0400], p=0.248 |
| Recall (validation threshold) | 0.9048 (queue 3,228) | 0.4762 (queue 1,483) | +0.4286 [+0.2222, +0.6364] |
| Brier, raw | 0.0629 | 0.0019 | +0.0610 [+0.0576, +0.0644] |
| Brier, Platt-calibrated | 0.00187 | 0.0019 | — |

Ranking differences on average precision and MCC are inside their bootstrap
intervals. The recall gain is real but comes at roughly twice the queue length.

## KEV membership is not a risk band, and high/critical are unreachable

A natural question is whether a CVE that is listed in CISA KEV comes back as `high`
or `critical`. It does **not**, and the engine never accepts KEV as an input: KEV is
the evaluation label of the study, never a feature, so a served decision is
reproducible from the model alone. On the test partition the 21 KEV-listed CVEs come
back as **15 `medium` and 6 `low`**, none above `medium`, with `final_risk_score`
between 0.3058 and 0.5061. A KEV CVE at CVSS 7.5 lands in `low`; so does one at CVSS
6.1.

The reason is the one in the second limitation below: at a 0.245% base rate,
`calibrated_probability` is at most a few percent, so `FinalRisk` is dominated by the
CVSS term and saturates around 0.53. Read the band as a severity-flavoured label, not
as an exploitation verdict, and never as a KEV oracle. Rank by `final_risk_score` and
cut by queue capacity; that is the operating point the study evaluates.

## Two limitations you must design around

**1. Raw scores are not probabilities.** They are ranked correctly but are badly
calibrated (Brier 0.0629). Always read `calibrated_probability`, never
`raw_model_score`, when a number is shown to a human.

**2. The `high` and `critical` bands are unreachable.** The bounds come from the
CVSS v3.1 severity scale on the unit interval, but once the score is calibrated on
a 0.245% base rate, `final_risk_score` cannot exceed about 0.53 even at CVSS 10.0.
On the test partition the `high` and `critical` bands hold zero records; everything
lands in `low` (8,421 records, 6 exploited) or `medium` (2,739 records, 15
exploited). Precision still rises monotonically between them, but a banding scheme
usable under calibrated probabilities has to be defined on **validation quantiles
of the score**, not on the CVSS scale. For that reason the metadata ships
`capacity_cutoffs_validation` (the score above which the top 0.5%, 1%, 2%, 5% and
10% of the validation backlog fall) and the package README recommends cutting by
capacity. Treating `risk_level` as an absolute severity rating is wrong.

## Prioritize with `priority_band`, not with `risk_level`

The CVSS-scale bounds were derived and verified on the raw score of the pre-revision
model, whose range reached 0.98: on that scale the `high` band held 20 records and
`critical` held 2, with precision rising monotonically to 0.50. They were then carried
over to the calibrated score without being re-derived, and the scale had moved from 0
to 0.98 down to 0 to 0.013. That is why the upper half of the scale is now empty and
why an exploited CVE can come back as `low`.

`priority_band` is the replacement. Its bounds are quantiles of
`calibrated_probability` on the validation partition, shipped in `metadata.json`:

| band | capacity | threshold on `calibrated_probability` |
|---|---|---|
| `critical` | top 5% of the validation backlog | ≥ 0.009084 |
| `high` | next 5% | ≥ 0.003753 |
| `medium` | next 15% | ≥ 0.001873 |
| `low` | remaining 75% | below 0.001873 |

The bounds are not an assumption. Three candidate scales were measured against a rule
fixed in advance, that per-band precision must fall monotonically from `critical` to
`low` and the property must survive out of sample:

| candidate scale | fine cuts (0.5/1/2/5%) | coarse cuts (5/10/25%) |
|---|---|---|
| `final_risk_score` | fails | fails |
| `calibrated_probability` | fails | passes |
| `raw_model_score` | fails | passes |

Fine cuts fail on every scale because the validation partition holds only 18 positives,
so a 0.5% cut is 38 records and one record flips the estimate. `FinalRisk` fails
everywhere because its top is CVSS dominated and its four-decimal rounding creates
large tie mass. `calibrated_probability` and `raw_model_score` rank identically, so the
readable one was chosen.

Applied unchanged to the held-out temporal test partition (11,160 records, 21
positives, base rate 0.188%):

| band | records | positives | precision | recall |
|---|---:|---:|---:|---:|
| `critical` | 644 | 9 | 0.013975 | 0.4286 |
| `high` | 547 | 5 | 0.009141 | 0.2381 |
| `medium` | 1,836 | 4 | 0.002179 | 0.1905 |
| `low` | 8,133 | 3 | 0.000369 | 0.1429 |

Precision falls monotonically, `critical` is 7.4 times the base rate and `low` is 0.2
times it, and 14 of the 21 exploited records land in the top two bands. The same
partition scored with the CVSS-scale bounds puts zero exploited records in the top two
bands. Every number here is reproduced in `metadata.json` under `priority_bands`.

## Blended-score sensitivity on the calibrated probability, for reference: test
average precision rises with the model weight (0.0295 at 0.5/0.5, 0.0331 at
0.7/0.3, 0.0347 at 0.8/0.2) against 0.0159 for the model alone and 0.0102 for CVSS
alone, while the operational cost of a 500-record queue is identical (1,590, with 10
of the 21 exploited CVEs caught) for every blend from 0.5 upward. Equal weights are
used because they sit in the middle of that flat region.

## Bundled artifact

`cve_risk_engine/artifacts/` holds the classifier, the Platt map, the CWE
frequency map fitted on the training partition, and a metadata file recording the
hyperparameters, the target definition, the split windows, the test metrics, the
paired intervals, the input SHA-256 digests and the limitation above. The artifact
reproduces the manuscript's test numbers exactly (average precision 0.015903,
ROC-AUC 0.880724), and `tests/reference_scores.csv` freezes 821 test records
including all 21 positives so anyone can re-verify that claim without the study
repo:

```bash
python -m pytest tests -q
```

## Rebuilding the artifact

The artifact is exported from the study pipeline, never hand-edited:

```bash
python scripts/export_artifact.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out src/cve_risk_engine/artifacts
python scripts/build_reference_scores.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out tests/reference_scores.csv
```

The first script asserts that the exported model's test average precision and
ROC-AUC are bit-identical to `revision/outputs/results.json`; if they drift, the
export fails rather than shipping a lookalike.

## Changelog

**0.1.2** — adds the operational band, and closes the four findings of the independent
audit of 0.1.1 plus one raised by a later regression and comparison run.

- Added `priority_band`, `priority_score` and `priority_action`, defined on validation
  quantiles of `calibrated_probability`. `risk_level` is unchanged, so existing
  consumers keep the same numbers, but it is now documented as the CVSS severity scale
  and explicitly not an exploitability verdict.
- `epss_score` and `cvss_score` are now range checked in `CveRiskEngine.score`. The core
  path previously accepted `-0.1`, `1.5` and `nan` and returned a confident looking
  score for all three. This was audit finding F-09.
- The OpenAPI schema version now comes from `__version__` instead of a hardcoded
  `0.1.0`, and the project name in the `api.py` docstring and `ImportError` is the real
  distribution name, so the instruction it prints can actually be followed. Findings
  F-01 and F-02.
- `metadata.json` now states the decision threshold on the scale it belongs to
  (`raw_model_score` 0.049528, not a calibrated probability), because reusing it on the
  calibrated scale returns an empty queue. Finding F-06.
- `metadata.json` now records the six-decimal rounding of returned scores and the
  consequence for anyone recomputing the published metrics from package output: ROC-AUC
  moves from 0.880724 to 0.881031 and the queue moves from 3,228 to 3,229. Finding F-10.
- The CLI `batch` output carries `priority_band` and `priority_action`; the HTTP
  response carries both plus `priority_score`.

**0.1.1** — first published release: calibrated model, reference scores, CLI and
optional HTTP service.

## Licence

MIT. The bundled artifact is trained on derived metadata from NVD, CISA and FIRST;
the upstream terms of those sources apply to the data.
