The other pages in this documentation describe how ThreatLens is meant to work. The reports here show what it actually does when the full pipeline is run over the project's CTI corpus. They give the evidence behind claims made elsewhere, such as whether the extraction is reliable, how long a run takes and whether repeated runs give the same answer.
An evaluation run (labelled ex2, ex3,
ex4, …) processes the whole corpus of 103 CTI report PDFs in
docs/database/CTI/ (third-party reports, kept locally and not
distributed with the repository or package; they are named by their ORKL
archive hash, and .dev_tools/orkl_scraper.py downloads reports
from that archive) into a fresh database,
db/exN_threatlens.db. Each report goes through the same steps as
the dashboard's Ingest page: LLM extraction, quality checks, persistence and
a rescore of every stored indicator. Runs are driven by
.dev_tools/run_ex_pipeline.py, which splits the corpus into
batches, and .dev_tools/run_ex_batches.sh, which chains those
batches.
To run one now:
threatlens-calibrate # once per machine: tune threatlens.conf [llm] for the Ollama server
python .dev_tools/run_ex_pipeline.py --db-path db/exN_threatlens.db --batch 1/3The runner creates the database with the migrations shipped in the
package (threatlens.db.migrate), seeds the reference tables if
they are empty, and takes every LLM setting (URL, models, batch size,
timeouts) and the quality weights from threatlens.conf. Check
its [quality.weights] first; the worker's [WARN]
lines flag a missing, unedited or uncalibrated file. The runner is a
repo-checkout tool and is not part of the pip package.
Because every run starts from an empty database and uses the same input, the run databases can be compared directly with each other.
A single run from start to finish: configuration, per-batch timing, the funnel from regex candidates to stored indicators, and per-report detail compared with ex3.
How far stored results differ between runs over the same corpus. This covers LLM verdicts, confidence, context fields, quality flags and stored values, scored for each report entry.
Each report is a snapshot of the code, model and configuration in use when it was written. It does not update when the pipeline changes. Check a report's run date and configuration before relying on its figures, and prefer the newest report when two disagree.
The runs below predate the single configuration file: their LLM batch
size and timeout came from the former llm_calibration.conf,
whose settings now live in threatlens.conf [llm]. The pipeline
itself is unchanged, so the figures stand.