ThreatLens · ZEIT3118 Capstone

Extraction Engine

The extraction engine turns a CTI report (PDF or plain text) into a typed set of candidate indicators of compromise, plus document metadata. It works in two stages that are kept strictly apart:

a candidate stage: deterministic regex rules decide where something worth looking at is in the text, and
a verification stage: a verifier reads each candidate's surrounding text and decides whether it is an IoC the report's authors are actually presenting.

Verification classifies; it never extracts. Every value in the output was found by the regex stage at a known offset, so a verifier cannot invent an indicator, and the start/end offsets can be trusted.

Two engines, one front end

There are two engines, chosen per run as llm (the default) or python. They are two modules, but not two separate extractors:

ioc_extract_llm_fs.py                         ioc_extract_python.py
┌──────────────────────────────────────┐
│ DocumentReader     (read PDF / text) │
│ MetadataExtractor  (title, date, …)  │ ◀──── imports ─── steps 1–3 and the
│ CTIExtractor       (regex candidates)│                   domain model
│ IoCExtractionResult (output object)  │
├──────────────────────────────────────┤      ┌───────────────────────────┐
│ DocumentProfiler + LLMVerifier       │      │ HeuristicVerifier         │
│ (Ollama)                             │      │ (local regex rules)       │
└──────────────────────────────────────┘      └───────────────────────────┘

ioc_extract_llm_fs.py owns the shared front end: reading, metadata, candidate extraction, the domain model (Extraction, Verification, IoCExtractionResult, …). ioc_extract_python.py imports all of that and adds only its own verifier. The consequences:

For the same report, both engines produce the identical candidate set: same values, same offsets, same candidate ids (stable_id, a hash of category, type and normalised value). Only the verdicts differ.
Both return the same IoCExtractionResult object, so every caller handles either engine with the same code.
A change to reading or candidate extraction belongs in ioc_extract_llm_fs.py; the python engine picks it up automatically.

	`llm` — ioc_extract_llm_fs.py	`python` — ioc_extract_python.py
Verifier	`LLMVerifier`: an Ollama-hosted model reads each candidate against a written rubric	`HeuristicVerifier`: keyword and structure rules around each candidate
Document profile	Yes: one `DocumentProfiler` call per report (structure summary + scope)	No
Context used	The candidate's ±300-character excerpt, the document metadata and the structure summary	A ±140-character window inside that excerpt, plus the document metadata
Reasons	The model's own reason and evidence quote for each decision	A fixed reason string for whichever rule fired
Needs	A reachable Ollama server with the model pulled	Nothing: no network, no settings files
Speed	Minutes per report (mean 4m 50s in the ex4 run)	Seconds per report
`result.model`	The verifier model, e.g. `qwen3.5:9b`	`python-heuristics`
`result.scope`	A `ScopeAssessment`	`None`
Role	The default, and the engine every evaluation run uses	A deterministic fallback for when no LLM is available

The engines are alternatives, not a pipeline: a single run uses exactly one of them. They meet in two places. If a report already in the database is ingested again with the other engine, its source is marked extraction_engine = 'llm+python' instead of being overwritten. And the quality checker can grade how far the two engines agree, but only when given both engines' JSON from its own command line (see Hand-off).

Pipeline

report (.pdf via PyMuPDF, or .txt/.md/.log/.csv/.json)
      │
      ▼
1  DocumentReader.load        text + SHA-256 of the file bytes + embedded PDF metadata
      │
      ▼
2  MetadataExtractor          title, authors, cti_source, source_url, publish_date
      │                       (conservative heuristics; each field records its method)
      ▼
3  CTIExtractor.extract       regex candidates, each with a ±300-char context excerpt
      │
      ├──────────── llm engine ────────────┐     ┌──── python engine ────┐
      ▼                                     │     ▼                       │
4  DocumentProfiler (1 LLM call)            │  4  HeuristicVerifier       │
      │  structure summary + scope          │     (per candidate)       │
      ▼                                     │                           │
5  LLMVerifier (batched LLM calls)          │                           │
      │  verdict, confidence, reason,       │                           │
      │  evidence per candidate             │                           │
      ▼                                     │                           │
6  _assess_scope                            │                           │
      │                                     │                           │
      ▼                                     ▼                           ▼
7  IoCExtractionResult  ── metadata, items (candidate + verification), model, scope

Steps 1–3: the shared front end

Reading. PDFs are read with PyMuPDF (page text joined in order). Text is normalised to repair artefacts that would break indicators: hyphenated tokens wrapped across lines, and Markdown escape backslashes that split defanged values such as example\[.\]com. The SHA-256 of the file bytes becomes the document's identity in the database.
Candidates. CTIExtractor runs the RULES table (34 typed regex rules: URLs, IPv4/IPv6, hashes, file names and paths, registry keys, e-mail, CVEs, crypto addresses, and more), then dedicated passes for domains, hostnames, scheme-less URLs, e-mail domains and intelligence entities (malware, threat actors, campaigns, tools). Defanged forms (hxxp, [.], [at]) are matched and refanged. Candidates nested inside a longer candidate are removed, and duplicates (same category, type and normalised value) are collapsed, keeping the one with the longest excerpt.
Deliberately over-inclusive. This stage is built for recall. It is meant to catch everything shaped like an indicator and leave the judgement to verification.

Steps 4–6: LLM verification

Profile once. DocumentProfiler sends the title, section headings and the first 2,000 characters (not the whole report). It gets back a short structure summary, such as "section 4 lists victim organisations, not attacker infrastructure", and a scope (tactical, operational or strategic) with a reason. A separate profiler model can be configured; by default it is the verifier model.
Verify in batches. LLMVerifier sends batch_size candidates per request (threatlens.conf [llm] batch_size; 5 by default, set by threatlens-calibrate). Each request carries the document metadata, the structure summary, and for each candidate its type, value, attributes and excerpt. The model never sees the whole document. It answers under a written rubric: citations, vendor pages and benign examples are rejected; malware names, actors, CVEs and ATT&CK techniques are context_entity; a value in the report's own IOC table is confirmed_ioc; and so on. Each decision comes with a confidence, a reason and an evidence quote.
Deterministic, structured output. Requests use Ollama's JSON-schema structured output at temperature 0 with thinking disabled, so the same model, settings and input give the same answer. The ex3 and ex4 runs produced identical results (see the deviation report).
Retries. A response is only checked structurally: valid JSON, and exactly one decision per candidate id. A failing batch is retried up to max_retries times; a truncated response grows num_predict and num_ctx before the retry. If a batch still fails, it is split in half and each half is retried on its own (recursively), since a failure that repeats at temperature 0 is deterministic and only a different request routes around it. Only a single candidate that still fails aborts the report.
Scope assessment. _assess_scope combines the profile's scope with flags taken from the accepted candidates: tactical content (network/host/e-mail observables), operational content (behaviour or TTPs) and strategic content (the profile itself).

Step 4 (python engine): heuristic verification

HeuristicVerifier checks rules in a fixed order and stops at the first that fires:

intelligence entities → context_entity (or rejected in pure reference context);
behaviour descriptions → context_entity;
reference wording just before the value, or a match on the report's own source → rejected;
per-type rules for e-mail addresses, hashes, host artefacts and network observables, using an IOC-section heading, malicious-context words (MALICIOUS_CONTEXT_RE), malicious actions (MALICIOUS_ACTION_RE), defanging and benign wording;
otherwise uncertain.

Why regex alone isn't enough, and why the LLM is the default

Left unchecked, the candidate stage produces plenty of noise: a citation URL, a vendor's own domain, an ATT&CK name that looks like a cryptonym, a victim's domain in a targets table, a signed binary mentioned only because malware abused it. Whether such a value is an IoC depends on what the surrounding sentence, or the report's structure, says. Matching the string itself cannot settle it.

The python verifier narrows the gap with keyword rules one layer up, but with the same limits:

No semantic reasoning. "No record of malicious activity was found at example.com" contains the same trigger words as a real detection.
No document-level context. Without a profile it cannot tell a victims table from an infrastructure table, or a side-loaded signed binary from the payload dropped beside it.
No justification. A keyword hit produces a status directly, with no explanation in terms of the actual sentence.

The LLM reads the excerpt against a written rubric and the document profile, and explains each decision. That is why it is the default; the python engine is there for when no LLM is available.

The verdict is final within the engine

analyse_document() attaches each verifier decision to its candidate unchanged. Nothing in the extraction engine adjusts, blends or second-guesses a verdict. Downstream, the quality checker can still reject a candidate on structural or STIX grounds whatever the verdict, but it does not re-run verification.

Where it is called from

Every caller goes through one function per engine, analyse_document():

Entry point	Calls the engine via	What happens to the result
`threatlens` (CLI)	`cli.py`	Summary printed; with `-o`, written as JSON. No database.
Dashboard Ingest page (file queue and "Download from URL")	`ingest_jobs.py` starts `python -m threatlens.dashboard.ingest_worker`, one process per report	Quality checks, written to the active database, rescored
`threatlens-ingest`	`ingest_worker.py`, run directly in the terminal	Same as the Ingest page
`.dev_tools/run_ex_pipeline.py` (evaluation runs)	`ingest_worker.run()`, per report in a batch	Same as the Ingest page
`python -m threatlens.extractors.ioc_extract_llm_fs` / `…ioc_extract_python`	Each module's own `main()`	Developer tool: JSON output, with extra flags (`--no-llm`, `--keep-duplicates`, per-setting overrides). The llm module reads `threatlens.conf` (or `--config FILE`) like the CLI, with each flag overriding its setting; the python module reads no settings.

Why there are two routes

The CLI is for looking, not storing. It runs only the extraction stage, so a change to the extractor, or the difference between the two engines, can be examined on one report without creating database rows. Its JSON output is a complete record of what the engine decided.
The web UI is for ingesting. Extraction is only the first of four steps, and the Ingest page needs the other three. There is one implementation of that pipeline, ingest_worker.py. The Ingest page, threatlens-ingest and the batch runner all use it, so a database built from the terminal matches one built from the browser.
The dashboard runs the engine in a separate process. An LLM run takes minutes. In a worker process it can be cancelled by killing the process, a crash in the extractor cannot take the dashboard down, and the worker's verbose log can be streamed to the page live. Jobs run one at a time because every job writes the same SQLite file and one LLM job already keeps Ollama busy.
The engine has no command-line or web code of its own in either route. Both callers pass in already-resolved settings and get back a Python object, so the engine stays testable on its own.

Settings

The python engine reads no settings. For the llm engine, both cli.py and ingest_worker.py take every setting from threatlens.conf, with built-in defaults for anything it leaves out: the Ollama URL ([ollama]), the verifier and profiler models ([models]) and the performance tuning ([llm]: batch size, timeouts, context sizes, retries). extractors/calibration.py (threatlens-calibrate) benchmarks the server and writes [ollama], the verifier model and [llm]; the dashboard's Configuration page edits the URL and models and keeps [llm]. (These used to be split with a separate llm_calibration.conf, looked up in the current directory by the CLI but in the repo root by the worker; there is now one file and one lookup.) Before an llm run, both callers print a warning for each setup step not yet done. See CLI for the full table and calibration.

Hand-off

The engine's output is always an IoCExtractionResult. How it leaves the engine depends on the caller.

From the CLI: JSON, on request

IoCExtractionResult ──to_dict()──▶ JSON file (-o)
                                     │  optional, by hand:
                                     ▼
                     python -m threatlens.quality.quality_checker out.json [--compare other.json]
                                     │  load_run() → check_report()
                                     ▼
                              printed quality report (no database)

The JSON contains metadata, verification_model, scope, summary, every candidate in indicators (with its verification), and accepted_by_category. The quality checker reads this format directly. Giving it both engines' JSON for the same report (--compare) is the only place the two verdicts are graded against each other. See CLI.

From the web UI and `threatlens-ingest`: in memory

IoCExtractionResult
      │  quality_checker.run_from_payload(result)   (calls result.to_dict(); no file)
      ▼
ExtractionRun ──▶ check_report([run], policy from the database's scoring_config)
      │
      ▼
QualityReport ──▶ persist_report() ──▶ sources, raw_documents, indicators,
      │                                 sightings, quality_flags   (commit)
      ▼
rescore_from_db() ──▶ timeliness / accuracy refreshed for stored indicators (commit)

The worker never writes the result to disk. It runs one engine, so check_report gets a single run and every candidate's agreement grade is single_verifier. What travels where:

From the extraction result	Ends up as
`metadata.file_sha256`	The document's identity (`raw_documents.sha256`); re-ingesting the same file reuses its source
`metadata` (title, date, URL, …)	`sources.raw_metadata`, `sources.url`; `publish_date` drives timeliness
`model`	`sources.verification_model`, and `sources.extraction_engine` (`llm` / `python`, widened to `llm+python` on re-ingest with the other engine)
Each candidate's value, type, corrected type, offsets	An `indicators` row and a `sightings` row, if it survives the quality checks
Each candidate's `context` excerpt	`sightings.excerpt`, one per report. The first report's excerpt is also kept in `indicators.context`, alongside the structured context the quality checker builds from the report's context entities (see Database & Schema).
Verdict + verdict confidence	Translated into `indicators.confidence` (maliciousness) and kept in `score_components`
`scope`, `summary`	Not stored. The quality checker works out the report's audience from the surviving candidates instead, though relevance currently scores 3 for every indicator (see Quality Checker).

See the Quality Checker for what happens to each candidate after hand-off.

Source

src/threatlens/extractors/ioc_extract_llm_fs.py: shared front end, domain model, LLM engine (analyse_document, DocumentProfiler, LLMVerifier)
src/threatlens/extractors/ioc_extract_python.py: python engine (HeuristicVerifier, analyse_document)
src/threatlens/extractors/calibration.py: benchmarks an Ollama server and writes the [llm] section of threatlens.conf (threatlens-calibrate)
src/threatlens/cli.py: the threatlens command
src/threatlens/dashboard/ingest_worker.py: the ingest pipeline (threatlens-ingest); ingest_jobs.py: the dashboard's queue around it