ThreatLens · ZEIT3118 Capstone

Quality Checker

The quality checker (src/threatlens/quality/quality_checker.py) takes the extraction engine's output for one report and decides what each candidate is, and how good it is as data. It answers the three quality problems named in the client's brief:

  1. Duplication: the same observable extracted more than once, or one value nested inside another.
  2. Missing context: indicators with little supporting information about who, what or which campaign.
  3. False positives: values shaped like IoCs that are not malicious, such as public resolvers, the publisher's own URL, or technique names.

Whatever survives is scored against the quality framework from the project's research paper (Table 7), giving each stored indicator a quality_score from 0 to 100 and a verbal grade.

The checker comes after extraction. It reads what the extraction engine produced and never goes back to the report or calls the LLM. Its core, check_report(), is pure: no database, no network, no file access. Writing to the database is a separate adapter, persist_report(), so the same checks run with or without a database.

How it is invoked

Nothing runs the quality checker on its own schedule. It is called in three ways, and only the first one writes to the database:

RouteWho calls itInputOutput
Ingest pipeline dashboard/ingest_worker.py, which sits behind the dashboard's Ingest page, the threatlens-ingest command and .dev_tools/run_ex_pipeline.py The IoCExtractionResult, in memory The SQLite database, committed
Standalone CLI python -m threatlens.quality.quality_checker, by hand One or two extractor JSON files, written by threatlens -o A console summary, plus a JSON report with -o. No database.
Library Any Python code, e.g. tests Either of the above A QualityReport object; the caller decides whether to persist it

The dashboard also uses parts of the module after ingest: rescore_from_db() runs from the Configuration page's rescore action (POST /api/config/rescore). The indicator detail page uses ScoringPolicy, maliciousness_from_verdict() and SCREENED_FLAGS to explain a stored score.

Route 1: the ingest pipeline (to the database)

This is the main use. For each report, ingest_worker.run() does:

analyse_document(pdf)                         extraction engine (llm or python)
      │  IoCExtractionResult (in memory)
      ▼
run_from_payload(result, source=pdf)          calls result.to_dict(); no file written
      │  ExtractionRun
      ▼
check_report([run], policy=ScoringPolicy.from_session(session))
      │  QualityReport
      ▼
persist_report(session, report,               sources, raw_documents, indicators,
               source_name=pdf.name,          sightings, quality_flags
               file_path=<data-folder-relative pdf>)
session.commit()
      │
      ▼
rescore_from_db(session)                      timeliness + accuracy refreshed for
session.commit()                              every stored indicator

The persist step and the rescore step are committed separately. If anything fails, the session is rolled back and the worker exits with code 1. The extraction result is never written to disk on this route. Only one engine runs per ingest, so check_report always gets a single run, and every candidate's agreement grade is single_verifier.

How to start an ingest is covered on the Dashboard and CLI pages. From a terminal:

threatlens-ingest report.pdf --db db/threatlens.db --engine llm

Before extracting, the worker runs the setup check and logs a [WARN] line if threatlens.conf is missing, invalid or still the unedited template, with either engine, since the weights it scores with come from that file. With the llm engine it also warns if the [llm] settings were never calibrated. The ingest still runs.

Route 2: the standalone CLI (to the console and JSON)

The module has its own command line for inspecting the quality stage without touching a database. It reads the JSON that the threatlens command writes with -o, so the two commands chain:

threatlens report.pdf -o llm.json
threatlens report.pdf --engine python -o python.json

python -m threatlens.quality.quality_checker llm.json
python -m threatlens.quality.quality_checker llm.json --compare python.json --show all
python -m threatlens.quality.quality_checker llm.json -o quality.json
ArgumentMeaning
extractionExtractor JSON to assess (required)
--compare FILEThe other engine's JSON for the same report. Turns on cross-verifier agreement grading; without it every candidate is single_verifier. This is the only place in the project where the two engines' verdicts are compared.
-o, --output FILEWrite the full quality report as JSON (parent directories are created)
--showsummary (default), indicators, screened, rejected or all: how much per-candidate detail to print

It exits with 0 on success, or 1 with error: … on stderr when an input cannot be read or parsed, or a value fails conformance. The help text calls the program threatlens-quality, but no such command is installed (it is not in pyproject.toml); use python -m as shown. With no database, the CLI takes its scoring settings from the built-in defaults plus threatlens.conf (see Settings).

Example console output (python engine, one of the test fixtures):

report:    SnappyClient – Malware Trends Tracker by ANY.RUN (2026-07-22)
verifiers: python
verdict:   10 indicators, 1 screened, 0 context entities, 6 rejected (from 17 candidates)
quality:   mean score 16.0/100 (Unreliable)
agreement: single_verifier=17
issues:
               4  sentence_fragment
               2  self_reference
               1  invalid_format
               1  span_overlap
note:      no extractor field supplies ['threat_actor', 'campaign', 'vulnerability', 'tool'], so completeness is capped for every indicator

Route 3: as a library

from threatlens.extractors.ioc_extract_llm_fs import analyse_document
from threatlens.quality.quality_checker import (
    check_report, load_run, persist_report, rescore_from_db, run_from_payload,
)

# In memory, as the ingest pipeline does
report = check_report([run_from_payload(analyse_document("report.pdf"))])

# Or from JSON files, as the CLI does
report = check_report([load_run("llm.json"), load_run("python.json")])

print(report.summary())        # counts, mean score, grade, agreement, issues
data = report.to_dict()        # what the CLI's -o writes

# Only if a database is wanted (the caller commits)
persist_report(session, report, source_name="report.pdf", file_path="reports/report.pdf")
session.commit()
rescore_from_db(session)
session.commit()

Data in

Everything the checker reads is normalised into an ExtractionRun: the report's metadata, the verification_model, the extractor's summary, and a tuple of Candidates. There are two ways to build one, and both go through the same parser, so they cannot drift apart:

Fields the checker uses from the extractor's output:

FieldUsed for
verification_modelWhich verifier produced the run. python-heuristics means the python engine; anything else is treated as the LLM.
metadata.publish_dateTimeliness, and the indicator's first_seen / last_seen
metadata.source_url, title, filenameThe self-reference check; the source's name and URL in the database
metadata.file_sha256The document's identity in the database
indicators[]: id, category, type, value, normalised value, offsets, context excerptThe candidates assessed by all four layers
indicators[].verification: status, confidence, reasonThe verdict used in reconciliation, translated into indicators.confidence

When two runs are given (CLI --compare), candidates are matched by candidate id. That works because both engines share the same candidate stage and produce identical ids for the same report (see Extraction Engine). The checker assesses the union of both runs' candidates. The extractor's scope and summary are not used.

Settings

Weights and bands come from a ScoringPolicy, resolved in this order. Each later source overrides the earlier ones:

  1. ScoringPolicy.default(), built into the module. It mirrors the scoring_config rows that seed.py writes.
  2. The database's scoring_config table: the latest row per dimension, read by ScoringPolicy.from_session(). Only the ingest pipeline (and rescore_from_db) has a session, so the standalone CLI skips this step.
  3. threatlens.conf (located by paths.config_file()): [quality.weights] and [quality.agreement], when present. The dashboard's Configuration page edits these. Existing indicators keep their old scores until they are rescored. The same file holds the LLM settings ([ollama], [models], [llm]), which threatlens-calibrate rewrites without touching these two sections.

The allowlist used for false-positive screening is seed.py's ALLOWLIST, in every route. check_report() accepts allowlist_entries, and allowlist.load_allowlist(session) can read the database's allowlist table, but no current caller passes it. Edits made to that table do not yet affect ingest.

What it does

ExtractionRun(s) ─▶ Layer 1 ─▶ Layer 2 ─▶ Layer 3 ─▶ decision ─▶ Layer 4 ─▶ QualityReport
                    structural  STIX 2.1   false-                 dimension
                    checks      conformance positive              scoring
                                            screen
LayerQuestionFlags it can raise
1. Structural checksIs this a well-formed observable? span_overlap, nested_value, filename_as_domain, duplicate_observable, sentence_fragment, trailing_punctuation, length_bounds, truncated_span, technique_idiom, self_reference
2. STIX 2.1 conformanceIs it an SCO (an observable, so a possible indicator), an SDO (context), or neither? not_stix_conformant, invalid_format, unmapped_ioc_type
3. False-positive screenIs it well-formed but not malicious? known_benign (allowlist), non_routable (private, loopback, reserved addresses)
DecisionApplying the verifier's verdict verifier_rejected, benign_observable, verifier_uncertain
4. Dimension scoringHow good is what is stored?(scores, not flags)

Every flag has a dimension and a severity (FLAG_DIMENSIONS). Severity drives the decision: error flags (malformed, non-STIX) reject, warning flags (benign) screen, and info flags (verifier_uncertain, truncated_span) keep the indicator with a caveat.

Decisions

Each candidate gets one Decision, applied in this order:

DecisionWhenIn the database
REJECTEDAny malformed flag, or no STIX representationNothing. No indicator, no sighting, no flag. It appears only in the JSON report and console output.
CONTEXTAn SDO: threat actor, malware, campaign, tool, vulnerability and so onNo row of its own. Its value is merged into indicators.context of every indicator stored from that report.
SCREENEDA screen flag, or the verifier judged it not malicious (verifier_rejected) or a benign observable (benign_observable)indicators row (with a quality score, confidence = 0), a sighting, and its flags
INDICATOREverything else: confirmed, uncertain, or no verdictindicators row, a sighting, and any flags (e.g. verifier_uncertain)

Structure comes first: a malformed value is rejected whatever either verifier said. Screening keeps the row so that a real observable the report mentions (a victim's domain, a legitimate binary the malware abused) stays on record and reviewable, without ending up in a malicious-indicator feed.

Verifier reconciliation

With two runs, compare_verifiers() grades how far the verdicts agree (Agreement: agree, weak, moderate, strong), and the LLM verdict wins among candidates that survive Layers 1–2. With one run, the grade is single_verifier. The grade multiplies the verdict-derived confidence (1.00 / 0.90 / 0.75 / 0.50, or 0.85 for a single verifier). When the two verdicts agree, their readings are averaged. The result becomes indicators.confidence.

Two scores, two meanings

ColumnMeaning
indicators.confidence (0–100)How certain it is that the indicator is malicious. Built from the verdict through maliciousness_from_verdict(), because the extractor's own verification.confidence is confidence in the verdict (the python engine rejects at 0.98; the LLM gives 0.0 to every rejection).
indicators.quality_score (0–100)How fit the indicator is as data. This module's composite of the dimension scores.

The quality dimensions

Each stored candidate (INDICATOR or SCREENED) is scored 0–5 on four dimensions. The composite is their weighted mean, scaled to 0–100, and graded Actionable (≥ 85), Reliable (≥ 75), Usable (≥ 65), Marginal (≥ 50) or Unreliable.

DimensionMeasuresDefault weight
TimelinessAge of the report's publish_date (under 7 days scores 5, under 30 days 4, under 90 days 3, under 365 days 2, older 1, unknown 0)0.25
AccuracySource reliability: the percentage of the report's candidates that were not rejected (≥ 90% scores 5, ≥ 80% 4, ≥ 70% 3, ≥ 55% 2). A report with fewer than 10 candidates is capped at 3.0.30
CompletenessThe share of the expected context keys (threat_actor, campaign, vulnerability, tool) filled from the report's context entities0.25
RelevanceHow well the report's audience (tactical vs strategic) matches the reader's role0.20

Notes on the current behaviour:

Data out

Every route first produces a QualityReport: the report's metadata, scored_at, the document context, the policy used, and one CandidateAssessment per candidate (decision, STIX kind, issues, dimension scores, quality score, grade, confidence, agreement). Where it goes next depends on the route. Nothing is passed back to the extraction engine.

To the database: persist_report()

Used by the ingest pipeline. report.scored_rows() builds an indicators row for each stored assessment and validates it against the schema (conformance.validate_indicator_row), so a bad value fails with the column named. db.ingest.ingest_scored_rows() then writes those rows unchanged, and raise_flags() records the issues. The caller commits.

TableWhat is written
raw_documentsThe report, keyed by metadata.file_sha256. The same file ingested again reuses its existing source.
sourcesName, URL, metadata plus candidate_count, reliability (= source reliability %), extraction_engine (llm, python, or llm+python after re-ingesting with the other engine) and verification_model
indicatorsOne row per unique (ioc_type, normalised): confidence, quality_score, score_components (dimensions, weights, grade, agreement, STIX kind), context (the document context merged with the extractor's excerpt), scored_at, and first_seen / last_seen: the earliest and latest publish dates of the reports that sighted it (never the ingest time). If the indicator already exists, its score, components and scored_at are updated and its dates re-derived; its confidence is left unchanged, and its context only fills in if it was empty.
sightingsOne per (indicator, source), with the excerpt and its offsets
quality_flagsOne per issue on a stored indicator: flag_type, dimension, severity, detail, status = 'open'. A flag type already recorded against that indicator is not added again. Rejected candidates have no indicator, so their flags are not stored.

After that, rescore_from_db() recalculates timeliness and accuracy for every indicator in the database, because timeliness changes with the calendar and accuracy depends on the reliability of the best source that reported each indicator. Completeness and relevance are read back from score_components. It updates quality_score, score_components and scored_at.

Not stored: rejected candidates, context entities as rows of their own, the audience profile, and the report-level summary. The dashboard reads everything else back from these tables.

To JSON and the console: the standalone CLI

The CLI prints the summary shown above. With -o it writes report.to_dict():

{
  "metadata":         { …the extractor's metadata… },
  "scored_at":        "2026-10-13T…Z",
  "summary": {
    "candidates", "indicators", "screened", "context_entities", "rejected",
    "verifiers", "mean_quality_score", "grade", "agreement", "issues",
    "unpopulatable_context_keys", "source_reliability",
    "audience_profile", "audience", "role"
  },
  "document_context": { "threat_actor": [...], "malware": [...], … },
  "assessments": [
    { "candidate_id", "indicator_type", "ioc_type", "value", "normalised",
      "decision", "stix", "dimensions", "quality_score", "grade",
      "confidence", "agreement", "issues": [{flag_type, dimension, detail, severity}] }
  ]
}

Unlike the database, the JSON report includes every candidate, rejected ones too, each with its flags. That makes it the place to see why something was dropped. Nothing reads this file back into the project; it is for inspection.

Extending it

Source